COOP is developing several python tools. Most are open-source packages installable via PyPI from anywhere the Internet is reaching: arnica, calcifer, h5cross, satis, nob, opentea, nobvisual, tiny-3-engine. Closer to Cerfacs core competencies, a limited set of tools require permissions on the Cerfacs Gitlab Forge: pyavbp, tekigo, lemmings, oms. The recommended way to use these tools is the standard virtual environment technique, using a ‘develop’ installation to ease the reading and edition of sources by Cerfacs users. Here follows the recipe.

Creating your own DEVELOP virtual env for OpenTEA3 Tools

We are encouraging the use of a pip virtual environment. We are not using an Conda/Anaconda virtual environments. Reading our primer on pip virtual environment) is a prerequisite before attempting an installation.

Create the sources depot

Create a GITLAB folder on the system, where there is enough space (usually not the $HOME, and where data will remain (A SCRATCH disk is wiped regularly)/

> mkdir GITLAB
Clone the git repositories

On the main page of the project, under the “Clone” button, you will get the way to access to this repository using either shh or https.

*Note ; If you need to give your credential at each pull/push, the drag will stop you from updating, which is bad practice. Enter your credentials on gitlab Forge, by pasting the id_rsa.pub content in > User settings > SSH Keys, (accessible by clicking on your profile avatar).

> git clone (forge address) open-source/tiny_3d_engine.git
> git clone (forge address) open-source/arnica
> git clone (forge address) open-source/calcifer.git
> git clone (forge address) open-source/lemmings.git
> git clone (forge address) open-source/ms_thermo.git

> git clone (forge address) cfd-apps/pyavbp.git

> git clone (forge address) opentea/opentea.git
> git clone (forge address) opentea/tekigo.git
> git clone (forge address) opentea/outilsMetiersSAFRAN.git
> git clone (forge address) opentea/outils-best-safran.git
A Bash script to create a COOP-like virtual environment

You can use this bash script to create a COOP environment on any machine with an access to internet and the GITLAB repositories cloned. Feel free to edit and season to your liking.

#!/bin/bash

echo "============"
echo "VENV DEV 0.1"
echo "============"

echo -n "Venv name to create here?: "
read vname
echo -n "Absolutie path to GITLAB folder?"
read gpath


echo "Creating new venv $vname"

deactivate
python3 -m venv  $vname
source $vname/bin/activate

echo "Preinstalling large packages from PyPI"

pip install --upgrade pip

for pkg in\
    numpy\
    scipy\
    h5py\
    lxml
do
    pip uninstall $pkg
    pip install $pkg
done

cpath=($pwd)

echo "Installing pkgs in Devops mode."
for pkg in \
    h5cross\
    nobvisual\
    tiny_3d_engine\
    arnica\
    calcifer\
    ms_thermo\
    opentea\
    tekigo\
    pyavbp\
    outilsMetiersSAFRAN
do
    pip uninstall $pkg
    echo "=========================================="
    cd "$gpath/$pkg"
    python setup.py develop
done

cd $cpath

All the GITLAB packages are installed in “develop” mode. In other words, what you have on your git is what you get in your venv. This make the process of correcting/contributing to any of the tool totally straightforward.

Managing your environment

Checking what is on develop mode

Now the environment is set up. If you want to check what package is in develop mode, use :

(opentea-cerfacs-develop) > pip list
Package                       Version    Location                                      
----------------------------- ---------- ----------------------------------------------
alabaster                     0.7.12     
arnica                        1.5.4      /Users/dauptain/GITLAB/arnica/src             
astroid                       2.3.3      
attrs                         19.3.0     
Babel                         2.8.0      
bleach                        3.1.4      
calcifer                      0.0.0      
calcifer-pde                  0.0.0b0    /Users/dauptain/GITLAB/calcifer               
certifi                       2020.4.5.1 
chardet                       3.0.4      
click                         7.1.1      
commonmark                    0.9.1      
coverage                      5.0.4      
cycler                        0.10.0     
docopt                        0.6.2      
docutils                      0.16       
et-xmlfile                    1.0.1      
f90nml                        1.1.2      
flinter                       0.2.2      /Users/dauptain/GITLAB/flint/src              
future                        0.18.2     
(...)

Each time the location is given, check it is one of your gitlab repositories.

Switching version in ‘develop’ mode.

The versions in develop are now controlled by Git. Use Git to change the source you are using.

(opentea-cerfacs-develop) /GITLAB/arnica > git checkout FEATURE/Myawesomebranch 
Keep Dependencies right

As you are now overriding the versions manually with git, you could introduce errors. Use the pip checking to solve your problems

(opentea-cerfacs-develop) a > pip check
No broken requirements found.

Of course, it assumes that every package is defining well the requirements.

Real-life example: if a feature is created thanks to TWO packages updates e.g. A-0.2.3-and B-2.0.1- (B requiring A), the developers have updated pkg A FIRST, with a new minor version in the setup.py -0.3.0-, THEN B with a new version -2.1- , and a requirement such as A>=0.3<1 …

We are stressing this because strict pakage requirements and the absolute respect of open-close principle enforced through semantic versionning is your lifeline here.


When internet is not available?

Well, that is bad news, but often happen on HPC clusters. For the time being there are two options.

Use the system packages

The easiest way is to take what is already here.

> python3 -m venv --system-site-packages opentea-cerfacs-develop

With this command you are basically cloning the packages of the base environement. You are hoping here that all your dependencies are already installed on the target machine, and this is usually not the case.

Use the wheels or a tarball

Wheels are a way to move a python package from a machine to an other, provided they are compatible. A tarball is a bundle or wheels created on one machine, and installed on one other. Again, the tarball must be compatible. If you have one, install it, and follow the initial steps.

If you do not, you are probably what they call an “Expert” in charge of an installation.

  • First the source and target machines must use simular processor.
  • Then the Operatin System must be similar and equal/older on the source machine w.r. to the target machine
  • The python MAJOR and MINOR version must be equal on the source machine w.r. to the target machine.

Use the shipy tool in the devops package to create a tarball.

Machine Compatibility: worked example

For the target (no internet) machine, we found (2020, May20th) the Python 3.6.4.

Python 3.6.4 (default, Jan 11 2018, 16:45:55)

The os is a Redhat 7.6

> lsb_release -a
LSB Version:    :core-4.1-amd64:core-4.1-noarch
Distributor ID: RedHatEnterpriseServer
Description:    Red Hat Enterprise Linux Server release 7.6 (Maipo)
Release:    7.6
Codename:   Maipo

The tarball is created on the source machine, with a python 3.6.8

Python 3.6.8 (default, Apr 25 2019, 21:02:35) 

We see more bugfixes, but the minor version is the same.

The os is a CentOS

>lsb_release -a
LSB Version:    :core-4.1-amd64:core-4.1-noarch:cxx-4.1-amd64:cxx-4.1-noarch:desktop-4.1-amd64:desktop-4.1-noarch:languages-4.1-amd64:languages-4.1-noarch:printing-4.1-amd64:printing-4.1-noarch
Distributor ID: CentOS
Description:    CentOS Linux release 7.6.1810 (Core) 
Release:    7.6.1810
Codename:   Core

And you must knwo that CentOS is fully compatible with RedHat. See wikipedia on CentOS. We see here that RH 7.6 is equivalent to CentOS 7.6.1810.

Wheels
  • move the wheel on the target machine
  • load your virtual environement
  • install it `pip install blah.whl“
Tarball

Use the install.py utility inside the tarball to install you environment.

  • Deactivate your virtual environment if you have one.
  • Load the correct python3.6.4 (e.g. module load python/3.6.4)
  • Move to the tarball folder
  • Use the utility , e.g. python3 install.py --system_site_packages True --venv_name venv_opentea_safran ../../.
  • Activate the newly installed environment source ../../venv_opentea_safran/bin/activate
  • If you reached this point, take a break!

Like this post? Share on: TwitterFacebookEmail


Antoine Dauptain is a research scientist focused on computer science and engineering topics for HPC.

Keep Reading


Published

Category

Tutorials

Tags

Stay in Touch