Photo John Barkiple on Unsplash

COOP is developing several python tools. Most are open-source packages installable via PyPI from anywhere the Internet is reaching: arnica, calcifer, h5cross, satis, nob, opentea, nobvisual, tiny-3-engine. Closer to Cerfacs core competencies, a limited set of tools require permissions on the Cerfacs Gitlab Forge: pyavbp, tekigo, lemmings, oms. The recommended way to use these tools is the ubiquitous virtual environment technique, using a ‘develop’ installation to ease the reading and edition of sources. Here follows the recipe.

Creating your own DEVELOP virtual env for OpenTEA3 Tools

We are encouraging the use of a pip virtual environment. Reading our primer on pip virtual environment) is a prerequisite before attempting an installation.

Create the sources depot

Create a GITLAB folder on the system, where there is enough space (usually not the $HOME, and where data will remain (A SCRATCH disk is wiped regularly)/

> mkdir GITLAB
Clone the git repositories

On the main page of the project, under the “Clone” button, you will get the way to access to this repository using either shh or https.

*Note ; If you need to give your credential at each pull/push, the drag will stop you from updating, which is bad practice. Enter your credentials on gitlab Forge, by pasting the content in > User settings > SSH Keys, (accessible by clicking on your profile avatar).

> git clone (forge address) cfd-apps/pyavbp.git 
> git clone (forge address) opentea/tekigo.git
A Bash script to create a COOP-like virtual environment

You can use this formula to re-create a COOP environment on any machine with an access to internet and the GITLAB repositories cloned (Kraken and Nemo both feature the python/3.7.7 module). Feel free to edit and season to your liking. This script should be stored in your virtual environments folder, usually something like $HOME/Python_venvs/.



echo "Rebooting venv named $VENV_NAME"

#clean the venv
rm -rf $VENV_NAME
#module load $PY_MODULE

# Pure pip
python3 -m venv --system-site-packages $VENV_NAME
source $VENV_NAME/bin/activate
# if the python is a COnda env
# conda create -n opentea_dev --clone base
# conda activate opentea_dev

python -m pip install --upgrade pip
python -m pip install --upgrade setuptools

# Add here the name of pkages you want to instal in develop mode
for repo in "pyavbp" "outilsMetiersSAFRAN" "tekigo"
    python -m pip install -e $LOCAL_GITLAB/$repo

All the GITLAB packages (here pyavbp, outilsMetiersSAFRAN, tekigo) are installed in “develop” mode. In other words, what you have on your git is what you get in your venv. This make the process of correcting/contributing to any of the tool totally straightforward.

Managing your environment

Checking what is on develop mode

Now the environment is set up. If you want to check what package is in develop mode, use :

(opentea-cerfacs-develop) > pip list
Package                       Version    Location                                      
----------------------------- ---------- ----------------------------------------------
alabaster                     0.7.12     
arnica                        1.5.4      /Users/dauptain/GITLAB/arnica/src             
astroid                       2.3.3      
attrs                         19.3.0     
Babel                         2.8.0      
bleach                        3.1.4      
calcifer                      0.0.0      
calcifer-pde                  0.0.0b0    /Users/dauptain/GITLAB/calcifer               
certifi                       2020.4.5.1 
chardet                       3.0.4      
click                         7.1.1      
commonmark                    0.9.1      
coverage                      5.0.4      
cycler                        0.10.0     
docopt                        0.6.2      
docutils                      0.16       
et-xmlfile                    1.0.1      
f90nml                        1.1.2      
flinter                       0.2.2      /Users/dauptain/GITLAB/flint/src              
future                        0.18.2     

Each time the location is given, check it is one of your gitlab repositories.

Switching version in ‘develop’ mode.

The versions in develop are now controlled by Git. Use Git to change the source you are using. Then ask python to update the versions.

(opentea-cerfacs-develop) /GITLAB/arnica > git checkout FEATURE/Myawesomebranch 
(opentea-cerfacs-develop) /GITLAB/arnica > python develop
Keep Dependencies right

As you are now overriding the versions manually with git, you could introduce errors. Use the pip checking to solve your problems

(opentea-cerfacs-develop) a > pip check
No broken requirements found.

Of course, it assumes that every package is defining well the requirements.

Real-life example: if a feature is created thanks to TWO packages updates e.g. A-0.2.3-and B-2.0.1- (B requiring A), the developers have updated pkg A FIRST, with a new minor version in the -0.3.0-, THEN B with a new version -2.1- , and a requirement such as A>=0.3<1 …

We are stressing this because strict pakage requirements and the absolute respect of open-close principle enforced through semantic versionning is your lifeline here.

Check your venv

One your venv is all set, list your packages are stored using the -v verbose output.

>python -m pip list -v

xarray                        0.17.0                 /awesomecomputer/products2/python3-3.7.5/Atos_7__x86_64/intel--                                                        
xesmf                         0.2.1                  /awesomecomputer/products2/python3-3.7.5/Atos_7__x86_64/intel--                                                        
xgboost                       1.5.1                  /awesomecomputer/python3-3.7.5/Atos_7__x86_64/intel--                                                        
yamio                         0.1.3                  /awesomecomputer/alphateam/username/venvs/venv_omg_37/lib/python3.7/site-packages                           

We got issues with some venv installations where some packeges were neither in the venv directory, nor in the global env directory. Indeed the vanv was build linked with a local hidden path, probably due to a caching of pip (see pip caching). If you end up in the same situation, consider the --no-cache-dir in your next pip installation.

When internet is not available?

Well, that is bad news, but often happen on HPC clusters. Most of the packages you want are already on the system. Using --system-site-packages when creating the venv will ensure you copy the pre-installed packages.

If some of your own packages are shipped with a wheel, simply build the wheel on a similar system (next section) and move the wheels into one of your folders my/awesome/GITLAB.

The formula to create the v-env changes only at the end:




echo "Rebooting venv named $VENV_NAME"

#clean the venv
rm -rf $VENV_NAME
module load $PY_MODULE
python3 -m venv --system-site-packages $VENV_NAME
source $VENV_NAME/bin/activate

# Add here the name of packages you want to instal in develop mode
for repo in "pyavbp" "outilsMetiersSAFRAN"
    python -m pip install -e $LOCAL_GITLAB/$repo --no-index --find-links $LOCAL_PIPY_PATH

Note the pip -e (editor mode) is equivalent to python develop. However using pip allows the options --no-index --find-links $LOCAL_PIPY_PATH necessary to fetch the wheels in a local folder.

Build your own wheel for a distant machine

When your dev must be provided as a wheel , and not as sources (no gitlab connections), you must build the wheel on an equivalent machine. Here is an ewample:

For the target (no internet) machine, we found (2020, May20th) the Python 3.6.4.

Python 3.6.4 (default, Jan 11 2018, 16:45:55)

Note: 3.6.4 is read as Major:3, Minor: 6, Bugfix: 4

The os is a Redhat 7.6

> lsb_release -a
LSB Version:    :core-4.1-amd64:core-4.1-noarch
Distributor ID: RedHatEnterpriseServer
Description:    Red Hat Enterprise Linux Server release 7.6 (Maipo)
Release:    7.6
Codename:   Maipo
Python 3.6.8 (default, Apr 25 2019, 21:02:35) 

We see more bugfixes, but the minor version is the same.

The os is a CentOS

>lsb_release -a
LSB Version:    :core-4.1-amd64:core-4.1-noarch:cxx-4.1-amd64:cxx-4.1-noarch:desktop-4.1-amd64:desktop-4.1-noarch:languages-4.1-amd64:languages-4.1-noarch:printing-4.1-amd64:printing-4.1-noarch
Distributor ID: CentOS
Description:    CentOS Linux release 7.6.1810 (Core) 
Release:    7.6.1810
Codename:   Core

And you must know that CentOS is fully compatible with RedHat. See wikipedia on CentOS. We see here that RH 7.6 is equivalent to CentOS 7.6.1810.

If you do not have the exact version of python (same major and minor , different bugfix), build your wheel with a version lower than the target machine, but with the same minor.

building machine target machine result
3.7.5 3.7.5 OK
3.7.4 3.7.5 OK
3.7.6 3.7.5 FAIL
3.6.5 3.7.5 FAIL
4.0.1 3.7.5 FAIL

Watch for vulnerabilities

Since you are moving your tools via wheels, make sure you are not moving malicious code by accident. Therefore, check your packages for vulnerability on a regular basis. There are several up-to-date packages to do this in one or two lines, such as safety.

Like this post? Share on: TwitterFacebookEmail

Antoine Dauptain is a research scientist focused on computer science and engineering topics for HPC.

Keep Reading





Stay in Touch