Photo John Barkiple on Unsplash
COOP is developing several python tools. Most are open-source packages installable via PyPI from anywhere the Internet is reaching: arnica, calcifer, h5cross, satis, nob, opentea, nobvisual, tiny-3-engine. Closer to Cerfacs core competencies, a limited set of tools require permissions on the Cerfacs Gitlab Forge: pyavbp, tekigo, lemmings, oms. The recommended way to use these tools is the ubiquitous virtual environment technique, using a ‘develop’ installation to ease the reading and edition of sources. Here follows the recipe.
Creating your own DEVELOP virtual env for OpenTEA3 Tools
We are encouraging the use of a pip virtual environment. Reading our primer on pip virtual environment) is a prerequisite before attempting an installation.
Create the sources depot
Create a GITLAB folder on the system, where there is enough space (usually not the $HOME, and where data will remain (A SCRATCH disk is wiped regularly)/
> mkdir GITLAB
Clone the git repositories
On the main page of the project, under the “Clone” button, you will get the way to access to this repository using either shh or https.
*Note ; If you need to give your credential at each pull/push, the drag will stop you from updating, which is bad practice. Enter your credentials on gitlab Forge, by pasting the id_rsa.pub content in > User settings > SSH Keys, (accessible by clicking on your profile avatar).
> git clone (forge address) cfd-apps/pyavbp.git
> git clone (forge address) opentea/tekigo.git
A Bash script to create a COOP-like virtual environment
You can use this formula to re-create a COOP environment on any machine with an access to internet and the GITLAB repositories cloned (Kraken and Nemo both feature the python/3.7.7 module). Feel free to edit and season to your liking. This script should be stored in your virtual environments folder, usually something like $HOME/Python_venvs/.
#!/bin/bash
# NEMO
VENV_NAME=venv_nemo_377
#PY_MODULE=python/3.7.7
LOCAL_GITLAB="/data/home/cfd/dauptain/GITLAB"
echo "Rebooting venv named $VENV_NAME"
#clean the venv
deactivate
rm -rf $VENV_NAME
#module load $PY_MODULE
# Pure pip
python3 -m venv --system-site-packages $VENV_NAME
source $VENV_NAME/bin/activate
# if the python is a COnda env
# conda create -n opentea_dev --clone base
# conda activate opentea_dev
python -m pip install --upgrade pip
python -m pip install --upgrade setuptools
# Add here the name of pkages you want to instal in develop mode
for repo in "pyavbp" "outilsMetiersSAFRAN" "tekigo"
do
python -m pip install -e $LOCAL_GITLAB/$repo
done
All the GITLAB packages (here pyavbp, outilsMetiersSAFRAN, tekigo) are installed in “develop” mode. In other words, what you have on your git is what you get in your venv. This make the process of correcting/contributing to any of the tool totally straightforward.
Managing your environment
Checking what is on develop mode
Now the environment is set up. If you want to check what package is in develop mode, use :
(opentea-cerfacs-develop) > pip list
Package Version Location
----------------------------- ---------- ----------------------------------------------
alabaster 0.7.12
arnica 1.5.4 /Users/dauptain/GITLAB/arnica/src
astroid 2.3.3
attrs 19.3.0
Babel 2.8.0
bleach 3.1.4
calcifer 0.0.0
calcifer-pde 0.0.0b0 /Users/dauptain/GITLAB/calcifer
certifi 2020.4.5.1
chardet 3.0.4
click 7.1.1
commonmark 0.9.1
coverage 5.0.4
cycler 0.10.0
docopt 0.6.2
docutils 0.16
et-xmlfile 1.0.1
f90nml 1.1.2
flinter 0.2.2 /Users/dauptain/GITLAB/flint/src
future 0.18.2
(...)
Each time the location is given, check it is one of your gitlab repositories.
Switching version in ‘develop’ mode.
The versions in develop are now controlled by Git. Use Git to change the source you are using. Then ask python to update the versions.
(opentea-cerfacs-develop) /GITLAB/arnica > git checkout FEATURE/Myawesomebranch
(opentea-cerfacs-develop) /GITLAB/arnica > python setup.py develop
Keep Dependencies right
As you are now overriding the versions manually with git, you could introduce errors. Use the pip checking to solve your problems
(opentea-cerfacs-develop) a > pip check
No broken requirements found.
Of course, it assumes that every package is defining well the requirements.
Real-life example: if a feature is created thanks to TWO packages updates e.g. A-0.2.3-and B-2.0.1- (B requiring A), the developers have updated pkg A FIRST, with a new minor version in the setup.py -0.3.0-, THEN B with a new version -2.1- , and a requirement such as A>=0.3<1 …
We are stressing this because strict pakage requirements and the absolute respect of open-close principle enforced through semantic versionning is your lifeline here.
Check your venv
One your venv is all set, list your packages are stored using the -v verbose output.
>python -m pip list -v
xarray 0.17.0 /awesomecomputer/products2/python3-3.7.5/Atos_7__x86_64/intel--19.0.5.281__openmpi--4.0.1/default/lib/python3.7/site-packages
xesmf 0.2.1 /awesomecomputer/products2/python3-3.7.5/Atos_7__x86_64/intel--19.0.5.281__openmpi--4.0.1/default/lib/python3.7/site-packages
xgboost 1.5.1 /awesomecomputer/python3-3.7.5/Atos_7__x86_64/intel--19.0.5.281__openmpi--4.0.1/default/lib/python3.7/site-packages
yamio 0.1.3 /awesomecomputer/alphateam/username/venvs/venv_omg_37/lib/python3.7/site-packages
We got issues with some venv installations where some packeges were neither in the venv directory, nor in the global env directory. Indeed the vanv was build linked with a local hidden path, probably due to a caching of pip (see pip caching). If you end up in the same situation, consider the --no-cache-dir in your next pip installation.
When internet is not available?
Well, that is bad news, but often happen on HPC clusters.
Most of the packages you want are already on the system.
Using --system-site-packages when creating the venv will ensure you copy the pre-installed packages.
If some of your own packages are shipped with a wheel, simply build the wheel on a similar system (next section) and move the wheels into one of your folders my/awesome/GITLAB.
The formula to create the v-env changes only at the end:
#!/bin/bash
#IRENE
VENV_NAME=venv_37
PY_MODULE=python3/3.7.5
LOCAL_PIPY_PATH="my/awesome/wheels"
LOCAL_GITLAB="my/awesome/GITLAB"
echo "Rebooting venv named $VENV_NAME"
#clean the venv
deactivate
rm -rf $VENV_NAME
module load $PY_MODULE
python3 -m venv --system-site-packages $VENV_NAME
source $VENV_NAME/bin/activate
# Add here the name of packages you want to instal in develop mode
for repo in "pyavbp" "outilsMetiersSAFRAN"
do
python -m pip install -e $LOCAL_GITLAB/$repo --no-index --find-links $LOCAL_PIPY_PATH
done
Note the pip -e (editor mode) is equivalent to python setupy.py develop.
However using pip allows the options --no-index --find-links $LOCAL_PIPY_PATH necessary to fetch the wheels in a local folder.
Build your own wheel for a distant machine
When your dev must be provided as a wheel , and not as sources (no gitlab connections), you must build the wheel on an equivalent machine. Here is an ewample:
For the target (no internet) machine, we found (2020, May20th) the Python 3.6.4.
Python 3.6.4 (default, Jan 11 2018, 16:45:55)
Note: 3.6.4 is read as Major:3, Minor: 6, Bugfix: 4
The os is a Redhat 7.6
> lsb_release -a
LSB Version: :core-4.1-amd64:core-4.1-noarch
Distributor ID: RedHatEnterpriseServer
Description: Red Hat Enterprise Linux Server release 7.6 (Maipo)
Release: 7.6
Codename: Maipo
Python 3.6.8 (default, Apr 25 2019, 21:02:35)
We see more bugfixes, but the minor version is the same.
The os is a CentOS
>lsb_release -a
LSB Version: :core-4.1-amd64:core-4.1-noarch:cxx-4.1-amd64:cxx-4.1-noarch:desktop-4.1-amd64:desktop-4.1-noarch:languages-4.1-amd64:languages-4.1-noarch:printing-4.1-amd64:printing-4.1-noarch
Distributor ID: CentOS
Description: CentOS Linux release 7.6.1810 (Core)
Release: 7.6.1810
Codename: Core
And you must know that CentOS is fully compatible with RedHat. See wikipedia on CentOS. We see here that RH 7.6 is equivalent to CentOS 7.6.1810.
If you do not have the exact version of python (same major and minor , different bugfix), build your wheel with a version lower than the target machine, but with the same minor.
| building machine | target machine | result |
|---|---|---|
| 3.7.5 | 3.7.5 | OK |
| 3.7.4 | 3.7.5 | OK |
| 3.7.6 | 3.7.5 | FAIL |
| 3.6.5 | 3.7.5 | FAIL |
| 4.0.1 | 3.7.5 | FAIL |
Watch for vulnerabilities
Since you are moving your tools via wheels, make sure you are not moving malicious code by accident. Therefore, check your packages for vulnerability on a regular basis. There are several up-to-date packages to do this in one or two lines, such as safety.