COOP is developing several python tools. Most are open-source packages installable via PyPI from anywhere the Internet is reaching:
tiny-3-engine. Closer to Cerfacs core competencies, a limited set of tools require permissions on the Cerfacs Gitlab Forge:
oms. The recommended way to use these tools is the standard virtual environment technique, using a ‘develop’ installation to ease the reading and edition of sources by Cerfacs users. Here follows the recipe.
We are encouraging the use of a pip virtual environment. We are not using an Conda/Anaconda virtual environments. Reading our primer on pip virtual environment) is a prerequisite before attempting an installation.
GITLAB folder on the system, where there is enough space (usually not the
$HOME, and where data will remain (A SCRATCH disk is wiped regularly)/
> mkdir GITLAB
On the main page of the project, under the “Clone” button, you will get the way to access to this repository using either shh or https.
*Note ; If you need to give your credential at each pull/push, the drag will stop you from updating, which is bad practice. Enter your credentials on gitlab Forge, by pasting the
id_rsa.pub content in
> User settings > SSH Keys, (accessible by clicking on your profile avatar).
> git clone (forge address) open-source/tiny_3d_engine.git > git clone (forge address) open-source/arnica > git clone (forge address) open-source/calcifer.git > git clone (forge address) open-source/lemmings.git > git clone (forge address) open-source/ms_thermo.git > git clone (forge address) cfd-apps/pyavbp.git > git clone (forge address) opentea/opentea.git > git clone (forge address) opentea/tekigo.git > git clone (forge address) opentea/outilsMetiersSAFRAN.git > git clone (forge address) opentea/outils-best-safran.git
You can use this bash script to create a COOP environment on any machine with an access to internet and the GITLAB repositories cloned. Feel free to edit and season to your liking.
#!/bin/bash echo "============" echo "VENV DEV 0.1" echo "============" echo -n "Venv name to create here?: " read vname echo -n "Absolutie path to GITLAB folder?" read gpath echo "Creating new venv $vname" deactivate python3 -m venv $vname source $vname/bin/activate echo "Preinstalling large packages from PyPI" pip install --upgrade pip for pkg in\ numpy\ scipy\ h5py\ lxml do pip uninstall $pkg pip install $pkg done cpath=($pwd) echo "Installing pkgs in Devops mode." for pkg in \ h5cross\ nobvisual\ tiny_3d_engine\ arnica\ calcifer\ ms_thermo\ opentea\ tekigo\ pyavbp\ outilsMetiersSAFRAN do pip uninstall $pkg echo "==========================================" cd "$gpath/$pkg" python setup.py develop done cd $cpath
All the GITLAB packages are installed in “develop” mode. In other words, what you have on your git is what you get in your venv. This make the process of correcting/contributing to any of the tool totally straightforward.
Now the environment is set up. If you want to check what package is in develop mode, use :
(opentea-cerfacs-develop) > pip list Package Version Location ----------------------------- ---------- ---------------------------------------------- alabaster 0.7.12 arnica 1.5.4 /Users/dauptain/GITLAB/arnica/src astroid 2.3.3 attrs 19.3.0 Babel 2.8.0 bleach 3.1.4 calcifer 0.0.0 calcifer-pde 0.0.0b0 /Users/dauptain/GITLAB/calcifer certifi 2020.4.5.1 chardet 3.0.4 click 7.1.1 commonmark 0.9.1 coverage 5.0.4 cycler 0.10.0 docopt 0.6.2 docutils 0.16 et-xmlfile 1.0.1 f90nml 1.1.2 flinter 0.2.2 /Users/dauptain/GITLAB/flint/src future 0.18.2 (...)
Each time the location is given, check it is one of your gitlab repositories.
The versions in develop are now controlled by Git. Use Git to change the source you are using.
(opentea-cerfacs-develop) /GITLAB/arnica > git checkout FEATURE/Myawesomebranch
As you are now overriding the versions manually with git, you could introduce errors. Use the pip checking to solve your problems
(opentea-cerfacs-develop) a > pip check No broken requirements found.
Of course, it assumes that every package is defining well the requirements.
Real-life example: if a feature is created thanks to TWO packages updates e.g. A-0.2.3-and B-2.0.1- (B requiring A), the developers have updated pkg A FIRST, with a new minor version in the setup.py -0.3.0-, THEN B with a new version -2.1- , and a requirement such as A>=0.3<1 …
We are stressing this because strict pakage requirements and the absolute respect of open-close principle enforced through semantic versionning is your lifeline here.
Well, that is bad news, but often happen on HPC clusters. For the time being there are two options.
The easiest way is to take what is already here.
> python3 -m venv --system-site-packages opentea-cerfacs-develop
With this command you are basically cloning the packages of the base environement. You are hoping here that all your dependencies are already installed on the target machine, and this is usually not the case.
Wheels are a way to move a python package from a machine to an other, provided they are compatible. A tarball is a bundle or wheels created on one machine, and installed on one other. Again, the tarball must be compatible. If you have one, install it, and follow the initial steps.
If you do not, you are probably what they call an “Expert” in charge of an installation.
- First the source and target machines must use simular processor.
- Then the Operatin System must be similar and equal/older on the source machine w.r. to the target machine
- The python MAJOR and MINOR version must be equal on the source machine w.r. to the target machine.
Use the shipy tool in the
devops package to create a tarball.
For the target (no internet) machine, we found (2020, May20th) the Python 3.6.4.
Python 3.6.4 (default, Jan 11 2018, 16:45:55)
The os is a Redhat 7.6
> lsb_release -a LSB Version: :core-4.1-amd64:core-4.1-noarch Distributor ID: RedHatEnterpriseServer Description: Red Hat Enterprise Linux Server release 7.6 (Maipo) Release: 7.6 Codename: Maipo
The tarball is created on the source machine, with a python 3.6.8
Python 3.6.8 (default, Apr 25 2019, 21:02:35)
We see more bugfixes, but the minor version is the same.
The os is a CentOS
>lsb_release -a LSB Version: :core-4.1-amd64:core-4.1-noarch:cxx-4.1-amd64:cxx-4.1-noarch:desktop-4.1-amd64:desktop-4.1-noarch:languages-4.1-amd64:languages-4.1-noarch:printing-4.1-amd64:printing-4.1-noarch Distributor ID: CentOS Description: CentOS Linux release 7.6.1810 (Core) Release: 7.6.1810 Codename: Core
And you must knwo that CentOS is fully compatible with RedHat. See wikipedia on CentOS. We see here that RH 7.6 is equivalent to CentOS 7.6.1810.
- move the wheel on the target machine
- load your virtual environement
- install it `pip install blah.whl“
Use the install.py utility inside the tarball to install you environment.
- Deactivate your virtual environment if you have one.
- Load the correct python3.6.4 (e.g.
module load python/3.6.4)
- Move to the tarball folder
- Use the utility , e.g.
python3 install.py --system_site_packages True --venv_name venv_opentea_safran ../../.
- Activate the newly installed environment
- If you reached this point, take a break!