Photo Annie Spratt on Unsplash
Here is our reference on how we handle packaging at COOP. Most - but not all - of our code is done in Python. Most - but not all - of our code is Open-source.
We try to stay close to the Structuring Your Project and Packaging Python Project guidelines. Some elements come also from the Python Packaging Authority: Github sample project.
Versioning
Cactus Model
Our packages are co-developed using Git, preferably on the internal GitLab server. We follow the semantic versioning standard using the COOP-Cactus model. There is a full post on the use of semver to explain why we do it this way.
In a nutshell, in semanting versionning, package version is given by tree numbers: major revision, minor revision, and bugfix (e.g.
1.3.2). The cactus model is a single main branch for developments, because our codes are usually small (<5000 lines)
Changelog
Keeping the CHANGELOG is mandatory. Here we stick to the reference keep a changelog.
Open-source mirrored packages
Our open-source packages are mirrored in Gitlab. The option to mirror a repo can be found under Settings -> Repository -> Mirroring repositories (internal note: ask a senior to create the gitlab mirror to avoid future credentials issues). Mirroring a repo allows external users to easily access the code and to host the documentation in Read the Docs.
Repository Structure
The structure looks like:
.
├── LICENSE
├── MANIFEST.in
├── README.md
├── CHANGELOG.md
├── .gitlab-ci.yml
├── .readthedocs.yaml
├── Dockerfile
├── Makefile
├── docs
│ ├── _build <-- generated by Sphinx
│ ├── source
│ │ ├── conf.py
│ │ ├── index.rst
│ │ ├── redirect.rst
│ │ ├── api
│ │ │ └── (...) <-- generated by Sphinx
│ │ ├── tutorials
│ │ │ └── *.rst, *.md, *.ipynb
│ │ ├── howto
│ │ │ └── *.rst, *.md, *.ipynb
│ │ └── explanations
│ │ └── *.rst, *.md, *.ipynb
│ ├── Makefile
│ ├── make.bat
├── setup.py
├── setup.cfg
├── src
│ ├── nameofpkg
│ │ ├── __init__.py
│ │ ├── cli.py
│ │ └── *.py
└── tests
├── conftest.py
├── unit
│ └── test_*.py
└── func
└── test_*.py
Setup
We use setup.cfg instead of a dynamic setup.py approach. The main reason supporting this choice is the declarative approach of setup.cfg. It assumes the existence of MANIFEST.in (controlled via include_package_data = True). The alternative is to keep include_package_data = False and add the following entry (in this example, the yaml files are shipped alongside the code):
[options.package_data]
* = *.yaml
MANIFEST.in tells what should be shipped alongside the code (e.g. yaml files). Typically it only contains one line (graft src).
We do not use a requirements.txt, since the setup.cfg deals with them. This not only applies to the package install requirements, but also to docs and test requirements, which can be set via options.extra_requires (in the installation process, the requirements to install can be specified via square brackets e.g. pip install opentea [docs]).
Git ignore
The .gitignore is typically:
*.pyc
*.pvsm
*.png
*.pdf
*.h5
*.xmf
*.swp
*.idea*
.coverage
.eggs
.DS_Store
arnica.egg-info
arnica.egg*
build
dist
docs/_build/
Indeed we do not store builds either for the package or its documentation.
src/nameofpkg/ folder
We use /src/nameofpkg/ folder, even though this solution diverges from the Python guide and many others sources. Besides being backed up by Python Packaging Authority sample project, it simplifies a lot many CI and shipping operations. A simple example is the running of tests from the repo’s root: without this structure, is pytest using directly the source code or an installed version? In other words, this structure creates “import parity”.
README,CHANGELOG, LICENSE
The README is compulsory. It describes the objective of the package, the way to install it, a very simple usage to illustrate, some acknowledgements with, authors and contributors and funding. It can be a md or rst file (both are well rendered by gitlab). If the README is a rst file, then it can be used directly in the docs without a link file (via .. include:: ../../README.rst).
The CHANGELOG is compulsory, and comply to the keep-a-changelog standard.
A LICENSE is compulsory. It is usually a LGPL for international projects and Cecill-B for french projects (a full post on the topic here).
Version number
The version is accessible as expected in a Python package: package_name.__version__. This is accomplished by creating a __version__ variable in the main __init__.py file.
Dockerfile (Optional)
Most of our packages do not require a Dockerfile as they are simply installed via pip. On the other hand, e.g. pyhip relies in docker to simplify its portability, as it is mainly written in C.
Makefile (Optional)
A typical Makefile looks like:
doc:
rm -r docs/_build
cd docs && make html
test:
pytest --cov=nobvisual
lint:
pylint src/nobvisual
wheel:
rm -rf build
rm -rf dist
python setup.py sdist bdist_wheel
upload_test:
twine upload --repository-url https://test.pypi.org/legacy/ dist/*
upload:
twine upload dist/*
Third party conf. file (optional)
.gitlab-ci.yml and .readthedocs.yaml contain configurations for GitLab CI pipelines (which are triggered every commit) and Read the Docs, respectively.
Testing
We use pytest for both unitary and functional tests. Tests are there to reduce the charge of refactoring! (read more on how much testing is too much?).
In medio stat virtus: Having a good coverage is good, but rushing to 100% can backfire nastily. Too much redundant, functional tests without a proper first layer of unitary tests will augment the workload during refactoring.
Linting
We apply the PEP 8 standard, with a minimum of 9 over 10. When we are tired, we use black to get rid of the small pylint stuff.
In medio stat virtus: Having a good linting is good, but rushing to 10 can backfire nastily. We had examples such as
tekigo/refinewhere the API became unreadable to stay below 5 args. The refactoring was hard.
Documentation
The code is documented using docstrings compatible with PEP 8. The documentation is generated with sphinx. A full post on documenting Python code here.
Do not add internal links to the docs (or README file), as they will not work for external users. This is specially relevant for links to images: the images will not render in e.g. PyPI.
A common (but dirty) way to avoid links is to ask, in conf.py, to copy README.md in docs at the documentation generation.
Doc structure
The docs structure contains a source folder with configuration and content files (rst, md, or ipynb), a _build folder containing the built html files (this folder should not be committed, as it is generated automatically), and sphinx makefiles (Makefile and make.bat), which are an output of sphinx-quickstart. changelog_link.md and readme_link.md allow to add the CHANGELOG and README contents directly to the docs without the need of having duplicated files (this requires myst_parser). The folder API/_generated should not be committed either, as it is also the output of an automatic process.
Add a package to Read the Docs
To add a project to Read the Docs, simply go to their website and import the project (internal note: ask a senior the credentials to COOPTeam-CERFACS account).
A typical configuration file looks like this (more details here):
# Required
version: 2
# Build documentation in the docs/ directory with Sphinx
sphinx:
configuration: docs/source/conf.py
build:
apt_packages:
- pandoc
# Optionally set the version of Python and requirements required to build your docs
python:
version: 3.8
install:
- method: pip
path: .
extra_requirements:
- docs
Installation
Installation differ between open-source and limited-diffusion packages.
Open-source packages
These packages can be used as stand-alone.
We upload our packages on PyPI. Since most of these pages is set from the setup.cfg, we must carefully choose our keywords, authors, website, and so on.
A typical setup.cfgis the following (the docs of setuptools):
[metadata]
name = OpenTEA
version = attr: opentea.__version__
author = CoopTeam-CERFACS
author_email = coop@cerfacs.com
description = Graphical User Interface engine based upon Schema
long_description = file: README.md
long_description_content_type = text/markdown
url = https://gitlab.com/cerfacs/opentea3
project_urls =
Homepage = https://gitlab.com/cerfacs/opentea3
Documentation = https://opentea.readthedocs.io/en/latest/
Bug Tracker = https://gitlab.com/cerfacs/opentea3/-/issues
classifiers =
Programming Language :: Python :: 3.7
Programming Language :: Python :: 3.8
Programming Language :: Python :: 3.9
License :: CeCILL-B Free Software License Agreement (CECILL-B)
keywords =
GUI
SCHEMA
Tkinter
[options]
package_dir =
= src
packages = find:
python_requires = >= 3.7
install_requires =
numpy>=1.16.2
h5py>=2.6.0
jsonschema
markdown
Pillow>=5.4.1
PyYAML>=3.13
nob>=0.4.1
nobvisual
requests
click
tiny_3d_engine>=0.2
zip_safe = False
include_package_data = True
[options.packages.find]
where = src
[options.entry_points]
console_scripts =
opentea3 = opentea.cli:main_cli
[options.extras_require]
tests=
pytest
docs =
myst-parser
sphinx_rtd_theme
Pushing to PyPI
For open-source packages you need to put to PyPI:
- Check that all tests run locally and linter gives a satisfactory score:
> make testand> make lint. - Commit all your changes, push to nitrox, and check that the CI passes.
- Update the
CHANGELOGand increment the version number. Commit this version update, and tag the commit with a tag name exactly matching the version number, and push all (commit + tag) to nitrox (needs to be improved). - Build distribution:
> make wheel - (Optional) Upload to
TestPyPI(must be done ifsetup.cfgchanges significantly):> make upload_test. - Upload to
PyPI:> make upload.
Limited diffusion packages
Refer to this page on How to install coop packages venvs.
Continuous integration
We use gitlab-ci to automate the continuous integration. Usually our pipelines include three stages: testing, linting and local documentation (or Read the Docs hosted documentation if open-source). A typical .gitlab-ci.yml configuration is the following:
stages:
- all
before_script:
- python -V
- pip install pip --upgrade
- pip install setuptools --upgrade
- pip install .[docs, tests]
test:
stage: all
image: devops
tags:
- docker.elmer.nitrox
script:
- pytest --cov kokiy
lint:
stage: all
image: devops
tags:
- docker.elmer.nitrox
script:
- pip install anybadge
- pylint kokiy --output-format=text --exit-zero | tee pylint.txt
- score=$(sed -n 's/^Your code has been rated at \([-0-9.]*\)\/.*/\1/p' pylint.txt)
- anybadge -l pylint -o --file=pylint.svg -v $score 2=red 4=orange 8=yellow 10=green
artifacts:
paths:
- pylint.svg
pages:
stage: all
image: devops
tags:
- docker.elmer.nitrox
script:
- cd docs
- make html
- mv _build/html/ ../public/
artifacts:
paths:
- public
only:
- master
Do not forget to remove pages for open-source packages configurations.
Add badges
Our repos have typical four badges: pipeline, coverage, pylint, and docs. These are useful to quickly get information about the repo. You can set them out by going to Settings -> General -> Badges.
With exception of docs, all the badges are set with link https://nitrox.cerfacs.fr/%{project_path}. The corresponding images urls are https://nitrox.cerfacs.fr/%{project_path}/badges/master/pipeline.svg, https://nitrox.cerfacs.fr/%{project_path}/badges/%{default_branch}/coverage.svg, and https://nitrox.cerfacs.fr/%{project_path}/-/jobs/artifacts/%{default_branch}/raw/pylint.svg?job=lint for pipeline, coverage, and pylint, respectively.
The coverage information is gathered in the pipeline log by a regular expression that can be set in Settings -> CI/CD -> Test coverage parsing.
docs is set only to open-source packages and is set with both link and badge image url https://readthedocs.org/projects/%{project_path}/badge/?version=latest.
CI troubleshooting
Ordered by frequency, and solution:
- linting Try to fix the code, and not the
.pylintrc. - importing If you add a new package in your
setup.py, the pipeline
Command Line Interface
We usually have a fast-start CLI in our packages. It is done with click.
The idea is to type the name of the package in the terminal to discover all the implemented commands. Read our dedicated post on CLI
Concluding remarks
Our standards are continuously improving (sometimes faster that our blog posts updates), so keep sure you peek into our repositories to get the most up-to-date information.