standard

Photo Annie Spratt on Unsplash

Here is our reference on how we handle packaging at COOP. Most - but not all - of our code is done in Python. Most - but not all - of our code is Open-source.

We try to stay close to the Structuring Your Project and Packaging Python Project guidelines. Some elements come also from the Python Packaging Authority: Github sample project.

Versioning

Cactus Model

Our packages are co-developed using Git, preferably on the internal GitLab server. We follow the semantic versioning standard using the COOP-Cactus model. There is a full post on the use of semver to explain why we do it this way.

In a nutshell, in semanting versionning, package version is given by tree numbers: major revision, minor revision, and bugfix (e.g. 1.3.2). The cactus model is a single main branch for developments, because our codes are usually small (<5000 lines)

Changelog

Keeping the CHANGELOG is mandatory. Here we stick to the reference keep a changelog.

Open-source mirrored packages

Our open-source packages are mirrored in Gitlab. The option to mirror a repo can be found under Settings -> Repository -> Mirroring repositories (internal note: ask a senior to create the gitlab mirror to avoid future credentials issues). Mirroring a repo allows external users to easily access the code and to host the documentation in Read the Docs.

Repository Structure

The structure looks like:

.
├── LICENSE
├── MANIFEST.in
├── README.md
├── CHANGELOG.md
├── .gitlab-ci.yml
├── .readthedocs.yaml
├── Dockerfile
├── Makefile
├── docs
   ├── _build <-- generated by Sphinx
   ├── source
      ├── conf.py
      ├── index.rst
      ├── redirect.rst
      ├── api
         └── (...) <-- generated by Sphinx
      ├── tutorials
         └── *.rst, *.md, *.ipynb
      ├── howto
         └── *.rst, *.md, *.ipynb
      └── explanations
          └── *.rst, *.md, *.ipynb
   ├── Makefile
   ├── make.bat
├── setup.py
├── setup.cfg
├── src
  ├── nameofpkg
    ├── __init__.py
    ├── cli.py
    └── *.py
└── tests
    ├── conftest.py
    ├── unit
      └── test_*.py
    └── func
       └── test_*.py
Setup

We use setup.cfg instead of a dynamic setup.py approach. The main reason supporting this choice is the declarative approach of setup.cfg. It assumes the existence of MANIFEST.in (controlled via include_package_data = True). The alternative is to keep include_package_data = False and add the following entry (in this example, the yaml files are shipped alongside the code):

[options.package_data]
* = *.yaml

MANIFEST.in tells what should be shipped alongside the code (e.g. yaml files). Typically it only contains one line (graft src).

We do not use a requirements.txt, since the setup.cfg deals with them. This not only applies to the package install requirements, but also to docs and test requirements, which can be set via options.extra_requires (in the installation process, the requirements to install can be specified via square brackets e.g. pip install opentea [docs]).

Git ignore

The .gitignore is typically:

*.pyc
*.pvsm
*.png
*.pdf
*.h5
*.xmf
*.swp
*.idea*
.coverage
.eggs
.DS_Store
arnica.egg-info
arnica.egg*
build
dist
docs/_build/

Indeed we do not store builds either for the package or its documentation.

src/nameofpkg/ folder

We use /src/nameofpkg/ folder, even though this solution diverges from the Python guide and many others sources. Besides being backed up by Python Packaging Authority sample project, it simplifies a lot many CI and shipping operations. A simple example is the running of tests from the repo’s root: without this structure, is pytest using directly the source code or an installed version? In other words, this structure creates “import parity”.

README,CHANGELOG, LICENSE

The README is compulsory. It describes the objective of the package, the way to install it, a very simple usage to illustrate, some acknowledgements with, authors and contributors and funding. It can be a md or rst file (both are well rendered by gitlab). If the README is a rst file, then it can be used directly in the docs without a link file (via .. include:: ../../README.rst).

The CHANGELOG is compulsory, and comply to the keep-a-changelog standard.

A LICENSE is compulsory. It is usually a LGPL for international projects and Cecill-B for french projects (a full post on the topic here).

Version number

The version is accessible as expected in a Python package: package_name.__version__. This is accomplished by creating a __version__ variable in the main __init__.py file.

Dockerfile (Optional)

Most of our packages do not require a Dockerfile as they are simply installed via pip. On the other hand, e.g. pyhip relies in docker to simplify its portability, as it is mainly written in C.

Makefile (Optional)

A typical Makefile looks like:

doc:
    rm -r docs/_build
    cd docs && make html

test:
    pytest --cov=nobvisual

lint:
    pylint src/nobvisual

wheel:
    rm -rf build
    rm -rf dist
    python setup.py sdist bdist_wheel

upload_test:
    twine upload --repository-url https://test.pypi.org/legacy/ dist/*

upload:
    twine upload dist/*
Third party conf. file (optional)

.gitlab-ci.yml and .readthedocs.yaml contain configurations for GitLab CI pipelines (which are triggered every commit) and Read the Docs, respectively.

Testing

We use pytest for both unitary and functional tests. Tests are there to reduce the charge of refactoring! (read more on how much testing is too much?).

In medio stat virtus: Having a good coverage is good, but rushing to 100% can backfire nastily. Too much redundant, functional tests without a proper first layer of unitary tests will augment the workload during refactoring.

Linting

We apply the PEP 8 standard, with a minimum of 9 over 10. When we are tired, we use black to get rid of the small pylint stuff.

In medio stat virtus: Having a good linting is good, but rushing to 10 can backfire nastily. We had examples such as tekigo/refine where the API became unreadable to stay below 5 args. The refactoring was hard.

Documentation

The code is documented using docstrings compatible with PEP 8. The documentation is generated with sphinx. A full post on documenting Python code here.

Do not add internal links to the docs (or README file), as they will not work for external users. This is specially relevant for links to images: the images will not render in e.g. PyPI.

A common (but dirty) way to avoid links is to ask, in conf.py, to copy README.md in docs at the documentation generation.

Doc structure

The docs structure contains a source folder with configuration and content files (rst, md, or ipynb), a _build folder containing the built html files (this folder should not be committed, as it is generated automatically), and sphinx makefiles (Makefile and make.bat), which are an output of sphinx-quickstart. changelog_link.md and readme_link.md allow to add the CHANGELOG and README contents directly to the docs without the need of having duplicated files (this requires myst_parser). The folder API/_generated should not be committed either, as it is also the output of an automatic process.

Add a package to Read the Docs

To add a project to Read the Docs, simply go to their website and import the project (internal note: ask a senior the credentials to COOPTeam-CERFACS account).

A typical configuration file looks like this (more details here):

# Required
version: 2

# Build documentation in the docs/ directory with Sphinx
sphinx:
   configuration: docs/source/conf.py

build:
  apt_packages:
    - pandoc

# Optionally set the version of Python and requirements required to build your docs
python:
   version: 3.8
   install:
    - method: pip
      path: .
      extra_requirements:
        - docs

Installation

Installation differ between open-source and limited-diffusion packages.

Open-source packages

These packages can be used as stand-alone.

We upload our packages on PyPI. Since most of these pages is set from the setup.cfg, we must carefully choose our keywords, authors, website, and so on.

A typical setup.cfgis the following (the docs of setuptools):

[metadata]
name = OpenTEA
version = attr: opentea.__version__
author = CoopTeam-CERFACS
author_email = coop@cerfacs.com
description = Graphical User Interface engine based upon Schema
long_description = file: README.md
long_description_content_type = text/markdown
url = https://gitlab.com/cerfacs/opentea3
project_urls =
    Homepage = https://gitlab.com/cerfacs/opentea3
    Documentation = https://opentea.readthedocs.io/en/latest/
    Bug Tracker = https://gitlab.com/cerfacs/opentea3/-/issues
classifiers =
    Programming Language :: Python :: 3.7
    Programming Language :: Python :: 3.8
    Programming Language :: Python :: 3.9
    License :: CeCILL-B Free Software License Agreement (CECILL-B)
keywords =
    GUI
    SCHEMA
    Tkinter

[options]
package_dir =
    = src
packages = find:
python_requires = >= 3.7
install_requires =
    numpy>=1.16.2
    h5py>=2.6.0
    jsonschema
    markdown
    Pillow>=5.4.1
    PyYAML>=3.13
    nob>=0.4.1
    nobvisual
    requests
    click
    tiny_3d_engine>=0.2
zip_safe = False
include_package_data = True

[options.packages.find]
where = src

[options.entry_points]
console_scripts =
    opentea3 = opentea.cli:main_cli

[options.extras_require]
tests=
    pytest
docs =
    myst-parser
    sphinx_rtd_theme
Pushing to PyPI

For open-source packages you need to put to PyPI:

  1. Check that all tests run locally and linter gives a satisfactory score: > make testand > make lint.
  2. Commit all your changes, push to nitrox, and check that the CI passes.
  3. Update the CHANGELOG and increment the version number. Commit this version update, and tag the commit with a tag name exactly matching the version number, and push all (commit + tag) to nitrox (needs to be improved).
  4. Build distribution: > make wheel
  5. (Optional) Upload to TestPyPI (must be done if setup.cfg changes significantly): > make upload_test.
  6. Upload to PyPI: > make upload.
Limited diffusion packages

Refer to this page on How to install coop packages venvs.

Continuous integration

We use gitlab-ci to automate the continuous integration. Usually our pipelines include three stages: testing, linting and local documentation (or Read the Docs hosted documentation if open-source). A typical .gitlab-ci.yml configuration is the following:

stages:
    - all

before_script:
    - python -V
    - pip install pip --upgrade
    - pip install setuptools --upgrade
    - pip install .[docs, tests]

test:
    stage: all
    image: devops
    tags:
        - docker.elmer.nitrox
    script:
        - pytest --cov kokiy

lint:
    stage: all
    image: devops
    tags:
        - docker.elmer.nitrox
    script:
        - pip install anybadge
        - pylint kokiy --output-format=text --exit-zero | tee pylint.txt
        - score=$(sed -n 's/^Your code has been rated at \([-0-9.]*\)\/.*/\1/p' pylint.txt)
        - anybadge -l pylint  -o --file=pylint.svg -v $score  2=red 4=orange 8=yellow 10=green
    artifacts:
        paths:
          - pylint.svg

pages:
    stage: all
    image: devops
    tags:
        - docker.elmer.nitrox
    script:
        - cd docs
        - make html
        - mv _build/html/ ../public/
    artifacts:
        paths:
            - public
    only:
        - master

Do not forget to remove pages for open-source packages configurations.

Add badges

Our repos have typical four badges: pipeline, coverage, pylint, and docs. These are useful to quickly get information about the repo. You can set them out by going to Settings -> General -> Badges.

With exception of docs, all the badges are set with link https://nitrox.cerfacs.fr/%{project_path}. The corresponding images urls are https://nitrox.cerfacs.fr/%{project_path}/badges/master/pipeline.svg, https://nitrox.cerfacs.fr/%{project_path}/badges/%{default_branch}/coverage.svg, and https://nitrox.cerfacs.fr/%{project_path}/-/jobs/artifacts/%{default_branch}/raw/pylint.svg?job=lint for pipeline, coverage, and pylint, respectively.

The coverage information is gathered in the pipeline log by a regular expression that can be set in Settings -> CI/CD -> Test coverage parsing.

docs is set only to open-source packages and is set with both link and badge image url https://readthedocs.org/projects/%{project_path}/badge/?version=latest.

CI troubleshooting

Ordered by frequency, and solution:

  • linting Try to fix the code, and not the .pylintrc.
  • importing If you add a new package in your setup.py, the pipeline

Command Line Interface

We usually have a fast-start CLI in our packages. It is done with click.

The idea is to type the name of the package in the terminal to discover all the implemented commands. Read our dedicated post on CLI

Concluding remarks

Our standards are continuously improving (sometimes faster that our blog posts updates), so keep sure you peek into our repositories to get the most up-to-date information.

Like this post? Share on: TwitterFacebookEmail


Antoine Dauptain is a research scientist focused on computer science and engineering topics for HPC.
Luís F. Pereira is an engineer that enjoys to develop science/engineering related software.

Keep Reading


Published

Category

Pitch

Tags

Stay in Touch