python projects to use `pyproject.toml`

@all In python 3.12 distutils is being removed and using pyproject.toml is the modern way to specify a project - you can still have a setup.py as a configuration file but pretty much any use of python setup.py ... has apparently been deprecated for some time.

To learn about this I’ve updated the hatherleigh project (on a branch) to use a pyproject.toml. I’ve done it in a way that is compatible with running pip install -r requirements/local.txt or pip install -r requirements/production.txt but the contents of those files have changed a bit.

pyproject.toml is the only file needed to specify a distribution but you can specify that it use requirements files for the dependencies - they have a restricted syntax - no -e or -r lines and git repos and local apps must have the <app name> @ <url to app> syntax.

Essentially what that means is I’ve made the production.txt and local.txt files simply stubs that read other files and for the reasons outlined are ignored by the pyproject.toml setup.

I’ve created a dev.txt (for development dependencies excluding the apps) and a release.txt (for the apps used in production), the local apps which need to be editable installs (the -e option) are read from a requirements file called apps.txt but this needs the full path to the app (and have the syntax <app-name> @ <full path to app> which we can’t put in the repo as that would make it difficult to share projects.

My solution is create a apps.txt from the release.txt (that also has the advantage that we only need to specify the apps once. I’ve created a branch on toolbox make-scss-test-optional… BTW: the branch started out as just a way to remove the new scss prompt when using my release script as that script processes the scss and less scripts. Hence the obscure name) that adds an option --create-apps-txt to kb.py to create this file (and added requirements/apps.txt to .gitignore in the hatherleigh project), I’ve also amended the create-venv script in dev-scripts to create this file too if the project has a release.txt file. I’ve amended the checks on the project to check if the project uses the pyproject.toml file and in that case check the release.txt file rather than production.txt. and since the local apps file is created from that it does not check the local apps.

The production.txt file now looks like::

  -r base.txt
  -r release.txt

local.txt file now looks like::

   -e .[dev]

-e.[dev] installs the project using the development dependencies specified in the pyproject.toml file, which reads the dev dependencies from requirements/base.txt, requirements/apps.txt and requirements/dev.txt.

The new files are - release.txt::

  kb-base==0.3.37
  kb-block==0.2.11
  kb-login==0.2.34
  kb-mail==0.1.87

dev.txt::

  beautifulsoup4
  black
  django-debug-toolbar
  django-extensions
  factory-boy
  freezegun
  GitPython
  pyOpenSSL
  pytest-cov
  pytest-django
  pytest-mock
  redis-dump-load
  rich
  semantic-version
  walkdir
  Werkzeug

apps.txt (generated using ~/dev/modules/toolbox/kb.py --create-apps-txt)

  kb-base @ file:///home/patrick/dev/app/base
  kb-block @ file:///home/patrick/dev/app/block
  kb-login @ file:///home/patrick/dev/app/login
  kb-mail @ file:///home/patrick/dev/app/mail

The pyproject.toml already existed in the hatherleigh project to configure black. Setuptools will use this file if it exists and creates the distribution slightly differently but does not use the modern config without a [project] section. This parts of this file that are used by setuptools now look like this:

  [project]
  name = "kb-hatherleigh"
  description = "go.hatherleigh website"
  dynamic = ["version", "dependencies", "readme", "authors", "classifiers", "optional-dependencies"]

  [build-system]
  requires = ["setuptools >= 61.0"]
  build-backend = "setuptools.build_meta"

  [tool.setuptools.dynamic]
  dependencies = {file = ["requirements/base.txt"]}
  optional-dependencies = {dev = {file = ["requirements/apps.txt", "requirements/dev.txt"]},production = {file = ["requirements/release.txt"]}}

The [build-system] specifies setuptools needs to be greater than 61, I’m not exactly sure that that means when using on older versions of Ubuntu. I don’t think it is an issue because you can install the latest version of setuptools using pip install --upgrade setuptools. The dynamic list in the project section essentially means those sections are created from the setup.py or as specified in the [tools.setuptools.dynamic] section.

To install a project for development you can use pip install -e .[dev] or the conventional method of pip install -r requirements/local.txt

To install in production you should be able to use

pip install <project name>[production]`
pip install -r requirements/production.txt`

We will need to change the commands that are run when we invoke toolbox/kb.py --release as that makes some calls to python setup.py ... see modernize-setup-py-project… the changes I’ve made simply allow for the new requirements structure. There are probably some changes we need to make to fabric as well.

I cannot see a way to make local dependencies (e.g. KB apps) editable installs within a project using setuptools. If changes are made to an app the only way I can see to incorporate these is to remove the build directory in the app and run pip install -e .[dev](or pip install -r requirements/local.txt) on the project. I’ve amended my set-branches script to remove the build directory from the app if the branch would be checked out.

I’ve not deployed using this yet but hopefully it should still work with the current fabric setup.

I’ll obviously test that before and wait for your feedback before merging it into the main branch.

python projects to use pyproject.toml

python projects to use `pyproject.toml`