# python projects to use `pyproject.toml` - [Ticket 7043](https://www.kbsoftware.co.uk/crm/ticket/7043/) - [Chat with Malcolm...](https://chat.kbsoftware.co.uk/kb/pl/x7z5torw4jdujp4u5598qr8qzy) @all In python 3.12 `distutils` is being removed and using `pyproject.toml` is the modern way to specify a project - you can still have a `setup.py` as a configuration file but pretty much any use of `python setup.py ...` has apparently been deprecated for some time. To learn about this I've updated the *hatherleigh* project (on a branch) to use a `pyproject.toml`. I've done it in a way that is compatible with running `pip install -r requirements/local.txt` or `pip install -r requirements/production.txt` but the contents of those files have changed a bit. `pyproject.toml` is the only file needed to specify a distribution but you can specify that it use requirements files for the dependencies - they have a restricted syntax - no `-e` or `-r` lines and git repos and local apps must have the ` @ ` syntax. Essentially what that means is I've made the `production.txt` and `local.txt` files simply stubs that read other files and for the reasons outlined are ignored by the `pyproject.toml` setup. I've created a `dev.txt` (for development dependencies excluding the apps) and a `release.txt` (for the apps used in production), the local apps which need to be editable installs (the `-e` option) are read from a requirements file called `apps.txt` but this needs the full path to the app (and have the syntax ` @ ` which we can't put in the repo as that would make it difficult to share projects. My solution is create a `apps.txt` from the `release.txt` (that also has the advantage that we only need to specify the apps once. I've created a branch on toolbox [make-scss-test-optional...](https://github.com/pkimber/toolbox/compare/master...make-scss-test-optional) BTW: the branch started out as just a way to remove the new scss prompt when using my `release` script as that script processes the scss and less scripts. Hence the obscure name) that adds an option `--create-apps-txt` to `kb.py` to create this file (and added `requirements/apps.txt` to `.gitignore` in the `hatherleigh` project), I've also amended the `create-venv` script in `dev-scripts` to create this file too if the project has a `release.txt` file. I've amended the checks on the project to check if the project uses the `pyproject.toml` file and in that case check the `release.txt` file rather than `production.txt`. and since the local apps file is created from that it does not check the local apps. The `production.txt` file now looks like:: ``` -r base.txt -r release.txt ``` `local.txt` file now looks like:: ``` -e .[dev] ``` `-e.[dev]` installs the project using the development dependencies specified in the `pyproject.toml` file, which reads the dev dependencies from `requirements/base.txt`, `requirements/apps.txt` and `requirements/dev.txt`. The new files are - release.txt:: ``` kb-base==0.3.37 kb-block==0.2.11 kb-login==0.2.34 kb-mail==0.1.87 ``` dev.txt:: ``` beautifulsoup4 black django-debug-toolbar django-extensions factory-boy freezegun GitPython pyOpenSSL pytest-cov pytest-django pytest-mock redis-dump-load rich semantic-version walkdir Werkzeug ``` apps.txt (generated using `~/dev/modules/toolbox/kb.py --create-apps-txt`) ``` kb-base @ file:///home/patrick/dev/app/base kb-block @ file:///home/patrick/dev/app/block kb-login @ file:///home/patrick/dev/app/login kb-mail @ file:///home/patrick/dev/app/mail ``` The `pyproject.toml` already existed in the `hatherleigh` project to configure `black`. Setuptools will use this file if it exists and creates the distribution slightly differently but does not use the modern config without a `[project]` section. This parts of this file that are used by setuptools now look like this: ``` [project] name = "kb-hatherleigh" description = "go.hatherleigh website" dynamic = ["version", "dependencies", "readme", "authors", "classifiers", "optional-dependencies"] [build-system] requires = ["setuptools >= 61.0"] build-backend = "setuptools.build_meta" [tool.setuptools.dynamic] dependencies = {file = ["requirements/base.txt"]} optional-dependencies = {dev = {file = ["requirements/apps.txt", "requirements/dev.txt"]},production = {file = ["requirements/release.txt"]}} ``` The `[build-system]` specifies setuptools needs to be greater than 61, I'm not exactly sure that that means when using on older versions of Ubuntu. I don't think it is an issue because you can install the latest version of `setuptools` using `pip install --upgrade setuptools`. The dynamic list in the project section essentially means those sections are created from the setup.py or as specified in the `[tools.setuptools.dynamic]` section. To install a project for development you can use `pip install -e .[dev]` or the conventional method of `pip install -r requirements/local.txt` To install in production you should be able to use ``` pip install [production]` pip install -r requirements/production.txt` ``` We will need to change the commands that are run when we invoke `toolbox/kb.py --release` as that makes some calls to `python setup.py ...` see [modernize-setup-py-project...](https://packaging.python.org/en/latest/guides/modernize-setup-py-project/#modernize-setup-py-project) the changes I've made simply allow for the new requirements structure. There are probably some changes we need to make to fabric as well. I cannot see a way to make local dependencies (e.g. KB apps) editable installs within a project using `setuptools`. If changes are made to an app the only way I can see to incorporate these is to remove the build directory in the app and run `pip install -e .[dev]`(or `pip install -r requirements/local.txt`) on the project. I've amended my `set-branches` script to remove the build directory from the app if the branch would be checked out. I've not deployed using this yet but hopefully it should still work with the current fabric setup. I'll obviously test that before and wait for your feedback before merging it into the main branch.