Python and NumPy version policy
This serves to inform our community of planned changes to the versions of Python and NumPy that PyTango will actively support. This topic was mentioned at the 36th Tango Community Meeting in June 2022. It has also been discussed at several PyTango developers' meetings. The proposal is outlined first, followed by a longer discussion of the reasoning.
TL;DR: the next PyTango release will require at least Python 3.9. In 2024 the requirement will increase to Python 3.10.
Proposal
Supported versions are determined based on each PyTango release's anticipated release date, as follows:
- All minor versions of Python released 42 months prior to that date, and at minimum the two latest minor versions.
- All minor versions of NumPy released at that date that meet the requirements in oldest-supported-numpy for the corresponding Python version and platform.
As Python minor versions are released annually, this means that PyTango will drop support for the oldest minor Python version every year, and also gain support for a new minor version every year.
Note: NumPy's NEP 29 policy requires that dependency versions are only changed on minor or major releases, however, as PyTango does not follow semantic versioning we allow changing the dependencies on any release, including a patch release. If PyTango ever changes to semantic versioning, then we can avoid such dependency updates on patch releases.
For example, a hypothetical PyTango release on 2023-07-14 would support:
Python | Platform | NumPy |
---|---|---|
3.9 | x86_64, win_amd64, win32, aarch64 | >=1.19.3 |
3.9 | arm64 (macOS) | >=1.21.0 |
3.10 | x86_64, win_amd64, win32, aarch64, arm64 | >=1.21.6 |
3.11 | x86_64, win_amd64, win32, aarch64, arm64 | >=1.23.2 |
A release after 5 April 2024 would require at least Python 3.10, and support Python 3.12.
Note: others platforms may be usable, but that would require users compiling from source and doing their own validation.
Rationale
Considerations when creating the proposal:
- What versions does Python itself support?
- What versions of Python do our dependencies support, e.g., NumPy?
- What versions of Python do our packaging tools support, e.g., Conda-Forge, Linux (Debian, Ubuntu, RedHat, etc.)?
- Risks of continuing to use unsupported versions.
- Effect on PyTango developers supporting multiple versions.
- Effect on PyTango users to migrate to newer versions of Python.
What versions does Python itself support?
As per PEP 602 Python has adopted a yearly release cycle, with 5 years of support. 1.5 years of full support, and then 3.5 more years of security fixes.
Currently, PyTango supports from Python 3.6 up to the latest, 3.11. However, Python is no longer providing security updates for all of these versions. For example:
- 3.6 reached the end-of-life phase on 2021-12-23.
- 3.7 will reach end-of-life in 2023-06.
We should not support PyTango running on versions of Python that the Python organisation does not support. This gives us a maximum support time window of 5 years.
What versions of Python do our dependencies support, e.g., NumPy?
NumPy is a core dependency for PyTango, and extremely common in the scientific computing community where Tango Controls is most used. We should ensure the NumPy and Python versions PyTango uses match those supported by the NumPy developers.
NumPy have chosen a Python support time window shorter than Python's own support period. This allows NumPy to adopt language features sooner, and reduces the binary artefacts that have to be created for each release. Their current version support policy is documented in the enhancement proposal NEP 29:
- All minor versions of Python released 42 months prior to the project, and at minimum the two latest minor versions.
- All minor versions of numpy released in the 24 months prior to the project, and at minimum the last three minor versions.
However, PyTango's main use of NumPy is for data structures rather than algorithms. NumPy's core data structures have not changed noticeably in many years. Hence, we can allow as wide a range of NumPy versions as possible, so as not to restrict the packages that users install alongside PyTango.
When building the PyTango C++ extension code, we should use the oldest compatible version of NumPy, as this ensures ABI compatibility if the user has a newer version of NumPy installed as a runtime dependency. This problem is solved by the oldest-supported-numpy package.
What versions of Python do our packaging tools support, e.g., Conda-Forge, Linux (Debian, Ubuntu, RedHat, etc.)?
We cannot provide binary for Python versions that are not supported by the package hosting services.
Other than PyPI.org, our primary packaging repository is Conda-forge. Conda-forge's Python policy is pragmatic and follows the lead of its core packages, including NumPy.
On Linux the update cycle (getting PyTango release into upstream Linux packages) is much slower. We don't have dedicated support for the packaging on these platforms, so their choice of Python version is not considered critical.
Risks of continuing to use unsupported versions.
Using Python versions that no longer have security updates presents a cyber security risk. We can reduce the risk by dropping support for Python versions before security updates end (5 years from release).
Effect on PyTango developers supporting multiple versions.
Supporting multiple versions of Python is quite straight forward with our continuous integration (CI) tooling. Even if the CI pipeline consumes additional compute time for each version, the different versions run in parallel, so this does not increase the total time that developers have to wait for builds.
However, it is still not zero cost:
-
we sometimes have to fix broken builds due to changes in dependencies.
-
there is a lost opportunity cost - we cannot use newer Python language features, or adopt packages that depend on these. Python is changing quickly, and the PyTango developers would like to take advantage of the improvements in the language, and possibly adopt new packages that depend on these. This can benefit users of PyTango as well.
A concrete example is the modernisation of PyTango's build system - some promising tools requires at least Python 3.8. If PyTango still has to support Python 3.6, then we cannot use this tooling, and need to invest more time in an alternative solution. Some of the feature our old build system dependencies have already been deprecated, and will be removed in Python 3.12.
Effect on PyTango users to migrate to newer versions of Python.
If a new release of PyTango drops support for an older version of Python, this will affect users still
"stuck" on that old version of Python. They cannot upgrade to newer PyTango releases and benefit from bug fixes or new features. They may need to spend significant effort updating their systems to newer Python versions first.
However, this is part of the maintenance cost of all software projects - eventually the work will have to be done,
it is just a question of how long it gets delayed. Users also have the option of not upgrading - the older versions
of PyTango will still be available.
Examples
If we only use Python's end-of-life support window, and oldest-supported-numpy config, a hypothetical PyTango release on 2023-07-14 would support:
Python | Platform | NumPy |
---|---|---|
3.8 | x86_64, win_amd64, win32 | >=1.17.3 |
3.8 | aarch64 | >=1.19.2 |
3.8 | arm64 (macOS) | >=1.21.0 |
3.9 | x86_64, win_amd64, win32, aarch64 | >=1.19.3 |
3.9 | arm64 (macOS) | >=1.21.0 |
3.10 | x86_64, win_amd64, win32, aarch64, arm64 | >=1.21.6 |
3.11 | x86_64, win_amd64, win32, aarch64, arm64 | >=1.23.2 |
If we use NumPy's shorter support window, a hypothetical PyTango release on 2023-07-14 would support:
Python | Platform | NumPy |
---|---|---|
3.9 | x86_64, win_amd64, win32, aarch64 | >=1.19.3 |
3.9 | arm64 (macOS) | >=1.21.0 |
3.10 | x86_64, win_amd64, win32, aarch64, arm64 | >=1.21.6 |
3.11 | x86_64, win_amd64, win32, aarch64, arm64 | >=1.23.2 |
Note: others platforms may be usable, but that would require users compiling from source and doing their own validation.
Decision
Follow the NumPy limit of 42 months for Python, but keep the NumPy support as wide as possible.
Implementation details
- Update "how to contribute" docs with wording from the proposal, and add additional steps to the release process.
- Add a page to the documentation with the full text of this issue's description (after any updates).
- Change the
python_requires">=3.6"
insetup.py
(or similar) to the correct version, e.g.,python_requires">=3.9"
. See NumPy's NEP 29 drop schedule. - We are already using
oldest-supported-numpy
as a requirement in the build system. - We do not have to change the minimum runtime dependency for NumPy. Currently, we have:
numpy (>=1.13.3)
.