Bump Python to 3.13

Opening an issue to track the progress of migrating to Python 3.13.x

I have a patch that currently works and is passing all of the python tests from the unit test except for 5.

Tests run:

br-user@10cdb01dd09a:/work$ ./support/testing/run-tests -o /tmp/ -d /tmp/ -j10 tests.package.test_python_midiutil.TestPythonPy3MidiUtil.test_run tests.package.test_python_attrs.TestPythonPy3Attrs.test_run tests.package.test_python_botocore.TestPythonPy3Botocore.test_run tests.package.test_python_flask_expects_json.TestPythonPy3FlaskExpectsJson.test_run tests.package.test_python_sdbus_networkmanager.TestPythonPy3SdbusNetworkmanager.test_run tests.package.test_python_hwdata.TestPythonHwdata.test_run tests.package.test_python_incremental.TestPythonPy3Incremental.test_run tests.package.test_python_boto3.TestPythonPy3Boto3.test_run tests.package.test_python_waitress.TestPythonWaitress.test_run tests.package.test_python_mako.TestPythonPy3Mako.test_run tests.package.test_python_mako.TestPythonPy3MakoExt.test_run tests.package.test_python_dicttoxml2.TestPythonPy3Dicttoxml2.test_run tests.package.test_python_pexpect.TestPythonPy3Pexpect.test_run tests.package.test_python_pyyaml.TestPythonPy3Pyyaml.test_run tests.package.test_python_pyndiff.TestPyNdiff.test_run tests.package.test_python_fastapi.TestPythonPy3Fastapi.test_run tests.package.test_python_pymupdf.TestPythonPy3PyMuPDF.test_run tests.package.test_python_magic_wormhole.TestPythonPy3MagicWormhole.test_run tests.package.test_python_pysmb.TestPythonPy3Pysmb.test_run tests.package.test_python_sympy.TestPythonPy3SymPy.test_run tests.package.test_python_munch.TestPythonPy3Munch.test_run tests.package.test_python_service_identity.TestPythonPy3ServiceIdentity.test_run tests.package.test_python_evdev.TestPythonPy3Evdev.test_run tests.package.test_python_segno.TestPythonPy3Segno.test_run tests.package.test_python_gobject.TestPythonPy3Gobject.test_run tests.package.test_python_jc.TestPythonPy3Jc.test_run tests.package.test_python_uvloop.TestPythonPy3Uvloop.test_run tests.package.test_python_git.TestPythonPy3Git.test_run tests.package.test_python_ubjson.TestPythonPy3Ubjson.test_run tests.package.test_python_pytest_asyncio.TestPythonPy3PytestAsyncio.test_run tests.package.test_python_dtschema.TestPythonPy3Dtschema.test_run tests.package.test_python_msgpack.TestPythonPy3MsgPack.test_run tests.package.test_python_gpiozero.TestPythonPy3Gpiozero.test_run tests.package.test_python_aexpect.TestPythonPy3Aexpect.test_run tests.package.test_python_iptables.TestPythonPy3Iptables.test_run tests.package.test_python_augeas.TestPythonAugeas.test_run tests.package.test_python_dbus_next.TestPythonPy3DBusNext.test_run tests.package.test_python_distro.TestPythonPy3Distro.test_run tests.package.test_python_txtorcon.TestPythonPy3Txtorcon.test_run tests.package.test_python_sdbus.TestPythonPy3Sdbus.test_run tests.package.test_python_automat.TestPythonPy3Automat.test_run tests.package.test_python_s3transfer.TestPythonPy3S3transfer.test_run tests.package.test_python_spake2.TestPythonPy3Spake2.test_run tests.package.test_python_tftpy.TestPythonTftpy.test_run tests.package.test_python_rsa.TestPythonPy3RSA.test_run tests.package.test_python_pathspec.TestPythonPathSpec.test_run tests.package.test_python_ml_dtypes.TestPythonPy3MlDtypes.test_run tests.package.test_python_django.TestPythonPy3Django.test_run tests.package.test_python_unittest_xml_reporting.TestPythonPy3UnitTestXmlReporting.test_run tests.package.test_python_txaio.TestPythonPy3Txaio.test_run tests.package.test_python_can.TestPythonPy3Can.test_run tests.package.test_python_qrcode.TestPythonPy3Qrcode.test_run tests.package.test_python_treq.TestPythonPy3Treq.test_run tests.package.test_python_construct.TestPythonPy3Construct.test_run tests.package.test_python_pytest.TestPythonPy3Pytest.test_run tests.package.test_python_cbor.TestPythonPy3Cbor.test_run tests.package.test_python.TestPython3Py.test_run tests.package.test_python.TestPython3PyPyc.test_run tests.package.test_python.TestPython3Pyc.test_run tests.package.test_python_cryptography.TestPythonPy3Cryptography.test_run tests.package.test_python_constantly.TestPythonPy3Constantly.test_run tests.package.test_python_pynacl.TestPythonPy3Pynacl.test_run tests.package.test_python_hid.TestPythonHid.test_run tests.package.test_python_gitdb2.TestPythonPy3Gitdb2.test_run tests.package.test_python_dbus_fast.TestPythonPy3DBusFast.test_run tests.package.test_python_passlib.TestPythonPy3Passlib.test_run tests.package.test_python_scipy.TestPythonPy3SciPy.test_run tests.package.test_python_rtoml.TestPythonPy3rtoml.test_run tests.package.test_python_pyalsa.TestPythonPy3PyAlsa.test_run tests.package.test_python_click.TestPythonPy3Click.test_run tests.package.test_python_rpi_gpio.TestPythonPy3RpiGpio.test_run tests.package.test_python_bitstring.TestPythonPy3Bitstring.test_run tests.package.test_python_pybind.TestPythonPybind.test_run tests.package.test_gst1_python.TestGst1Python.test_run tests.package.test_python_argh.TestPythonPy3Argh.test_run tests.package.test_python_yamllint.TestPythonYamllint.test_run tests.package.test_python_hkdf.TestPythonPy3Hkdf.test_run tests.package.test_python_twisted.TestPythonPy3Twisted.test_run tests.package.test_python_autobahn.TestPythonPy3Autobahn.test_run tests.package.test_python_networkmanager_goi.TestPythonPy3NetworkmanagerGoi.test_run tests.package.test_python_pyqt5.TestPythonPyQt5.test_run tests.package.test_python_paho_mqtt.TestPythonPahoMQTT.test_run tests.package.test_python_minimalmodbus.TestPythonPy3Minimalmodbus.test_run tests.package.test_micropython.TestMicroPython.test_run tests.package.test_python_smmap2.TestPythonPy3Smmap2.test_run tests.package.test_python_avro.TestPythonAvro.test_run tests.package.test_python_mpmath.TestPythonPy3MpMath.test_run tests.package.test_python_jmespath.TestPythonPy3Jmespath.test_run tests.package.test_python_colorzero.TestPythonPy3Colorzero.test_run tests.package.test_python_asn1crypto.TestPythonPy3Asn1Crypto.test_run tests.package.test_python_ruamel_yaml.TestPythonPy3RuamelYaml.test_run tests.package.test_python_flask.TestPythonPy3Flask.test_run tests.package.test_python_gnupg.TestPythonPy3GnuPG.test_run tests.package.test_ipython.TestIPythonPy3.test_run

Failures:

======================================================================
ERROR: test_run (tests.package.test_python_gnupg.TestPythonPy3GnuPG.test_run)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/work/support/testing/tests/package/test_python.py", line 137, in test_run
    self.run_sample_scripts()
  File "/work/support/testing/tests/package/test_python.py", line 132, in run_sample_scripts
    self.assertRunOk(cmd, timeout=self.timeout)
  File "/work/support/testing/infra/basetest.py", line 89, in assertRunOk
    out, exit_code = self.emulator.run(cmd, timeout)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work/support/testing/infra/emulator.py", line 173, in run
    output = self.repl.run_command(cmd, timeout=timeout)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/pexpect/replwrap.py", line 103, in run_command
    if self._expect_prompt(timeout=timeout) == 1:
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/pexpect/replwrap.py", line 65, in _expect_prompt
    return self.child.expect_exact([self.prompt, self.continuation_prompt],
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/pexpect/spawnbase.py", line 421, in expect_exact
    return exp.expect_loop(timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/pexpect/expect.py", line 181, in expect_loop
    return self.timeout(e)
           ^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/pexpect/expect.py", line 144, in timeout
    raise exc
pexpect.exceptions.TIMEOUT: Timeout exceeded.
<pexpect.pty_spawn.spawn object at 0x769da214ad50>
command: /usr/bin/qemu-system-arm
args: [b'/usr/bin/qemu-system-arm', b'-serial', b'stdio', b'-display', b'none', b'-m', b'256', b'-initrd', b'/tmp/TestPythonPy3GnuPG/images/rootfs.cpio', b'-dtb', b'/tmp/versatile-pb-5.10.202.dtb', b'-M', b'versatilepb', b'-device', b'virtio-rng-pci', b'-kernel', b'/tmp/kernel-versatile-5.10.202', b'-append', b'console=ttyAMA0']
buffer (last 100 chars): 'reated\r\r\n'
before (last 100 chars): "batch --with-colons --passphrase-fd 0 --symmetric --armor\r\r\ngpg: directory '/root/.gnupg' created\r\r\n"
after: <class 'pexpect.exceptions.TIMEOUT'>
match: None
match_index: None
exitstatus: None
flag_eof: False
pid: 1239594
child_fd: 16
closed: False
timeout: 5
delimiter: <class 'pexpect.exceptions.EOF'>
logfile: None
logfile_read: <_io.TextIOWrapper name='/tmp/TestPythonPy3GnuPG-run.log' mode='a+' encoding='UTF-8'>
logfile_send: None
maxread: 2000
ignorecase: False
searchwindowsize: None
delaybeforesend: 0.05
delayafterclose: 0.1
delayafterterminate: 0.1
searcher: searcher_string:
    0: '[BRTEST# '
    1: '[BRTEST+ '

======================================================================
ERROR: test_run (tests.package.test_python_scipy.TestPythonPy3SciPy.test_run)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/work/support/testing/infra/basetest.py", line 74, in setUp
    self.b.build()
  File "/work/support/testing/infra/builder.py", line 105, in build
    raise SystemError("Build failed")
SystemError: Build failed

======================================================================
ERROR: test_run (tests.package.test_python_spake2.TestPythonPy3Spake2.test_run)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/work/support/testing/tests/package/test_python.py", line 137, in test_run
    self.run_sample_scripts()
  File "/work/support/testing/tests/package/test_python.py", line 132, in run_sample_scripts
    self.assertRunOk(cmd, timeout=self.timeout)
  File "/work/support/testing/infra/basetest.py", line 89, in assertRunOk
    out, exit_code = self.emulator.run(cmd, timeout)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work/support/testing/infra/emulator.py", line 173, in run
    output = self.repl.run_command(cmd, timeout=timeout)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/pexpect/replwrap.py", line 103, in run_command
    if self._expect_prompt(timeout=timeout) == 1:
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/pexpect/replwrap.py", line 65, in _expect_prompt
    return self.child.expect_exact([self.prompt, self.continuation_prompt],
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/pexpect/spawnbase.py", line 421, in expect_exact
    return exp.expect_loop(timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/pexpect/expect.py", line 181, in expect_loop
    return self.timeout(e)
           ^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/pexpect/expect.py", line 144, in timeout
    raise exc
pexpect.exceptions.TIMEOUT: Timeout exceeded.
<pexpect.pty_spawn.spawn object at 0x769da21675d0>
command: /usr/bin/qemu-system-arm
args: [b'/usr/bin/qemu-system-arm', b'-serial', b'stdio', b'-display', b'none', b'-m', b'256', b'-initrd', b'/tmp/TestPythonPy3Spake2/images/rootfs.cpio', b'-dtb', b'/tmp/versatile-pb-5.10.202.dtb', b'-M', b'versatilepb', b'-device', b'virtio-rng-pci', b'-kernel', b'/tmp/kernel-versatile-5.10.202', b'-append', b'console=ttyAMA0']
buffer (last 100 chars): 'ke2.py\r\r\n'
before (last 100 chars): 'python sample_python_spake2.py\r\r\n'
after: <class 'pexpect.exceptions.TIMEOUT'>
match: None
match_index: None
exitstatus: None
flag_eof: False
pid: 2783661
child_fd: 15
closed: False
timeout: 5
delimiter: <class 'pexpect.exceptions.EOF'>
logfile: None
logfile_read: <_io.TextIOWrapper name='/tmp/TestPythonPy3Spake2-run.log' mode='a+' encoding='UTF-8'>
logfile_send: None
maxread: 2000
ignorecase: False
searchwindowsize: None
delaybeforesend: 0.05
delayafterclose: 0.1
delayafterterminate: 0.1
searcher: searcher_string:
    0: '[BRTEST# '
    1: '[BRTEST+ '

======================================================================
ERROR: test_run (tests.package.test_python_evdev.TestPythonPy3Evdev.test_run)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/work/support/testing/infra/basetest.py", line 74, in setUp
    self.b.build()
  File "/work/support/testing/infra/builder.py", line 105, in build
    raise SystemError("Build failed")
SystemError: Build failed

======================================================================
FAIL: test_run (tests.package.test_python_aexpect.TestPythonPy3Aexpect.test_run)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/work/support/testing/tests/package/test_python.py", line 137, in test_run
    self.run_sample_scripts()
  File "/work/support/testing/tests/package/test_python.py", line 132, in run_sample_scripts
    self.assertRunOk(cmd, timeout=self.timeout)
  File "/work/support/testing/infra/basetest.py", line 90, in assertRunOk
    self.assertEqual(
AssertionError: 1 != 0 : 
Failed to run: python sample_python_aexpect.py
output was:
  Traceback (most recent call last):
    File "/root/sample_python_aexpect.py", line 1, in <module>
      import aexpect
    File "/usr/lib/python3.13/site-packages/aexpect/__init__.py", line 35, in <module>
    File "/usr/lib/python3.13/site-packages/aexpect/remote.py", line 50, in <module>
  ModuleNotFoundError: No module named 'pipes'

----------------------------------------------------------------------
Ran 94 tests in 21397.065s

FAILED (failures=1, errors=4)
br-user@0fe9bf45cbd0:/work$

TestPythonPy3Aexpect: needs to be bumped to include https://github.com/avocado-framework/aexpect/commit/f2b888b1bd453a46820d38b62fd2b5db1228e4b7
TestPythonPy3GnuPG: the rng in QEMU is taking > 5 seconds to initialize, so the test times out

# time python ./sample_python_gnupg.py 
random: python: uninitialized urandom read (24 bytes read)
gpg --status-fd 2 --no-tty --no-verbose --fixed-list-mode --batch --with-colons --list-config --with-colons
gpg --pinentry-mode loopback --status-fd 2 --no-tty --no-verbose --fixed-list-mode --batch --with-colons --passphrase-fd 0 --symmetric --armor
gpg: directory '/root/.gnupg' created
random: crng init done
gpg: keybox '/root/.gnupg/pubring.kbx' created
[GNUPG:] NEED_PASSPHRASE_SYM 9 3 8
[GNUPG:] BEGIN_ENCRYPTION 2 9
[GNUPG:] END_ENCRYPTION
gpg --pinentry-mode loopback --status-fd 2 --no-tty --no-verbose --fixed-list-mode --batch --with-colons --passphrase-fd 0 --decrypt
gpg: AES256.CFB encrypted data
[GNUPG:] NEED_PASSPHRASE_SYM 9 3 8
gpg: encrypted with 1 passphrase
[GNUPG:] BEGIN_DECRYPTION
[GNUPG:] DECRYPTION_INFO 2 9 0
[GNUPG:] PLAINTEXT 62 23
[GNUPG:] PLAINTEXT_LENGTH 20
[GNUPG:] DECRYPTION_OKAY
[GNUPG:] GOODMDC
[GNUPG:] END_DECRYPTION
gpg --pinentry-mode loopback --status-fd 2 --no-tty --no-verbose --fixed-list-mode --batch --with-colons --passphrase-fd 0 --decrypt
gpg: AES256.CFB encrypted data
[GNUPG:] NEED_PASSPHRASE_SYM 9 3 8
gpg: encrypted with 1 passphrase
[GNUPG:] BEGIN_DECRYPTION
[GNUPG:] DECRYPTION_INFO 2 9 0
[GNUPG:] ERROR symkey_decrypt.maybe_error 11_BAD_PASSPHRASE
[GNUPG:] DECRYPTION_FAILED
gpg: decryption failed: Bad session key
[GNUPG:] END_DECRYPTION
[GNUPG:] FAILURE gpg-exit 33554433
real    0m 5.94s
user    0m 1.06s
sys     0m 4.55s

TestPythonPy3Spake2: the rng in QEMU is taking > 5 seconds to initialize, so the test times out
TestPythonPy3Evdev: seems to be hitting an issue due to files included from the build host. This is fixed in https://github.com/gvalkov/python-evdev/commit/3ff9816e08be95b331ea9dadc18fc17f1e04e272 so would be addressed with a version bump to 1.9.0 (tested and it works)
TestPythonPy3SciPy: failing due to numpy failing to build

root@192e8eb10c63:/work# tail -n20 /tmp/TestPythonPy3SciPy-build.log 
Host machine cpu: arm926ej-s
Target machine cpu family: arm
Target machine cpu: arm926ej-s
Program python3 found: YES (/tmp/TestPythonPy3SciPy/host/bin/python3)
Found pkg-config: YES (/tmp/TestPythonPy3SciPy/host/bin/pkgconf) 2.3.0
Run-time dependency python found: YES 3.13
Has header "Python.h" with dependency python-3.13: YES 
Compiler for C supports arguments -fno-strict-aliasing: YES 
Library m found: YES
Run-time dependency openblas found: YES 0.3.29
Run-time dependency lapack found: YES 3.10.1
Program _build_utils/process_src_template.py found: YES (/tmp/TestPythonPy3SciPy/host/bin/python3 /tmp/TestPythonPy3SciPy/build/python-numpy-1.25.0/numpy/_build_utils/process_src_template.py)
Program _build_utils/tempita.py found: YES (/tmp/TestPythonPy3SciPy/build/python-numpy-1.25.0/numpy/_build_utils/tempita.py)

../tmp/TestPythonPy3SciPy/build/python-numpy-1.25.0/numpy/meson.build:203:49: ERROR: Could not get pkg-config variable and no default provided for <PkgConfigDependency lapack: True None>

A full log can be found at /tmp/TestPythonPy3SciPy/build/python-numpy-1.25.0/build/meson-logs/meson-log.txt
make[1]: *** [package/pkg-generic.mk:279: /tmp/TestPythonPy3SciPy/build/python-numpy-1.25.0/.stamp_configured] Error 1
make: *** [Makefile:23: _all] Error 2
make: Leaving directory '/tmp/TestPythonPy3SciPy'

root@192e8eb10c63:/work# tail -n20 /tmp/TestPythonPy3SciPy/build/python-numpy-1.25.0/build/meson-logs/meson-log.txt
Got pkg-config variable libdir : /tmp/TestPythonPy3SciPy/host/arm-buildroot-linux-gnueabi/sysroot/usr/lib
env[PKG_CONFIG_PATH]: 
env[PKG_CONFIG_SYSROOT_DIR]: /tmp/TestPythonPy3SciPy/host/arm-buildroot-linux-gnueabi/sysroot
env[PKG_CONFIG_LIBDIR]: /tmp/TestPythonPy3SciPy/host/arm-buildroot-linux-gnueabi/sysroot/usr/lib/pkgconfig:/tmp/TestPythonPy3SciPy/host/arm-buildroot-linux-gnueabi/sysroot/usr/share/pkgconfig
env[PKG_CONFIG]: /tmp/TestPythonPy3SciPy/host/bin/pkgconf
-----------
Called: `/tmp/TestPythonPy3SciPy/host/bin/pkgconf --variable=openblas_config lapack` -> 0
env[PKG_CONFIG_PATH]: 
env[PKG_CONFIG_SYSROOT_DIR]: /tmp/TestPythonPy3SciPy/host/arm-buildroot-linux-gnueabi/sysroot
env[PKG_CONFIG_LIBDIR]: /tmp/TestPythonPy3SciPy/host/arm-buildroot-linux-gnueabi/sysroot/usr/lib/pkgconfig:/tmp/TestPythonPy3SciPy/host/arm-buildroot-linux-gnueabi/sysroot/usr/share/pkgconfig
env[PKG_CONFIG]: /tmp/TestPythonPy3SciPy/host/bin/pkgconf
-----------
Called: `/tmp/TestPythonPy3SciPy/host/bin/pkgconf --print-variables lapack` -> 0
stdout:
includedir
libdir
pcfiledir
-----------

../tmp/TestPythonPy3SciPy/build/python-numpy-1.25.0/numpy/meson.build:203:49: ERROR: Could not get pkg-config variable and no default provided for <PkgConfigDependency lapack: True None>

The fact that pkgconf is printing the values shows the pc file is available in the sysroot/libdir paths so this seems a bit odd.

Regardless, both scipy and numpy need to be bumped for Python 3.13 support: scipy to 1.14.1+ and numpy to 2.1.0+