micropython-lib does not install expected libraries

Note: There are two presentations of this issue; they could be filed as separate issues, but they share a root cause, so Iʼm including both here. I also didnʼt know how minimal “minimal” was meant to be in the checklist so I included one set of files with no Linux at all, and two (one of each presentation) that is minimally changed from qemu_riscv64_virt_defconfig to demonstrate on a bootable system. I think I could squeeze that down more (I donʼt actually need eg. the network settings in those defconfigs) but it is already a small change and it was easier to work with a minimal diff from a bootable defconfig.

Check-list

  • I did not find the issue in the existing issues
  • I can reproduce the issue with unmodified Buildroot from this repository, not from a fork somewhere else
  • I can reproduce the issue on the latest commit of the branch I'm using:
    • master [as of last night; the two commits between cloning and this issue report do not look relevant so I did not re-run]
    • stable (i.e. 20NN.MM.x - please specify) [Does not build on my machine; would reproduce on 2025.11.x if I cherry-picked 875d6aed]
    • LTS (i.e. 20NN.02.x - please specify) [Does not build on my machine; might reproduce on 2025.02.x if I cherry-picked 875d6aed, but I have not tested]
  • I can reproduce the issue after running make clean; make
  • I attached the full build log file (e.g. make 2>&1 |tee build.log). Please ignore the build directory; it is related to how I first discovered these issues, but as asserted both above and below I am using an unmodified buildroot.
  • I attached a minimal defconfig file that can reproduce the issue (make BR2_DEFCONFIG=$(pwd)/issue_defconfig savedefconfig)
  • I also attached the configuration for kconfig-based packages that are enabled (and necessary to reproduce the issue), most notably:
    • Not attached because it is all default.

What I did

  • Buildroot commit sha1: 2026.02-rc1-1-g5c7d9ca80e
  • Distribution of the build machine: Arch Linux (no OS version), updated within the last week, gmake 4.4.1, gcc 15.2.1 20260209

Here, describe what you did:

  • any special environment variables: CC, CXX, TARGET, CROSS_COMPILE, etc…: No relevent environment variables set
  • the commands you ran: Only the first of these is for building; the rest are for diagnosing the problem 
    $ make 2>&1 |tee build.log
$ head target/usr/lib/micropython/os/__init__.py
$ head target/usr/lib/micropython/time.py
$ head target/usr/lib/micropython/re.py
  • For the bootable builds, I additionally:
    • run images/start_qemu.sh
    • log in as root and run micropython
    • run import os,
    • type os.O followed by hitting tab a couple times to check for autocomplete (no options are presented, as explained below); alternately dir(os) should show everything (and will not show any fields beginning with O_).
    • type os.path and hit enter
    • Attempt to import json and if successful run json.loads('17')

What happens

Micropython has multiple versions of some libraries, and they are shadowing each other in unintuitive ways. The os and time modules have built-in options even without micropython-lib, and two different versions included in micropython-lib (in the python-stdlib section and the unix-ffi section); there are five other modules in the unix-ffi part of micropython-lib that shadow built-in modules but have no python-stdlib version. Unfortunately, collect_micropython_lib.py does not account for this, which can cause problems in two different ways:

  • With libffi, some unix-ffi modules are not present when they should be; I first discovered this issue because of missing features that are in the unix-ffi/os module.
  • Without libffi (and sometimes with, because of upstream bugs), some unix-ffi modules shadow more-functional non-unix-ffi modules.

For the os and time modules I see the python-stdlib version, not the unix-ffi version (I think I have seen the unix-ffi version of time.py but I canʼt reproduce that; it might be nondeterministic or it might be because of poking around I did trying to figure out the issue before switching to a clean environment for the report).

Additionally, for the other unix-ffi modules I always see the unix-ffi version even when libffi is not built.

For the bootable tests:

  • import os works every time, as it should
  • There are no tab-complete suggestions for os.O, which there would be in the unix-ffi module (this is why I even started investigating in the first place; I want the extra options in os)
  • os.path says it is <module 'os.path' from '/usr/lib/micropython/os/path.py'>, which indicates we have the python-stdlib version instead of the builtin version. Upstream also had this issue at one point and fixed it but we donʼt use the same build system.
  • When the system is built without libffi, import json fails because itʼs trying to use the unix-ffi version.
  • When the system is built with libffi, json.loads('17') fails with an overflow error from re.py, which indicates we do have the buggy unix-ffi version of both the json and re modules. That is a useful diagnostic, although obviously also undesirable behavior as discussed below.

What was expected

Initially, I expected that if a module existed in both python-stdlib and unix-ffi, I would get the unix-ffi version if libffi is built, and that I would not get any unix-ffi modules with libffi not built.

After further research, however, I am less sure this is the desired result. The unix-ffi version of the re library is buggy, and although in many cases itʼs still preferable to the built-in one, it should be optional. The unix-ffi json library depends on the unix-ffi re library and is completely broken with the re bug, so shouldnʼt be installed at all probably. Those are the only ones I have personally had issues with but I could easily be missing other issues; I think some people might prefer not to install any of the unix-ffi libraries, even with both libffi and micropython-lib.

So a better response might be:

  • If micropython-lib is selected and libffi is not, install all the micropython-lib modules except those in unix-ffi. This is simple, and if that were everything it would be a relatively small change to micropythonʼs Config.in and collect_micropython_lib.py files.
  • If micropython-lib and libffi are both selected, things get more complicated. I think more options should be presented but I donʼt know what the best options are; thereʼs a tradeoff between the flexibility of offering more options and the extra work of translating more of the manifest system to KConfig.
    • At minimum, I think there should be a three-way selection between no unix-ffi, unix-ffi only for modules that are not available elsewhere, and all of unix-ffi (ensuring that os and time take precedence over the python-stdlib options).
    • The last option should either have an option to exclude json and re, or have a warning. There are times when the unix-ffi version of re is still better, so it shouldnʼt unconditionally exclude that. I donʼt know if any of the other modules in unix-ffi that override builtin modules are broken in a way you might prefer to exclude them.
    • For reference, the unix-ffi modules that are available elsewhere are json (broken-by-dependency), machine, os, time, re (broken), select, and socket.
  • If for whatever reason the re module is not set to be installed, then BR2_PACKAGE_PCRE2 shouldnʼt be auto-selected when installing micropython-lib.

Extra information

It looks like some of the micropython tools have updated, and might include some more of the collect_micropython_lib.py functionality. Some of the methods used by collect_micropython_lib.py have also changed so those calls would need to be rewritten. Iʼm not sure, but this might be simpler if the micropython version were first bumped to something more recent.

Edited by Daniel Houck