Failing child processes when tracking missing git branches

Some child processes appear to crash when continuing after an element fails to track its git branch due to a missing branch, resulting in buildstream getting stuck waiting for their completion.

See this log of the stderr of bst --on-error continue build --track base-system.bst on the booting-base-system-scratch branch of buildstream-tests: error.log

A minimal test case for my system:

project.conf:

name: test-weird-crashes

element-path: elements

elements/test.bst:

kind: manual

depends:
  - another_test.bst
  - another_test1.bst
  - another_test2.bst
  - another_test3.bst
  - another_test4.bst
  - another_test5.bst
  - another_test6.bst

sources:
- kind: git
  url: git://git.baserock.org/delta/busybox
  track: 1_2_3_1

  ref: 1231a93bd05f8999fb25560fae17e7bc1d272a4f
config:
  configure-commands:
    - echo 'Test'

elements/another_test{1..6}.bst:

kind: manual

config:
  configure-commands:
    - echo 'This is only here to be a dependency...'

Running bst --on-error continue build --track test.bst, you should see messages like these:

Unknown child process pid 3158, will report returncode 255
Unknown child process pid 3162, will report returncode 255

This seems like a very specific multithreading issue, so it is probably hard to reproduce...

changed the description

I'll look into this...

Please refrain from referring to any external resources in issue reports, especially pastebins, this practice leads to broken bug history, or in some cases external resources change content (like referring to web pages or git repos without precise commit shas in the URI).

Ideally we should have an attachment only policy, but i dont know how well gitlab supports these, if at all.

changed the description

This has nothing to do with retrying failures, it happens regularly, every time I build GNOME with bst build --track

mentioned in issue #125 (closed)

Additional info, I think this is related.

So, when I last reproduced this (which is every time), an additional thing happened; at first I thought it was related to #118 (closed) (failing to timeout on network), but to my surprise, this is the end of my log:

[00:00:00][766ffb3a][build:core-deps/gsound.bst          ] SUCCESS Running strip-commands
[--:--:--][766ffb3a][build:core-deps/gsound.bst          ] START   Caching Artifact
[00:00:00][766ffb3a][build:core-deps/gsound.bst          ] SUCCESS Caching Artifact
[00:01:37][766ffb3a][build:core-deps/gsound.bst          ] SUCCESS gnome/core-deps-gsound/766ffb3a-build.24545.log
[01:05:07][????????][ main:core/meta-gnome-core.bst      ] FAILURE Build failed
=============================================================================== 01:05:08 core/meta-gnome-core.bst (202/202) ===============================================================================
                                                                           (Tracked:167 28 0)→ (Fetched:0 195 0)→ (Built:102 0 2)                                                                           
                                                                                          ~~~~~ Active Tasks ~~~~~                                                                                          
[01:05:00][Track:core/glib-networking.bst] [01:05:00][Track:core/gnome-backgrounds.bst] [01:05:00][Track:core-deps/gcab.bst] [01:05:00][Track:core-deps/json-glib.bst]
[01:04:59][Track:core-deps/m4-common.bst ] [01:04:57][Track:core-deps/libsoup.bst     ] [01:04:49][Track:core-deps/vte.bst ] 

Pipeline Summary
  Total:       202
  Session:     202
  Track Queue: processed 167, skipped 28,  failed 0 
  Fetch Queue: processed 0,   skipped 195, failed 0 
  Build Queue: processed 102, skipped 0,   failed 2 

So BuildStream exited of it's own accord, with seemingly active tracking tasks.

However it seems to me that in this particular case, these tracking tasks completed a long time ago (they were active for over an hour), and the frontend failed to remove them from the status area and everything.

This would seem related to not being able to match which child process had exited, and it would seem that this is happening in the frontend (i.e. these pids are probably main task pids, not the children calling into git and whatnot).

In case this is useful, here is the output of bst show after the above weird thing happened, note that some elements were never tracked (but also have no failures printed in the summary).

      cached 4807cd8a base/base-system.bst 
      cached 3a90c6af base/base-configure.bst 
      cached f008f794 base/linker-priority.bst 
      cached 09450ab3 base/ninja.bst 
      cached 3450f8a4 base.bst 
      cached e53adc09 core-deps/gettext.bst 
      cached e4450b33 core-deps/yelp-xsl.bst 
      cached 33cb39be core-deps/yelp-tools.bst 
      cached 2d2cba4d core-deps/gtk-doc.bst 
      cached 491e11d9 core-deps/glib.bst 
      cached 14c89cda core-deps/gobject-introspection.bst 
      cached 9bf0d8db core-deps/meson.bst 
      cached 2af558d0 core-deps/gdk-pixbuf.bst 
      cached b594deac core-deps/gnome-common.bst 
      cached 458be2eb core-deps/libcroco.bst 
      cached 8ecd44f5 core-deps/harfbuzz.bst 
      cached bec093e0 core-deps/pango.bst 
      cached baf959e4 core-deps/vala.bst 
   buildable b7bac3c8 core-deps/librsvg.bst 
      cached 2338f66b core-deps/atk.bst 
      cached 6914a51d core-deps/wayland.bst 
      cached 0aba7b47 core-deps/wayland-protocols.bst 
      cached 7ca97794 core/at-spi2-core.bst 
      cached db251e7f core/at-spi2-atk.bst 
      cached a8f1dd38 core-deps/gtk+-3.bst 
      cached bc4ac8a6 core-deps/hicolor-icon-theme.bst 
     waiting 265c4439 core/adwaita-icon-theme.bst 
      cached 00ee6f0a core/dconf.bst 
      cached 68c3ac94 core/accountsservice.bst 
      cached de49b06f core-deps/libgudev.bst 
      cached b0a7ac9e core/upower.bst 
      cached baa3df43 core/gdm.bst 
      cached 780ad426 core/gsettings-desktop-schemas.bst 
no reference ???????? core/glib-networking.bst 
no reference ???????? core/gnome-backgrounds.bst 
no reference ???????? core-deps/gcab.bst 
no reference ???????? core-deps/json-glib.bst 
no reference ???????? core-deps/m4-common.bst 
no reference ???????? core-deps/libsoup.bst 
     waiting ???????? core-deps/appstream-glib.bst 
      cached aec55259 core-deps/libgusb.bst 
      cached 6dff5d2b core-deps/colord.bst 
      cached 4d046e9f core-deps/colord-gtk.bst 
      cached 1f25b80b core-deps/pcre2.bst 
no reference ???????? core-deps/vte.bst 
      cached 86e3828e core-deps/iso-codes.bst 
      cached bb6fc4bb core/gnome-desktop.bst 
     waiting ???????? core/gnome-color-manager.bst 
      cached 0c94b4c4 core-deps/libmbim.bst 
      cached 5f660ff5 core-deps/libqmi.bst 
      cached 282279ab core-deps/ModemManager.bst 
      cached e0835423 core-deps/gstreamer.bst 
      cached c97131e0 core-deps/gst-plugins-base.bst 
      cached b9bcec28 core-deps/cogl.bst 
      cached 0e956697 core-deps/libwacom.bst 
      cached f5f415c8 core-deps/libinput.bst 
     waiting ???????? core-deps/clutter.bst 
     waiting ???????? core-deps/clutter-gtk.bst 
      cached 48fce9e2 core-deps/libsecret.bst 
      cached 5f593c59 core-deps/brotli.bst 
     waiting ???????? core-deps/geoclue.bst 
      cached 3245244c core-deps/gst-libav.bst 
      cached 4b341dc8 core-deps/graphene.bst 
      cached fc4f1db1 core-deps/gst-plugins-bad.bst 
      cached 4a9a15d4 core-deps/gst-plugins-good.bst 
      cached 10597807 core-deps/libnotify.bst 
      cached 0d78f28c core-deps/woff2.bst 
     waiting ???????? core-deps/WebKit.bst 
     waiting ???????? core-deps/librest.bst 
      cached 4709c3d7 core-deps/telepathy-glib.bst 
     waiting ???????? core-deps/gnome-online-accounts.bst 
      cached 2022730d core-deps/gmime3.bst 
     waiting ???????? core-deps/totem-pl-parser.bst 
     waiting ???????? core-deps/grilo.bst 
      cached f23a48dd core-deps/ibus.bst 
      cached 8d35aece core-deps/ibus-anthy.bst 
      cached 5c0e8661 core-deps/ibus-hangul.bst 
      cached 583f8cf3 core-deps/libpinyin.bst 
      cached b2544a3f core-deps/ibus-libpinyin.bst 
      cached 2d44fd83 core-deps/libgnomekbd.bst 
      cached 374e4c55 core-deps/libgtop.bst 
      cached d7c1cb5e core-deps/sound-theme-freedesktop.bst 
      cached 159b9988 core-deps/py3cairo.bst 
     waiting ???????? core-deps/pygobject.bst 
     waiting ???????? core/NetworkManager.bst 
     waiting ???????? core-deps/clutter-gst.bst 
      cached fe6d01b2 core-deps/gnome-video-effects.bst 
     waiting ???????? core/cheese.bst 
      cached 6dea9759 core/gnome-bluetooth.bst 
     waiting ???????? core-deps/geocode-glib.bst 
     waiting ???????? core-deps/libgweather.bst 
     waiting ???????? core/PackageKit.bst 
     waiting ???????? core/gnome-settings-daemon.bst 
      cached 8b687ea9 core-deps/mobile-broadband-provider-info.bst 
     waiting ???????? core/network-manager-applet.bst 
     waiting ???????? core/gnome-control-center.bst 
      cached 91a6cb30 core/gnome-getting-started-docs.bst 
      cached 9848ff07 core-deps/gcr.bst 
      cached aa68dde8 core/gnome-keyring.bst 
     waiting ???????? core/gnome-initial-setup.bst 
      cached c2bc6721 core-deps/xtrans.bst 
     waiting ???????? core/gnome-session.bst 
     waiting ???????? core-deps/uhttpmock.bst 
     waiting ???????? core-deps/libgdata.bst 
     waiting ???????? core-deps/evolution-data-server.bst 
      cached 3f6357f9 core-deps/mozjs52.bst 
     waiting ???????? core-deps/gjs.bst 
      cached bd593fae core-deps/telepathy-logger.bst 
      cached d42ac01d core-deps/zenity.bst 
     waiting ???????? core/mutter.bst 
     waiting ???????? core/gnome-shell.bst 
      cached 30e87b95 core/gnome-menus.bst 
     waiting ???????? core/gnome-shell-extensions.bst 
      cached 1987179a core-deps/cantarell-fonts.bst 
     waiting a5ec6384 core/gnome-themes-standard.bst 
      cached 2635aad1 core/gnome-user-docs.bst 
      cached fd8c11f7 core-deps/gnome-autoar.bst 
     waiting ???????? core-deps/tracker.bst 
     waiting ???????? core-deps/libgrss.bst 
      cached fac4d0c0 core-deps/libgsf.bst 
     waiting ???????? core-deps/tracker-miners.bst 
     waiting ???????? core/nautilus.bst 
     waiting ???????? core/gnome-user-share.bst 
     waiting ???????? core/gvfs.bst 
      cached 1d1557b5 core/mousetweaks.bst 
     waiting ???????? core-deps/pyatspi2.bst 
      cached d00cc71e core-deps/speech-dispatcher.bst 
     waiting ???????? core/orca.bst 
     waiting ???????? core-deps/gssdp.bst 
     waiting ???????? core-deps/gupnp.bst 
     waiting ???????? core-deps/gupnp-av.bst 
      cached 1c521de5 core-deps/gupnp-dlna.bst 
      cached 53456982 core-deps/libgee.bst 
      cached ca7c052e core-deps/libmediaart.bst 
     waiting ???????? core/rygel.bst 
     waiting ???????? core/telepathy-mission-control.bst 
     waiting ???????? core/vino.bst 
     waiting ???????? core/meta-gnome-core-shell.bst 
     waiting ???????? core/baobab.bst 
     waiting ???????? core-deps/libpeas.bst 
     waiting ???????? core/eog.bst 
      cached 4f7c0a8a core-deps/libdazzle.bst 
     waiting ???????? core-deps/libhttpseverywhere.bst 
     waiting ???????? core/epiphany.bst 
      cached 33349f0a core-deps/libgxps.bst 
     waiting ???????? core/evince.bst 
      cached c3897792 core/file-roller.bst 
     waiting ???????? core-deps/gspell.bst 
     waiting ???????? core-deps/gtksourceview-3.bst 
     waiting ???????? core/gedit.bst 
      cached a10d813e core-deps/gtk-vnc.bst 
     waiting ???????? core-deps/libgovirt.bst 
      cached 111c9d24 core-deps/osinfo-db-tools.bst 
     waiting ???????? core-deps/libosinfo.bst 
      cached 50c2fa68 core-deps/libvirt-glib.bst 
     waiting ???????? core-deps/phodav.bst 
      cached f6468c7b core-deps/spice-protocol.bst 
     waiting ???????? core-deps/spice-gtk.bst 
     waiting ???????? core/gnome-boxes.bst 
     waiting ???????? core/gnome-calculator.bst 
     waiting ???????? core/gnome-calendar.bst 
     waiting ???????? core/gnome-characters.bst 
      cached 766ffb3a core-deps/gsound.bst 
     waiting ???????? core/gnome-clocks.bst 
     waiting ???????? core-deps/folks.bst 
     waiting ???????? core-deps/libchamplain.bst 
     waiting ???????? core/gnome-contacts.bst 
     waiting ???????? core/gnome-disk-utility.bst 
     waiting ???????? core-deps/libzapojit.bst 
     waiting ???????? core-deps/libgfbgraph.bst 
     waiting ???????? core-deps/gnome-online-miners.bst 
     waiting ???????? core-deps/libgepub.bst 
     waiting ???????? core/gnome-documents.bst 
      cached d468991b core/gnome-font-viewer.bst 
     waiting ???????? core/gnome-logs.bst 
     waiting ???????? core/gnome-maps.bst 
      cached e29ab9d6 core-deps/gom.bst 
     waiting ???????? core-deps/grilo-plugins.bst 
     waiting ???????? core/gnome-music.bst 
      cached f2c4a2ad core-deps/babl.bst 
     waiting ???????? core-deps/gegl.bst 
     waiting ???????? core/gnome-photos.bst 
   buildable a96fe7ec core/gnome-screenshot.bst 
     waiting ???????? core-deps/fwupd.bst 
     waiting ???????? core/gnome-software.bst 
      cached 2843dcaa core-deps/mm-common.bst 
      cached ed93f83b core-deps/libsigcplusplus-2.0.bst 
      cached 8ccd34ca core-deps/glibmm-2.4.bst 
      cached aa392123 core-deps/atkmm-1.6.bst 
      cached eff0170b core-deps/cairomm-1.0.bst 
      cached d13f49d9 core-deps/pangomm-1.4.bst 
      cached dd7eba30 core-deps/gtkmm-3.bst 
     waiting 2556691a core/gnome-system-monitor.bst 
     waiting ???????? core/gnome-terminal.bst 
     waiting ???????? core/gnome-todo.bst 
     waiting ???????? core/gnome-weather.bst 
     waiting ???????? core/simple-scan.bst 
     waiting ???????? core/sushi.bst 
     waiting ???????? core/totem.bst 
     waiting ???????? core/yelp.bst 
     waiting ???????? core/meta-gnome-core-utilities.bst 
     waiting ???????? core/meta-gnome-core.bst 

created branch 107-failing-child-processes-when-tracking-missing-git-branches

mentioned in commit 57bfce24

mentioned in issue #128 (closed)

mentioned in commit cf3ab8dd

mentioned in merge request !124 (merged)

mentioned in commit 6976ba5d

closed via merge request !124 (merged)

added High_impact and removed Critical labels