Elements with an open workspace cannot be fetched

Summary

It is impossible to fetch a source when its element has its workspace open.

This is a slightly obscure use-case because the normal invocation of bst workspace open expects the source to already be fetched, but that is not required if you use --no-checkout.

Steps to reproduce

From doc/examples/autotools:

  1. bst workspace open --no-checkout --directory hello-workspace hello.bst
  2. bst source fetch hello.bst

NOTE: If you have already fetched this source, you will need to point buildstream at an empty source cache to see this behaviour.

What is the current bug behavior?

  1. After the workspace is opened, but before bst source fetch is called, hello.bst is in state "waiting"
  2. In the logs for bst source fetch, hello.bst's dependency (base/alpine.bst) is fetched, but not hello.bst.

What is the expected correct behavior?

  • In the logs for bst source fetch, hello.bst is fetched as well

Relevant logs and/or screenshots

jonathanmaw@fafnir:~/workspace/buildstream/buildstream/doc/examples/autotools$ bst -c buildstream.conf source fetch hello.bst
[--:--:--][][] START   Fetch
[--:--:--][][] START   Loading elements
[00:00:00][][] SUCCESS Loading elements
[--:--:--][][] START   Resolving elements
[00:00:00][][] SUCCESS Resolving elements
[--:--:--][][] START   Resolving cached state
[00:00:00][][] SUCCESS Resolving cached state
[--:--:--][][] START   Checking sources
[00:00:00][][] SUCCESS Checking sources

BuildStream Version 1.3.0+1163.gc96afb86
  Session Start: Monday, 07-01-2019 at 15:45:36
  Project:       autotools (/home/jonathanmaw/workspace/buildstream/buildstream/doc/examples/autotools)
  Targets:       hello.bst

User Configuration
  Configuration File:      /home/jonathanmaw/workspace/buildstream/buildstream/doc/examples/autotools/buildstream.conf
  Log Files:               /tmp/buildstream/logs
  Source Mirrors:          /tmp/buildstream/sources
  Build Area:              /tmp/buildstream/build
  Artifact Cache:          /tmp/buildstream/artifacts
  Strict Build Plan:       Yes
  Maximum Fetch Tasks:     10
  Maximum Build Tasks:     4
  Maximum Push Tasks:      4
  Maximum Network Retries: 2

Pipeline
fetch needed fc05961cf6dfded6a2818cdbc603ff91e72e18d518a6315a19b106d85f1a42e3 base/alpine.bst 
     waiting b713a35f5efb5035e0cf7d9968e06aad7cc5b2e1296ed5abd0c776e1d00aa64d base.bst 
     waiting ???????????????????????????????????????????????????????????????? hello.bst Workspace: /home/jonathanmaw/workspace/buildstream/buildstream/doc/examples/autotools/hello-workspace
===============================================================================
[--:--:--][fc05961c][fetch:base/alpine.bst               ] START   autotools/base-alpine/fc05961c-fetch.5868.log
[--:--:--][fc05961c][fetch:base/alpine.bst               ] START   Fetching https://bst-integration-test-images.ams3.cdn.digitaloceanspaces.com/integration-tests-base.v1.x86_64.tar.xz
[00:00:14][fc05961c][fetch:base/alpine.bst               ] SUCCESS Fetching https://bst-integration-test-images.ams3.cdn.digitaloceanspaces.com/integration-tests-base.v1.x86_64.tar.xz
[00:00:14][fc05961c][fetch:base/alpine.bst               ] SUCCESS autotools/base-alpine/fc05961c-fetch.5868.log
[00:00:15][][] SUCCESS Fetch

Pipeline Summary
  Total:       3
  Session:     1
  Fetch Queue: processed 1, skipped 0, failed 0

Possible fixes

I gather that whether an element's source is fetched depends on whether its state shows up as fetch needed. As the element has an open workspace, it is perfectly possible to build without fetching the source.

I can see two possible solutions:

  • bst fetch will try to fetch even if the state isn't fetch needed.
  • Elements that do not have a fetched source, but do have an open workspace have a new state, and will be fetched by bst source fetch, but won't be fetched as part of a normal build.

Other relevant information