Fixing VCS reduction

Background

Currently we have a version of VCS reduction that unconditionally removes the VCS history. Unfortunately this results in certain projects breaking and it also blocks some future projects, for example dogfooding, which would require git history in order for it to be usable for development.

To fix this, i suggest the following:

The variable vcs-historybe added with a default value of drop This would allow for the default functionality to be that all VCS history is removed

The variable would also have the option of keep This would allow for projects or elements to retain all of their VCS history as this can be changed at both a project and element level.

In addition, the variables vcs-history-git and vcs-history-bzr would be added with the default value %{vcs-history} With the option to be independently assigned values.

In elements, any of those may be overridden.

The Source plugins for git and bzr, look for a history key in their source dictionary, if absent, the value %{vcs-history-git} or %{vcs-history-bzr} is used as appropriate.

If the resolved value is keep then behaviour is changed so that no VCS history is removed In this case, no entry to the cache key is made

If the resolved value is drop then behaviour is changed to that:

1. The cache key now contains an entry {"vcs-history":"drop"}
2. Where possible flags for clone/branch are adjusted to reduce the amount of data moved into that directory before removal. E.g. adding -s to the git clone.
3. Once the source is staged, the .git or .bzr directory is removed.

This approach means that we can have the removal of VCS history, without breaking projects which rely on having VCS history available. This can also be expanded on in the future to add a reduce or limit option, to allow for the reduction of VCS history instead of the complete removal of it. This could be handled in a similar fashion without causing significant issue since each time it is an active opt-in to the new behaviour which can be documented in the NEWS file.

Affected areas / concerns

1. It's possible that the variables (vcs-history etc) may need to be namespaced further in a bst "domain"?
2. This requires a number of global variables to be defined and validated
3. This requires changes to the bzr and the git Source handlers.

Task description

• Add vcs-history, vcs-history-git, vcs-history-bzr variables to projectconfig.yaml
• Set default of vcs-history-git and vcs-history-bzr to vcs-history
• Set default of vcs-history to drop
• Add functionality to remove the VCS history based on flags`
• If flag is not set to keep: add {"vcs-history":"drop"} to the cache key

Few comments:

• Do we have any evidence that bzr provides similar functionality as git describe, and that there are projects out there which call into bzr during their build phase ?
  • If not, we don't have any reason to do this for bzr
  • Also, as I suspect this is only going to be useful for git, is this a duplicate of #487 (closed) ?
• You are proposing a mode of keep, but we don't want this to be an option at all.
  • Rather, we are only interested in an option to stage the minimum amount of information required to satisfy git describe, probably as a boolean option.
  • We now have two working approaches prototyped in shell scripts by @richardmaw-codethink and @juergbi, let's prefer @juergbi's approach as it will not include any objects in the .git/ directory; later we can change this for @richardmaw-codethink's script if we find that there are problems with this, currently it's known to work though with a few git describe repositories.
• The configuration to opt-in to staging the minimum amount of history to satisfy git describe should be in the form of a configuration option of the git plugin proper; not a variable.
• The option to stage the minimal git history should come with a warning in the docs about the tag/branch state of the repository not being considered in the cache key calculation

Regarding versioning, the new config options to git will require a bump to BST_FORMAT_VERSION, and a documentation update to the git plugin to mention which format version the new config option was added in.

Shall we close this and move the relevant parts of this to #487 (closed) ?

The important parts of this are already captured in #487 (closed), closing this.

closed