Roll out new changelog approach to all GitLab projects
Table of contents
- Introduction
- High-level strategy
- Editing commits to add trailers
- Rollout date
- Tasks
-
Frequently asked questions:
- Why are we making this change?
- Why don't we keep using files as input?
- Why not use merge requests as input?
- Why not use conventional commits instead of Git trailers?
- How do I add the Git trailer to an existing commit?
- How we we correct typos in commits?
- What if a community contribution is missing the trailer, but the author doesn't know how to edit commits?
- How do I make sure my changelog entry includes a link to its merge request?
- How do I mark a changelog entry as being specific to GitLab EE?
- How do I override the merge request to link to?
- Are we planning on adding support for using merge requests as input?
- Aren't commit messages meant for developers, while changelog entries are meant for users?
- Can other metadata be added using Git trailers?
- Should we not be rolling this out to other projects first?
- What values can I use in the Changelog trailer?
- Where can I read more about how this works?
Introduction
We are in the process of rolling out a new way of managing and generating changelogs at GitLab, starting with GitLab Pages, Omnibus and Gitaly. Once rolled out to those projects, we want to roll out this setup to all remaining projects.
When we decide on a time frame for the roll out, we'll roll out the new process to all projects at once; instead of doing it incrementally. This makes it easier to work on the projects, as you don't have to switch between different workflows until all projects have adopted the new approach.
For this migration we'll only focus on projects that Release Tools is aware of, which are the projects most developers are likely to work on. We have many other projects with changelogs, but migrating all those over would be too much work. We hope these projects will at some point move over themselves.
High-level strategy
We will pick a date to roll out the new setup. This will likely be on the 22nd
(after the release has been tagged), so we don't have to convert any existing
YAML changelog files. From this point forward, the old approach is no longer in
use, and merge requests introducing changes in changelogs/unreleased
are to be
rejected.
Should we need to convert the YAML files, we can do so using the following script:
script.rb
# frozen_string_literal: true
require 'yaml'
require 'shellwords'
commit_file = '/tmp/commit.txt'
Dir['{./ee/changelogs/unreleased/*.yml,./changelogs/unreleased/*.yml}'].each do |path|
data = YAML.load_file(path)
meta = `git log --format="%h\t%ad\t%an <%ae>" #{path}`.split("\n").last
sha, date, author = meta.split("\t")
if data['merge_request']
message = <<~MSG.strip
#{data['title']}
This change was first introduced in merge request
https://gitlab.com/gitlab-org/gitlab/-/merge_requests/#{data['merge_request']}.
Changelog: #{data['type']}
MSG
else
message = <<~MSG.strip
#{data['title']}
This change was first introduced in commit #{sha}.
Changelog: #{data['type']}
MSG
end
if path.include?('/ee/')
message += "\nEE: true"
end
File.open(commit_file, 'w') do |file|
file.write(message)
end
if system("git rm #{path}")
system(%Q{git commit -F #{commit_file} --author "#{author}" --date "#{date}"})
else
abort("Failed to remove #{path}")
end
File.unlink(commit_file)
end
Editing commits to add trailers
To make it easier to add trailers to existing commits, we can turn the following script into something official:
script.rb
# frozen_string_literal: true
require 'yaml'
require 'open3'
require 'optparse'
require 'securerandom'
def git(cmd, *args)
output, status = Open3.capture2e('git', cmd, *args)
unless status.success?
abort "Failed to run `git #{cmd}`: #{output}"
end
output.strip
end
category = nil
options = OptionParser.new do |o|
o.banner = "Usage: #{__FILE__} [OPTIONS] [SHA] [CATEGORY]"
o.separator("\nOptions:\n")
o.on('-h', '--help', 'Shows this help message') do
puts(o)
exit
end
end
_, status = Open3.capture2e('git', 'rebase', '--show-current-patch')
if status.success?
abort 'A git rebase is already taking place. You must either finish or ' \
'cancel any existing Git rebase sessions'
end
unless git('status').include?('nothing to commit')
abort 'You must commit all your changes before you can run this command'
end
argv = options.parse!(ARGV)
config_path = File.expand_path('.gitlab/changelog_config.yml')
unless File.file?(config_path)
abort "The changelog configuration file #{config_path} doesn't exist"
end
sha = argv[0]
category = argv[1]
unless sha
abort "You must specify a SHA of a commit as the firstr argument"
end
unless category
abort "You must specify a changelog category as the second argument"
end
config = YAML.load_file(config_path)
unless config['categories']
abort 'No changelog categories are configured'
end
unless config['categories'][category]
abort "The changelog category #{category.inspect} isn't supported. " \
"The supported values are: #{config['categories'].keys.join(', ')}"
end
branch = git('rev-parse', '--abbrev-ref', 'HEAD')
if branch == 'master' || branch == 'main'
abort "This script can't be run on the project's default branch"
end
# To ensure the proper newlines are always used, we strip out any Windows
# newlines (just in case they are present).
old_message = git('log', '-1', '--format=%B', sha)
.gsub("\r\n", "\n")
.gsub("\r", "\n")
new_message = git(
'interpret-trailers',
'--trailer',
"Changelog=#{category}",
'--if-exists=replace',
stdin_data: old_message
)
# Sometimes a user may forget to include an empty line between the subject and
# body. For example, they may have a commit like so:
#
# Add fried chicken wings
# Fixed: 123
#
# If we don't add an empty line after the subject, Git will add the trailer like
# so:
#
# Add fried chicken wings
# Fixed: 123
# Changelog: added
#
# But if we then try to parse trailers we won't find it, because the subject
# isn't separated from the body.
#
# To solve this, we always add an empty line after the subject; unless one is
# already present.
new_lines = new_message.split("\n")
if new_lines[1] && !new_lines[1].empty?
new_lines.insert(1, '')
new_message = new_lines.join("\n")
end
tmp_branch = "changelog/#{SecureRandom.hex(3)}"
git('checkout', '-b', tmp_branch, sha)
git('commit', '--amend', '-F', '-', '--allow-empty', stdin_data: new_message)
git('rebase', '--rebase-merges', '--onto', tmp_branch, sha, branch)
git('checkout', branch)
git('branch', '-D', tmp_branch)
Rollout date
May 22nd, 2021
Tasks
Preparing
-
Determine what release is the first release to use this new setup -
Add changelog config for https://gitlab.com/gitlab-org/gitlab (with support for EE only changes) -
Add changelog config for https://gitlab.com/gitlab-org/gitaly -
For projects that use CHANGELOG
instead ofCHANGELOG.md
, modify the former to notify viewers that we have moved to the latter -
Set up MR that updates the GitLab documentation for this new workflow -
Set up FAQ in the documentation to cover questions we expect to be asked frequently -
Document the use of the EE: true
trailer for EE specific changes -
Update CHANGELOG-EE.md
so it contains a note about the file no longer being updated -
Devise a way to block merge requests from introducing new YAML changelog files after the cutover date
Announce
-
Announce in Slack -
Announce in the engineering week in review -
Ask managers to discuss this in their weekly/relevant team meetings -
Post another reminder in the week of May 12
Frequently asked questions:
Why are we making this change?
To simplify the process of producing changelog entries and converting these into a changelog file, in a way that all GitLab users can benefit from.
Why don't we keep using files as input?
First, so users don't need to write similar information twice: once in the commit message, and once in the changelog input file.
Second, it makes it easier to automatically associate extra information. For example, in the new commit based approach you don't need to ammend the merge request ID into any files; GitLab instead determines the merge request automatically based on the commit.
Why not use merge requests as input?
Not all projects that use GitLab also use merge requests (as heavily as we do). This means such projects wouldn't be able to benefit from a changelog feature built into GitLab. But all projects that use GitLab do use Git commits.
In addition, we need a way that supports our security mirror setup. If we were to use merge requests, we'd only be able to include merge requests from either the public project, or the security mirror; not both.
Using commits as input should hopefully also lead to more well written commit messages, something we've been wanting for a long time as this aids the development and debugging workflows.
More information on this can be found in epic &351 (closed).
conventional commits instead of Git trailers?
Why not useGit trailers are supported by Git natively, the conventional commit workflow is not. This means no extra tools are necessary to work with Git trailers.
In addition, the use of Git trailers means one doesn't have to pollute the commit subject with various tags. Instead, the trailer tags go at the end of the commit message. This makes it easier to read commit messages, especially when following the common commit guidelines such as those listed here.
How do I add the Git trailer to an existing commit?
If the commit is the last commit, you can simply use git commit --amend
. If an
earlier commit needs to be modified, you have to use git rebase -i
to rebase
that commit and the commits that follow it. We also have an experimental script
for this
here,
but it's not clear yet if we want to make this an official script.
How we we correct typos in commits?
In general, we don't/can't. If absolutely needed, we can do the following:
- Revert the commit(s) to edit
- Revert the revert, using the proper commit message (based on the original message)
This works because our changelog API ignores commits that are both added and reverted in the same release.
Fortunately, we expect this scenario to be rare. For example, for Omnibus we found the need for this to be very rare: #1551 (comment 519745360).
In general we can also combat this by reviewing commit messages similar to reviewing changes, something we should be doing anyway.
What if a community contribution is missing the trailer, but the author doesn't know how to edit commits?
There are two options:
- Have a reviewer/maintainer commit to the merge request to correct the commit messages
- Edit the merge commit message to include the trailer (and a readable subject line), leaving the individual commits as-is
Generally it's best to first ask the author to correct their commits, only resorting to the above two options when needed.
How do I make sure my changelog entry includes a link to its merge request?
You don't need to do anything for this, as GitLab automatically determines what merge request introduced a changelog entry.
How do I mark a changelog entry as being specific to GitLab EE?
Add EE: true
as a trailer in addition to the Changelog
trailer. For example:
Commit subject line
Commit body goes here.
Changelog: added
EE: true
How do I override the merge request to link to?
Add the MR
trailer and set its value to the full URL of the merge request. For
example:
Commit subject line
Commit body goes here.
Changelog: added
MR: https://gitlab.com/foo/bar/merge_requests/-/123
Are we planning on adding support for using merge requests as input?
No, support for additional input sources (merge requests, issues, etc) is not planned.
Aren't commit messages meant for developers, while changelog entries are meant for users?
There is no reason we can't use commit messages/subjects for both groups. In fact, as almost all our work is public there's a big chance non developers are also reading our commits.
It's not difficult to write a commit subject readable to those who don't have deep knowledge of the underlying changes. Even when they do, a helpful commit subject can make it easier to quickly understand what's going on.
Here are some examples of what good subject lines (that go in a changelog) should look like:
- gitlab-org/omnibus-gitlab@d7db53d0
- gitlab-org/omnibus-gitlab@a9de7c8f
- gitlab-org/omnibus-gitlab@1b0c2ce3
Can other metadata be added using Git trailers?
Yes, though we currently don't use trailers other than the Changelog
, EE
and
MR
trailers. In the future we could extend this to allow linking to additional
data, such as epics or issues; or we could derive that from the merge request.
None of this is planned at this time.
Should we not be rolling this out to other projects first?
We have! We started with GitLab Pages (#1525 (closed)) followed by Omnibus (#1551 (closed)). Rolling this out per project is time consuming, as each rollout takes at least one release cycle. As such we are rolling out the remaining projects together.
What values can I use in the Changelog trailer?
The following values are currently supported:
- added
- fixed
- changed
- deprecated
- removed
- security
- performance
- other
These values are the same as the ones used in the YAML based changelog approach.
Where can I read more about how this works?
https://docs.gitlab.com/ee/api/repositories.html#generate-changelog-data