Enable verbatim_patch_id feature flag by default

See merge request !7587



Merged-by: Mustafa Bayar <mbayar@gitlab.com>
Approved-by: Mustafa Bayar <mbayar@gitlab.com>
Approved-by: Timothy Schumacher <tschumacher@gitlab.com>
Co-authored-by: Marc Shaw <mshaw@gitlab.com>

Bundle-uri capability: Only advertise when bundle exist

See merge request !7572



Merged-by: Mustafa Bayar <mbayar@gitlab.com>
Approved-by: Toon Claes <toon@gitlab.com>
Approved-by: Mustafa Bayar <mbayar@gitlab.com>
Reviewed-by: Toon Claes <toon@gitlab.com>
Reviewed-by: Mustafa Bayar <mbayar@gitlab.com>
Co-authored-by: Olivier Campeau <ocampeau@gitlab.com>

recovery: Fix WAL entry tar extraction path

Closes #6605

See merge request !7582



Merged-by: James Liu <jliu@gitlab.com>
Approved-by: James Fargher <jfargher@gitlab.com>
Approved-by: James Liu <jliu@gitlab.com>
Reviewed-by: James Fargher <jfargher@gitlab.com>
Reviewed-by: James Liu <jliu@gitlab.com>
Co-authored-by: Mustafa Bayar <mbayar@gitlab.com>

go: Update module github.com/rubenv/sql-migrate to v1.7.1

See merge request !7518



Merged-by: James Liu <jliu@gitlab.com>
Approved-by: Divya Rani <drani@gitlab.com>
Approved-by: James Liu <jliu@gitlab.com>
Co-authored-by: GitLab Renovate Bot <gitlab-bot@gitlab.com>

Change patch-id generation type from stable to verbatim to treat whitespace changes as meaningful changes.

MR: gitlab.com/gitlab-org/gitaly/-/merge_requests/7587
Changelog: added

When a client performs a git-clone (SSH or Smarthttp)
Gitaly instructs the `git-upload-pack` command to
systematically advertise the bundle-uri capability.

This situation causes the Git client to request a
bundle-uri even when one does not exists, thus adding
unnecessary latency to the overall `clone` operation.
It also makes it hard to properly monitor the usage
of bundle-uri by clients; we can't correlate a request
for bundle-uri with an actual usage of it.

The proposed changes have the `git-upload-pack` command
advertise bundle-uri only when a bundle exists for the
given repository.

More context:

For SSH, this change does not provide much benefit since
the whole packfile negociation flow is performed on a single
connection. In other words, the flow starts when the client
starts the upload-pack service, and it ends when the client
received all the objects it needed.

For SmartHTTP, however, this is different. Each request/command
sent by the client (`ls-refs`, `bundle-uri`, `fetch`) is sent
on different, stateless, RPC calls. There is no way to know
what command the client is sending to the git-upload-pack process
until that process is actually started, which means that the Git
config injected into the process must be computed in advance for
every request; this implies computing the SignedURL of the bundle
if it exist. By advertising `bundle-uri` capability only
when a bundle exists for the given repository, we make sure
the client won't send a `bundle-uri` command if no bundle exist.

For SmartHTTP it not only reduces by 1 the number of round-trip
request and the number of Git config computation, but it also
makes it easier to monitor the use of `bundle-uri` feature, since
we can be sure that when a client sends the `bundle-uri` command,
it is because:
1. A bundle exists
2. The server advertised the capability
3. The client support the capability

References:
#6572

Rename bundleManager to bundleURIManager

See merge request !7573



Merged-by: Olivier Campeau <ocampeau@gitlab.com>
Approved-by: Toon Claes <toon@gitlab.com>
Approved-by: Eric Ju <eju@gitlab.com>

The name of the bundle URI manager is inconsistent
across the program and in some part not very
descriptive.

The bundle URI manager is now renamed to
`bundleURIManager`.

References:

This commit addresses the following comment:
!7542 (comment 2298693589)

We download the archived WAL entry, extract it to a temporary dir and
then finally move it to the partition's log directory. Currently this
results in repeated LSN number in the final path such as:
"/.../+gitaly/partitions/d4/73/2/wal/0000000000005/0000000000005/".

This happens because we are extracting the tar into a path such as
"/tempDirectory/0000000000005" but the tar file itself also contains
the directory information such as:

➜  tar -tvf 0000000000005.tar
drwxrwsrwx  0 git    git         0 Jan 22 12:28 0000000000005/
-rw-rw-rw-  0 git    git        41 Jan 22 12:28 0000000000005/1
-rw-rw-rw-  0 git    git       830 Jan 22 12:28 0000000000005/MANIFEST

And then finally we move it to its final destination in log with path
like "../wal/0000000000005" using os.Rename(srcPath, destPath), which
then results in final path with repeated LSN number.

Changes in this MR:
- Extracted the tar into a staging directory and then validated that
the log entry exists at the correct path. Afterwards passed the path
of the log entry directly to the log manager which fixes the issue.
- Update unit tests to use realistic tar file structure.

housekeeping: Add offloading bucket sink

Closes #6513

See merge request !7488



Merged-by: Eric Ju <eju@gitlab.com>
Approved-by: Olivier Campeau <ocampeau@gitlab.com>
Reviewed-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Reviewed-by: Eric Ju <eju@gitlab.com>
Reviewed-by: Olivier Campeau <ocampeau@gitlab.com>

To make sure offloading sink can working correctly and is capable to
deal with retry and cancellation under long delay, various test cases
are added for each each operation.

This commit adds tests to ensure proper configuration of sink options
when creating a bucket sink. The tests also validate that:
- Default values are correctly set.
- Special cases, such as WithNoRetry(), properly configure the
  configuration as expected.

We have retry and cancel support in the offloading sink. In order
test those, we have to mock a bucket that can simulate delay and error.

In this commit, we add a mock bucket. The bucket is based on fileblob
bucket but intercept its response with cuostizable delay and error.
With such simulation, we can write test cases to examine the
offloading sink's behavior under long delay and frequent error.

The offloading storage is a cloud object store where we can save
our offloaded object to. As part of repository offloading,
interacting with the offloading storage is required for operations
such as downloading, uploading, deleting, and listing.

This commit introduces an new sink implementation to serve as an
abstraction where we can interact with the offloading storage.

localrepo: Exercise `CloneBundle()` fsck configuration in test

See merge request !7579



Merged-by: James Liu <jliu@gitlab.com>
Approved-by: Emily Chui <echui@gitlab.com>
Approved-by: James Liu <jliu@gitlab.com>
Co-authored-by: Justin Tobler <jtobler@gitlab.com>

Starting in Git version 2.46.0, executing git-clone(1) on a bundle
performs fsck checks when `transfer.fsckObjects` is enabled. Prior to
this, this configuration was always ignored and fsck checks were not
run. Unfortunately, fsck message severity configuration is ignored by
Git only for bundle clones.

This issue is resolved in Git version 2.48.0, but until this gets rolled
out to GitLab, disable `transfer.fsckObjects` so bundles containing fsck
errors can continue to be cloned. This matches behavior prior to Git
version 2.46.0.

In Git versions prior to 2.46.0, performing git-clone(1) on a bundle
did not perform fsck checks even when `transfer.fsckObjects` was
enabled. Add a test case exercising an fsck error being caught when
cloning from a bundle.

[ci skip]

[ci skip]

[ci skip]

repository: Fix flaky `TestGetConfig`

See merge request !7577



Merged-by: James Liu <jliu@gitlab.com>
Approved-by: James Liu <jliu@gitlab.com>
Co-authored-by: Justin Tobler <jtobler@gitlab.com>

In 810ed5cf (git: Wire up Git v2.48 execution environment, 2025-01-14),
the `TestGetConfig` test was modified to support Git version 2.48. Using
a feature flag to guard to select the expected output doesn't work if
the bundled Git version get overwritten as done in the nightly CI
pipelines.

Fix this by checking the actual version of Git being used during test
execution.

Add the `IsGitVersionLessThan()` function to compare the Git version in
use compared to a specified version.

template: Add note to announce Git rollouts

See merge request !7574



Merged-by: Eric Ju <eju@gitlab.com>
Approved-by: Eric Ju <eju@gitlab.com>
Co-authored-by: Justin Tobler <jtobler@gitlab.com>

Updating Git versions in Gitaly can have widespread impacts. Add note in
the Git version rollout issue template to announce the change.

go-toolchain: Downgrade gitaly-init-cgroups version

See merge request !7569



Merged-by: Olivier Campeau <ocampeau@gitlab.com>
Approved-by: Sami Hiltunen <shiltunen@gitlab.com>
Approved-by: Eric Ju <eju@gitlab.com>

The gitaly-init-cgroups tool has an inconsistent
'toolchain' version in the go.mod file. All other
tools are on 1.22.6, but this one was on 1.22.8.

This MR downgrades the 'toolchain' version of this
tool to 1.22.6, so it is consistent with all other
tools and binaries of the project.

This change is also motivated by the fact that having
different go.mod files with inconsistent `toolchain'
versions across the same project can cause some version
conflicts when people uses IDE, which prevents building
and debugging.

backup: Confirm recovered log entries are being applied

See merge request !7555



Merged-by: Mustafa Bayar <mbayar@gitlab.com>
Approved-by: Divya Rani <drani@gitlab.com>
Approved-by: Mustafa Bayar <mbayar@gitlab.com>
Reviewed-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Reviewed-by: Divya Rani <drani@gitlab.com>
Co-authored-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>

Roll docs linting Docker image forward

See merge request !7567



Merged-by: Emily Chui <echui@gitlab.com>
Approved-by: Emily Chui <echui@gitlab.com>
Approved-by: Nao Hashizume <nhashizume@gitlab.com>
Co-authored-by: Evan Read <eread@gitlab.com>

Add nodejs back to .tool-versions

See merge request !7570



Merged-by: Stan Hu <stanhu@gmail.com>
Approved-by: Toon Claes <toon@gitlab.com>
Approved-by: Olivier Campeau <ocampeau@gitlab.com>

markdownlint-cli2 needs nodejs (npm specifically). Omitting
nodejs caused the GDK CI jobs to fail in
gitlab!178256.

ci: Extend nightly testing for reftables and SHA256

See merge request !7562



Merged-by: Emily Chui <echui@gitlab.com>
Approved-by: Emily Chui <echui@gitlab.com>
Co-authored-by: Justin Tobler <jtobler@gitlab.com>

Upgrade markdownlint-cli2 to 0.17.1

See merge request !7565



Merged-by: Emily Chui <echui@gitlab.com>
Approved-by: Emily Chui <echui@gitlab.com>
Co-authored-by: Evan Read <eread@gitlab.com>
Co-authored-by: Nao Hashizume <nhashizume@gitlab.com>

go: Update module github.com/git-lfs/git-lfs/v3 to v3.6.1

See merge request !7559



Merged-by: James Liu <jliu@gitlab.com>
Approved-by: James Liu <jliu@gitlab.com>
Co-authored-by: GitLab Renovate Bot <gitlab-bot@gitlab.com>

Remove cc:@andrashorvath from Production Training.md

See merge request !7563



Merged-by: John Cai <jcai@gitlab.com>
Approved-by: John Cai <jcai@gitlab.com>
Co-authored-by: Andras Horvath <ahorvath@gitlab.com>

Makefile: Build and install Git v2.48.0

See merge request !7560



Merged-by: Patrick Steinhardt <psteinhardt@gitlab.com>
Approved-by: James Liu <jliu@gitlab.com>
Approved-by: Karthik Nayak <knayak@gitlab.com>
Reviewed-by: Patrick Steinhardt <psteinhardt@gitlab.com>
Reviewed-by: James Liu <jliu@gitlab.com>
Co-authored-by: Justin Tobler <jtobler@gitlab.com>