Starting in Git version 2.46.0, executing git-fetch(1) on a bundle
performs fsck checks when `transfer.fsckObjects` is enabled. Prior to
this, this configuration was always ignored and fsck checks were not
run. Unfortunately, fsck message severity configuration is ignored by
Git only for bundle fetches.

Until this is supported by Git, disable `transfer.fsckObjects` so
bundles containing fsck errors can continue to be fetched. This matches
behavior prior to Git version 2.46.0.

In Git versions prior to 2.46.0, performing git-fetch(1) on a bundle
did not perform fsck checks even when `transfer.fsckObjects` was
enabled. Add a test case exercising an fsck error being caught when
fetching from a bundle.

go: Update module golang.org/x/sys to v0.26.0

See merge request !7340



Merged-by: Ahmad Sherif <ahmad@gitlab.com>
Approved-by: Emily Chui <echui@gitlab.com>
Approved-by: Ahmad Sherif <ahmad@gitlab.com>
Co-authored-by: GitLab Renovate Bot <gitlab-bot@gitlab.com>

Clean up dragonboat dependency

See merge request !7319



Merged-by: Divya Rani <drani@gitlab.com>
Approved-by: Divya Rani <drani@gitlab.com>
Reviewed-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Co-authored-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>

This commit removes configs that are deemed irrelevant to the new
etcd/raft library:
* RaftAddr: As the new library is quite flexible for the transportation
  layer, we'll use the Gitaly gRPC server for Raft. No need for a
  dedicated Raft address.
* InitialMembers: the new approach doesn't host a metadata group using
  Raft groups anymore. A node needs to request to join the cluster.
  Thus, no need to have a default set of peers.
* ReplicationFactor: we'll move the replication factor to partition or
  repository level.

A prior commit cleaned up all dragonboat-based components. This commit
removes dragonboat from go.mod.

Some notes:
* github.com/lni/vfs is a dependency of dragonboat. We patched it to
  fix a minor patch.
* github.com/HdrHistogram/hdrhistogram-go is a dependency of a handful
  of packages, notably labkit. Somehow after running `go mod tidy`, that
  package is mentioned in `go.mod`. It was only in `go.sum` before.

In the latest decision, we would like to pivot the etc/raft library as a
replacement for dragonboat. Those two libraries are absolutely different
in terms of philosophy and design. Also, as the project switched the
direction to bottom-up, we'll start by injecting etcd/raft directly into
the WAL transaction manager. Thus, it makes a lot of sense to remove
components built based on dragonboat at the moment. It doesn't make any
sense to keep two of them co-exist at the same time. Of course, we don't
want to throw everything away.

The history is preserved in Git so that it's trivial to cherry-pick
necessary components from the history later.

Support Template: Update issue triage link

See merge request !7342



Merged-by: James Liu <jliu@gitlab.com>
Approved-by: Evan Read <eread@gitlab.com>
Approved-by: James Liu <jliu@gitlab.com>
Co-authored-by: Evan Read <eread@gitlab.com>
Co-authored-by: Katrin Leinweber <kleinweber@gitlab.com>

check: Ensure unpacked binaries are cleaned up

See merge request !7334



Merged-by: Justin Tobler <jtobler@gitlab.com>
Approved-by: Justin Tobler <jtobler@gitlab.com>
Reviewed-by: James Liu <jliu@gitlab.com>
Reviewed-by: Justin Tobler <jtobler@gitlab.com>
Co-authored-by: James Liu <jliu@gitlab.com>

go: Update module golang.org/x/time to v0.7.0

See merge request !7341



Merged-by: John Cai <jcai@gitlab.com>
Approved-by: John Cai <jcai@gitlab.com>
Co-authored-by: GitLab Renovate Bot <gitlab-bot@gitlab.com>

tools/gotestsum: Update module gotest.tools/gotestsum to v1.12.0

See merge request !6963



Merged-by: John Cai <jcai@gitlab.com>
Approved-by: Sami Hiltunen <shiltunen@gitlab.com>
Approved-by: John Cai <jcai@gitlab.com>
Co-authored-by: GitLab Renovate Bot <gitlab-bot@gitlab.com>

[ci skip]

[ci skip]

[ci skip]

Remove redundant fsyncs

See merge request !7325



Merged-by: Emily Chui <echui@gitlab.com>
Approved-by: Emily Chui <echui@gitlab.com>
Reviewed-by: Sami Hiltunen <shiltunen@gitlab.com>
Reviewed-by: Justin Tobler <jtobler@gitlab.com>
Reviewed-by: Emily Chui <echui@gitlab.com>
Co-authored-by: Sami Hiltunen <shiltunen@gitlab.com>

It's hard to tell what the problem is as we have two identical error
messages, and don't emit the name of the offending directory.

The `gitaly check` command instantiates a GitLabHookManager to verify
that Gitaly is able to access Rails' /internal/check endpoint. The hook
manager requires an ExecCommandFactory which requires Git binaries to be
configured properly. Due to this, we must unpack the embedded Git
binaries into `cfg.RuntimeDir` when this command is executed.

There were three problems with the implementation:

- We didn't clean up the runtime directory, so multiple invocations of
  the command would fail when it tries to unpack binaries into an
  existing location.
- We use `cfg.RuntimeDir` directly as configured in config.toml. We
  should instead be creating subdirectories like we do elsewhere.
- We didn't clean up leftover runtime directories, which may exist if
  a previous process crashed.

Address these problems by unpacking binaries into a subdirectory of
`cfg.RuntimeDir`, and ensure the subdirectory is cleaned up at the
beginning and end of execution. We use `config.SetupRuntimeDirectory()`
which is the same mechanism used by the sever initialisation.

repository: Improve git clone error handling

Closes #6411

See merge request !7332



Merged-by: Toon Claes <toon@gitlab.com>
Approved-by: Justin Tobler <jtobler@gitlab.com>
Approved-by: Toon Claes <toon@gitlab.com>
Reviewed-by: Toon Claes <toon@gitlab.com>
Reviewed-by: Justin Tobler <jtobler@gitlab.com>
Reviewed-by: Mustafa Bayar <mbayar@gitlab.com>
Co-authored-by: Mustafa Bayar <mbayar@gitlab.com>

We were previously fsyncing outside of the lock block in startPartition.
This led to fsyncing on every request. We're now only fsyncing when
setting up the partition but this leads to doing a potentially longer
running operations in the lock block. Holding the lock while the fsync
is running blocks other unrelated partitions from being retrieved
concurrently.

This commit adds additional synchronization to ensure multiple partitions
can be started concurrently. The partitions start in uninitialized state.
The goroutine starting the partition signals to the other goroutines waiting
for the partition to start when it has finished setting it up. This allows
it to release the lock while setting up the state directories and fsyncing
them.

Move the partition reference counting out from newPartitionHandle.
It will make it easier in later commits to not wait for the
partition to be fully set up while holding the lock.

partititon type has closed and partitionClosed channels. The naming
is confusing, so rename with a more descriptive name.

startPartition is currently attempting to create the partition's
state directory every time a partition handle is being retrieved.
Doing so, it also fsyncs the state directory and the parent
directories it possibly created. This leads to every single request
running three unnecessary fsync operations which leads to bad
performance. Some file systems flush more than just the file being
flushed. Flushing would also end up locking the pages while they are
flushed out. This leads to delays in operations that typically would
be fast such as linkat and mkdir.

Attempt to set up the partition's state directory only if it isn't
running yet.

Begin used to be the place where a partition was started. As it was
responsible for setting up a partition, it created its state
directory and fsynced it. Partition starting logic has since been
moved to startPartition, which also creates the partition's state
directory and fsyncs it. Doing this twice is redundant, and affects
performance with file systems that flush out more writes than just
the fsynced file itself. Remove the redundant logic.

Mark SSHUploadPack RPC as deprecated

See merge request !7324



Merged-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Approved-by: Ahmad Sherif <ahmad@gitlab.com>
Approved-by: James Fargher <jfargher@gitlab.com>
Reviewed-by: James Fargher <jfargher@gitlab.com>
Reviewed-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>

In command options, we have two paths: one with sidechannel and one
without it. The without version doesn't have any caller. This commit
unifies them and uses sidechannel.

SSHUploadPackWithSidechannel has been widely adopted. That version
uses Sidechannel protocol, which was an optimization to address
significant performance problems in the past. At the point of writing,
all known clients migrated to the *sidechannel version. There's also no
traffic on GitLab SaaS.

This commit marks that RPC as deprecated to prepare for future removal.

Issue: #6410
Changelog: deprecated

cgroups: Provide existing manager to ExecCommandFactory

See merge request !7333



Merged-by: John Cai <jcai@gitlab.com>
Approved-by: Justin Tobler <jtobler@gitlab.com>
Approved-by: John Cai <jcai@gitlab.com>
Reviewed-by: Justin Tobler <jtobler@gitlab.com>
Co-authored-by: James Liu <jliu@gitlab.com>

gitaly-backup: Add partition backup command

Closes #6272

See merge request !7300



Merged-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Approved-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Reviewed-by: James Fargher <jfargher@gitlab.com>
Reviewed-by: Quang-Minh Nguyen <qmnguyen@gitlab.com>
Reviewed-by: Mustafa Bayar <mbayar@gitlab.com>
Co-authored-by: Mustafa Bayar <mbayar@gitlab.com>

We're going to be adding another test function to check_test.go. Since
the test server initialisation logic is already duplicated, let's
extract it into its own function first.

We initialise a cgroup manager during server initialisation, but we
don't provide it to the constructor of the ExecCommandFactory. This
causes us to initialise two instances of the cgroup manager; once in the
server initialisation, and again within the factory's constructor.

Creating cgroup managers is an idempotent operation, which is why it's
been working fine as-is, but it does waste a lot of cycles. Fix this by
providing the outer cgroup manager to the factory constructor.

CreateRepositoryFromURL RPC returns generic git command error
when remote repository is not found. This causes issues on Geo
to not recognize the error correctly. We should interpret the
not found errors from git clone command correctly to Gitaly's
structured not found error.

Introduce new error detail to be used in CreateRepositoryFromURL RPC

go: Update module golang.org/x/text to v0.19.0

See merge request !7330



Merged-by: Justin Tobler <jtobler@gitlab.com>
Approved-by: Justin Tobler <jtobler@gitlab.com>
Co-authored-by: GitLab Renovate Bot <gitlab-bot@gitlab.com>