- Dec 05, 2024
-
-
Sami Hiltunen authored
Transaction manager should focus on conflict checking concurrent transactions. Transactions are expected to be correct in isolation so doing extra validation on them is not necessary. Remove the validation logic around alternates. We remove tests that were covering validation failures. Tests that are exercising concurrent behaviors were updated to assert the conflict.
-
Sami Hiltunen authored
Alternate linking operation have so far been recorde in transaction manager as we didn't have a general API for staging file system operations in transactions. As we have such an API now, move the linking operation to be recorded in the handler. We reuse the logic from the handler in the tests and remove the custom linking logic.
-
Sami Hiltunen authored
Split the ObjectPool.Link method into a function so it can be called without having an object pool at hand. This will be used in transaction manager's tests as the object pool constructor enforces certain format for relative paths.
-
- Sep 04, 2024
-
-
Sami Hiltunen authored
storagectx package was a workaround that was needed in the past as the 'git' package would have had cyclic depencies with the 'storage' pacakge if we define the Transaction interface there. We've since restructured the 'git' package to solve the structural issue there and moved the Transaction interface into the 'storage' package. There's no reason for the storagectx package to exist anymore. This commit merges the package into the 'storage' package to simplify.
-
Sami Hiltunen authored
RunWithTransaction runs a callback with a transaction found in the context. This is somewhat an awkward interface with the current uses. Sometimes we want to actually extract the transaction from the context, not just do things with it. Doing so currently requires declaring a variable outside of the callback and assigning into it within the callback. Sometimes we need to return an error from the callback. As we don't support propagating an error, the error variable needs to be declared outside of the callback, assigned in the callback, and then checked outside of the callback. This commit replaces the function with ExtractTranscation that simply extracts the transaction from the context. This is still convenient for uses when we just need to call something on the transaction but is a lot more convenient for the above two cases. For the first case, we can now simply extract the transaction into the target variable. For the second case, we can simply return directly from the if block if an error occurs.
-
Sami Hiltunen authored
There was previously a cyclic dependency between the 'git' and 'storage' packages. We've solved this now by splitting the implementation related bits out from the 'git' package and only type definitions and interfaces there. With the cycle out of the way, we can now define the Transaction interface in a single location and reuse it across all the sites that previously were using one of the other interface definitions. This commit defines the interface of a Transaction in the storage package. We also replace duplicate definitions of the interface in other packages.
-
- Jul 01, 2024
-
-
Sami Hiltunen authored
Transactions currently rely on rewriting the relative path of a repository to implement snapshot isolation. This has the downside that the relative paths get a prefix such as '+gitaly/staging/xx/yy/...'. For large parts of the application this is not a problem the handlers are generally just accessing a repository at a given path. This is a leaky abstraction though as the prefix is not part of the relative path and it is visible to the handlers. This has an impact on code that operates directly on the relative path in some manner. This could be for example logging, using the relative path as a key, or operating directly on file paths. The rewriting could be implemented transparently if we instead rewrite the storage directory's root path as seen by the transaction. Storage directory's path doesn't have a meaning unlike the relative path which is used to identify a repository. This commit takes the first step towards that by wiring context into GetStorageName of Locator. The injected context allows us to later plug transaction scoped logic into Locator that returns the rewritten storage root of a transaction when attempting to access a given storage. This way we can wire the indirection through the entire application without having to touch each site generating storage paths separately. The commit is large as we access locator to generate storage paths in many places, and we have to drill the context down into each of them. The commit simply wires the context through all call sites into GetStorageName.
-
- Apr 04, 2024
-
-
Justin Tobler authored
Both `objectpool.FromRepo()` and `objectpool.LinkedToRepository()` use `getAlternateObjectDir()` to read the alternates file. Refactor the code to use `stats.AlternatesInfoForRepository()` instead because it is more robust and simplifies error handling. The only behavior change is that `objectpool.FromRepo()` is slightly more strict and returns an error if a repository contains an empty alternates file. Previously it would only return an error if the alternates file was missing or malformed. An empty alternates file should be treated as invalid and return an error in the same manner.
-
- Feb 20, 2024
-
-
Sami Hiltunen authored
Transaction currently provides a method to stage an alternate update into the transaction. The path is the relative path of the repository that we want to link the alternate to. This commit simplifies the external API to just marking that the file was updated. This signals to the transaction manager to check the file and to stage any updates. This simplified API makes it easier to move the alternate updates to be logged through the physical logging protocol by simply logging the staged alternate file of the transaction.
-
- Dec 05, 2023
-
-
Sami Hiltunen authored
This commit enables transactions for LinkRepositoryToObjectPool. This requires just recording the updated alternate link in the transaction. Transactions require the object pool and the member repository to be in the same partition. This may not be the case when Praefect attempts to reconciliate alternate link differences during replication. Moving repositories between partitions is difficult, so for now we'll simply ignore the pool linking failure in Praefect and leave accept deviating alternate links.
-
- Sep 13, 2023
-
-
Justin Tobler authored
If a repository is already linked to the object pool, cast a vote to ensure that repository state is consistent among replicas.
-
- May 10, 2023
-
-
Patrick Steinhardt authored
We're about to release Gitaly v16.0. As we've landed a bunch of previously announced removals it's thus time to bump our Go module version from v15 to v16.
-
- Dec 13, 2022
-
-
Patrick Steinhardt authored
Object pools provide two ways to calculate the path of a pool: `FullPath()` returns the path without verifying whether the pool exists, while `Path()` verifies that the target exists. The latter is what we already use in the context of the `localrepo.Repo` package, and it gives us more safeguards to ensure that the object pool really exists when it ought to exist. Refactor the code to use `Path()` instead of `FullPath()`.
-
- May 20, 2022
-
-
John Cai authored
This commit changes the major version in the package name from v14 to v15 Updating go.mod & go.sum with new module name v15 Update Makefile to bump major version to v15 Update the gitaly package name in the Makefile. Also update gitaly-git2go-v14 -> gitaly-git2go-v15. We need to keep gitaly-git2go-v14 for a release however, for zero downtime upgrades. This pulls directly from a sha that is v14. Update package name from v14->v15 for auth, client, cmd, internal packages This commit changes the package name from v14 to v15 in go and proto files in the internal, auth, client, cmd packages. proto: Update major package number in package name tools: Change major version number in package name from v14 to v15 gitaly-git2go: Change the package name from v14 to v15 update module updater for v15 Update the documentation for the module updater to reflect v15
-
- Feb 07, 2022
-
-
Patrick Steinhardt authored
Convert the objectpool package to retrieve repository paths via the localrepo interface.
-
- Nov 03, 2021
-
-
Patrick Steinhardt authored
The UnlikRepostioryFromObjectPool RPC is not used by anything anymore, and the implementation from it is dangerous given that it doesn't actually unlink a repository from its object pool: it only tries to remove a remote named after the pool's project path, which we wouldn't ever create in the first place. The RPC has thus been deprecated in release v14.3. Remove the RPC and its backing code. Changelog: removed
-
- Oct 26, 2021
-
-
Patrick Steinhardt authored
This reverts commit 369fbdb1 (Merge branch 'pks-objectpool-drop-ondisk-remotes' into 'master', 2021-10-25), which introduced a regression in Rails' test suite.
-
- Oct 25, 2021
-
-
Patrick Steinhardt authored
The UnlikRepostioryFromObjectPool RPC is not used by anything anymore, and the implementation from it is dangerous given that it doesn't actually unlink a repository from its object pool: it only tries to remove a remote named after the pool's project path, which we wouldn't ever create in the first place. The RPC has thus been deprecated in release v14.3. Remove the RPC and its backing code. Changelog: removed
-
- Oct 11, 2021
-
-
Patrick Steinhardt authored
With 159948cc (Merge branch 'pks-tx-extended-file-locking' into 'master', 2021-09-20), we have introduced extended file locking such that modifications to on-disk files will always lock, vote on and then commit the change in a racefree way when transactions are enabled. The code has been enabled in production now without any observed issues and can thus be deemed stable. Remove the feature flag such that we always use extended file locking. Changelog: fixed
-
- Sep 20, 2021
-
-
Patrick Steinhardt authored
With Go 1.16, the ioutil package was deprecated. In addition to being moved into the os package, `ioutil.ReadDir()` was also changed to not stat(3P) all dir entries anymore. As a result, the caller now has to do so manually. This is a performance improvement in some cases where the caller didn't require any of the file information, but really only wanted to read the directory's entries. Adapt trivial usecases of `ioutil.ReadDir()` which do not require this information with usage of `os.ReadDir()`. This leaves a few callsites of the old `ioutil.ReadDir()` function for future conversion.
-
Patrick Steinhardt authored
With Go 1.16, the ioutil package was deprecated. Replace our usage of `ioutil.TempFile()` with `os.CreateTemp()` to adapt accordingly.
-
- Sep 17, 2021
-
-
Patrick Steinhardt authored
When linking a repository into an object pool, then we must write the alternates file to point to the object pool's object storage such that it reuses objects from it. This is currently done in a non-transactional way and can thus easily race with concurrent updates of the same gitconfig. Convert it to use our new locking file writer and voting infrastructure to make this update race-free. Changelog: fixed
-
- Aug 09, 2021
-
-
Pavlo Strokov authored
The change removes os.Remove from lint rules exclusion and fixes all the places where it causes the issue. If removal ends up with an error we try to log it if the logger is accessible. Otherwise the error is just omitted. We don't return it back to the caller because we don't want functional changes here.
-
- May 27, 2021
-
-
Pavlo Strokov authored
The new "v14" version of the Gitaly module is named to match the next GitLab release. The module versioning is needed in order to pull gitaly as a dependency in other projects. The change updates all imports to include v14 version. The go.mod file was modified as well after go mod tidy execution. And the changes in dependency licenses are reflected in the NOTICE file. Part of: #3177
-
- Feb 17, 2021
-
-
Patrick Steinhardt authored
Now that our localrepo interface provides all functionality of the remote package, let's convert callsites to use the localrepo interface instead and remove the now-unused remote package. While at it, this commit also moves over a testcase for deletion of local branches to ensure that test coverage doesn't decrease because of the package deletion.
-
- Feb 16, 2021
-
-
Pavlo Strokov authored
The change replaces all usages of the git.NewCommand function with git.CommandFactory.New method. The New method of the ExecCommandFactory changed to use newCommand method to execute git commands. The new parameter was added to some functions to pass git.CommandFactory down to the place where git.NewCommand was used before. Part of: #2699
-
- Jan 25, 2021
-
-
Patrick Steinhardt authored
Traditionally, the gitconfig is not managed by the Gitaly project but by the various projects which deploy Gitaly or by the administrator himself if he chooses to do a source-based installation. Admins may have to potentially update the gitconfig on each upgrade of Gitaly while we as the Gitaly team need to ensure that all projects which deploy Gitaly (Docker, CNG, Omnibus and the likes) have the same gitconfig. Going by Murphy's law, this means that the git configuration for ways of installing Gitaly is going to diverge or has already diverged. Running in an environment where we don't know that a set of config entries is set can be dangerous and may lead to incorrect results, failures or even data loss in the case of "core.fsyncObjectFiles" and "gc.auto". This commit thus moves the global configuration into the Gitaly project to avoid any inconsistent environments. This allows us to ensure that required options are set while being quicker to iterate in case we need to change the configuration. The injection is done via the `git.ConfigPair{}` mechanism, which will put all config entries on the command line via `git -c <key>=<value>` pairs. While it has the downside of being verbose, the current set of entries we inject is quite limited and contains only five different options, so we shouldn't be near any command line limits. Furthermore, we're currently in the process of upstreaming a new way of injecting config entries via the environment, which we can migrate to as soon as this git version has been released. The list of injected configuration entries is taken from our source installation instructions.
-
- Jan 14, 2021
-
-
Patrick Steinhardt authored
Back when we introduced the Git DSL, we still had conflicting sets of safe and unsafe functions. Because of this legacy, our safe set of functions is still has the "Safe" prefix. This commit now ends that chapter and renames `SafeCmd()`. In alginment with the preceding renames, this is being renamed to `NewCommand()`.
-
- Dec 16, 2020
-
-
Pavlo Strokov authored
The catfile package directly or indirectly uses config.Config value. And as we are about to remove config.Config from the code we need to break this dependency and substitute with other abstractions. For now those are storage.Locator and alternates.Env instead of the old alternates.PathAndEnv that depends on the global var. It was decided to extend constructor function of the catfile to accept storage.Locator as a dependency as it is already injected in most of the "server"s which handle requests. The other option could be creation of the "factory", but it will require a new dependency to be injected into all existing services which is a lot more changes and adds currently unnecessary abstraction that needs to be managed. Part of: #2699
-
- Dec 14, 2020
-
- Dec 11, 2020
-
-
Stan Hu authored
This is needed to support fast forking. When an object pool is provided, forking can cheap in terms of disk space and time since the clone only needs to fetch the references. The `git clone` can take in a `--reference <repository>` flag and will output an alternates file with absolute paths. To maintain the use of relative paths, after the fork is successful we recreate the alternates file. Relates to gitlab#24523
-
- Nov 12, 2020
-
-
Paul Okstad authored
Once the cache middleware was removed, it created new failures in CI that alert us to where ref hook options are needed. This updates those places and removes places where it is no longer needed due to updates in internal/git/subcommand.go
-
- Oct 28, 2020
-
-
Pavlo Strokov authored
In order to break dependency on the shared global config.Config variable the Locator interface is extended with InfoAlternatesPath and it used all over the places where func InfoAlternatesPath was used before. Part of: #2699
-
- Apr 22, 2020
-
-
Patrick Steinhardt authored
Previous to commit c02b9ad1 (UnlinkRepositoryFromObjectPool: stop removing objects/info/alternates, 2019-03-29), we removed the "objects/info/alternates" file for a repository when unlinking it from its object pool. While the commit changed it to not delete the file anymore, it forgot to update a comment that claims we still do. Update the comment to avoid confusion.
-
- Jan 09, 2020
-
-
Jacob Vosmaer authored
-
- Nov 13, 2019
-
-
John Cai authored
Adds an RPC to get a repository's object pool. Also added a method under internal/git/objectpool to get an object pool of a repository.
-
- Oct 10, 2019
-
-
Zeger-Jan van de Weg authored
Closes #1944
-
- Jul 31, 2019
-
-
Jacob Vosmaer authored
-
- Jul 16, 2019
-
-
John Cai authored
-
- Jul 08, 2019
-
-
John Cai authored
-