Skip to content
Snippets Groups Projects
  1. Dec 05, 2024
    • Sami Hiltunen's avatar
      Remove alternate validation logic from transaction manager · 7ac2a8db
      Sami Hiltunen authored
      Transaction manager should focus on conflict checking concurrent
      transactions. Transactions are expected to be correct in isolation
      so doing extra validation on them is not necessary. Remove the
      validation logic around alternates. We remove tests that were covering
      validation failures. Tests that are exercising concurrent behaviors
      were updated to assert the conflict.
      7ac2a8db
    • Sami Hiltunen's avatar
      Record alternate link operations in handler · 3b820d58
      Sami Hiltunen authored
      Alternate linking operation have so far been recorde in transaction
      manager as we didn't have a general API for staging file system
      operations in transactions. As we have such an API now, move the
      linking operation to be recorded in the handler.
      
      We reuse the logic from the handler in the tests and remove the
      custom linking logic.
      3b820d58
    • Sami Hiltunen's avatar
      Split objectpool.Link from the object pool instance · 0497901b
      Sami Hiltunen authored
      Split the ObjectPool.Link method into a function so it can be called
      without having an object pool at hand. This will be used in transaction
      manager's tests as the object pool constructor enforces certain format
      for relative paths.
      0497901b
  2. Sep 04, 2024
    • Sami Hiltunen's avatar
      Merge storagectx package into storage · d2044727
      Sami Hiltunen authored
      storagectx package was a workaround that was needed in the past as
      the 'git' package would have had cyclic depencies with the 'storage'
      pacakge if we define the Transaction interface there. We've since
      restructured the 'git' package to solve the structural issue there
      and moved the Transaction interface into the 'storage' package.
      There's no reason for the storagectx package to exist anymore. This
      commit merges the package into the 'storage' package to simplify.
      d2044727
    • Sami Hiltunen's avatar
      Replace RunWithTransaction with ExtractTransaction · 8dfa45b8
      Sami Hiltunen authored
      RunWithTransaction runs a callback with a transaction found in the
      context. This is somewhat an awkward interface with the current uses.
      
      Sometimes we want to actually extract the transaction from the context,
      not just do things with it. Doing so currently requires declaring a
      variable outside of the callback and assigning into it within the
      callback.
      
      Sometimes we need to return an error from the callback. As we don't
      support propagating an error, the error variable needs to be declared
      outside of the callback, assigned in the callback, and then checked
      outside of the callback.
      
      This commit replaces the function with ExtractTranscation that simply
      extracts the transaction from the context. This is still convenient for
      uses when we just need to call something on the transaction but is a lot
      more convenient for the above two cases. For the first case, we can now
      simply extract the transaction into the target variable. For the second
      case, we can simply return directly from the if block if an error occurs.
      8dfa45b8
    • Sami Hiltunen's avatar
      Define Transaction interface in storage package · a051d4dd
      Sami Hiltunen authored
      There was previously a cyclic dependency between the 'git' and 'storage'
      packages. We've solved this now by splitting the implementation related
      bits out from the 'git' package and only type definitions and interfaces
      there. With the cycle out of the way, we can now define the Transaction
      interface in a single location and reuse it across all the sites that
      previously were using one of the other interface definitions.
      
      This commit defines the interface of a Transaction in the storage
      package. We also replace duplicate definitions of the interface in
      other packages.
      a051d4dd
  3. Jul 01, 2024
    • Sami Hiltunen's avatar
      Take a context into GetStorageName · 91c19501
      Sami Hiltunen authored
      Transactions currently rely on rewriting the relative path of a
      repository to implement snapshot isolation. This has the downside
      that the relative paths get a prefix such as '+gitaly/staging/xx/yy/...'.
      For large parts of the application this is not a problem the handlers
      are generally just accessing a repository at a given path. This is a
      leaky abstraction though as the prefix is not part of the relative path
      and it is visible to the handlers. This has an impact on code that
      operates directly on the relative path in some manner. This could be
      for example logging, using the relative path as a key, or operating
      directly on file paths.
      
      The rewriting could be implemented transparently if we instead
      rewrite the storage directory's root path as seen by the transaction.
      Storage directory's path doesn't have a meaning unlike the relative
      path which is used to identify a repository.
      
      This commit takes the first step towards that by wiring context into
      GetStorageName of Locator. The injected context allows us to later
      plug transaction scoped logic into Locator that returns the rewritten
      storage root of a transaction when attempting to access a given storage.
      This way we can wire the indirection through the entire application
      without having to touch each site generating storage paths separately.
      
      The commit is large as we access locator to generate storage paths
      in many places, and we have to drill the context down into each of
      them. The commit simply wires the context through all call sites into
      GetStorageName.
      91c19501
  4. Apr 04, 2024
    • Justin Tobler's avatar
      objectpool: Refactor reading alternates file · 4d1e0d13
      Justin Tobler authored
      Both `objectpool.FromRepo()` and `objectpool.LinkedToRepository()` use
      `getAlternateObjectDir()` to read the alternates file. Refactor the code
      to use `stats.AlternatesInfoForRepository()` instead because it is more
      robust and simplifies error handling.
      
      The only behavior change is that `objectpool.FromRepo()` is slightly
      more strict and returns an error if a repository contains an empty
      alternates file. Previously it would only return an error if the
      alternates file was missing or malformed. An empty alternates file
      should be treated as invalid and return an error in the same manner.
      4d1e0d13
  5. Feb 20, 2024
    • Sami Hiltunen's avatar
      Simplify transaction API for updating alternate · 960a5513
      Sami Hiltunen authored
      Transaction currently provides a method to stage an alternate update
      into the transaction. The path is the relative path of the repository
      that we want to link the alternate to.
      
      This commit simplifies the external API to just marking that the file
      was updated. This signals to the transaction manager to check the file
      and to stage any updates.
      
      This simplified API makes it easier to move the alternate updates to
      be logged through the physical logging protocol by simply logging the
      staged alternate file of the transaction.
      960a5513
  6. Dec 05, 2023
    • Sami Hiltunen's avatar
      Enable transactions for LinkRepositoryToObjectPool · a15c2cee
      Sami Hiltunen authored
      This commit enables transactions for LinkRepositoryToObjectPool.
      This requires just recording the updated alternate link in the
      transaction.
      
      Transactions require the object pool and the member repository to
      be in the same partition. This may not be the case when Praefect
      attempts to reconciliate alternate link differences during
      replication. Moving repositories between partitions is difficult, so
      for now we'll simply ignore the pool linking failure in Praefect and
      leave accept deviating alternate links.
      a15c2cee
  7. Sep 13, 2023
  8. May 10, 2023
  9. Dec 13, 2022
    • Patrick Steinhardt's avatar
      git/objectpool: Convert usage of `FullPath()` to `Path()` · 79812edc
      Patrick Steinhardt authored
      Object pools provide two ways to calculate the path of a pool:
      `FullPath()` returns the path without verifying whether the pool exists,
      while `Path()` verifies that the target exists. The latter is what we
      already use in the context of the `localrepo.Repo` package, and it gives
      us more safeguards to ensure that the object pool really exists when it
      ought to exist.
      
      Refactor the code to use `Path()` instead of `FullPath()`.
      79812edc
  10. May 20, 2022
    • John Cai's avatar
      Update go package name from v14 to v15 · cd77c046
      John Cai authored
      This commit changes the major version in the package name from v14 to
      v15
      
      Updating go.mod & go.sum with new module name v15
      
      Update Makefile to bump major version to v15
      
      Update the gitaly package name in the Makefile. Also update
      gitaly-git2go-v14 -> gitaly-git2go-v15. We need to keep
      gitaly-git2go-v14 for a release however, for zero downtime upgrades.
      This pulls directly from a sha that is v14.
      
      Update package name from v14->v15 for auth, client, cmd, internal packages
      
      This commit changes the package name from v14 to v15 in go and proto
      files in the internal, auth, client, cmd packages.
      
      proto: Update major package number in package name
      
      tools: Change major version number in package name from v14 to v15
      
      gitaly-git2go: Change the package name from v14 to v15
      
      update module updater for v15
      
      Update the documentation for the module updater to reflect v15
      cd77c046
  11. Feb 07, 2022
  12. Nov 03, 2021
    • Patrick Steinhardt's avatar
      objectpool: Drop UnlinkRepostioryFromObjectPool RPC · 2f2a001d
      Patrick Steinhardt authored
      The UnlikRepostioryFromObjectPool RPC is not used by anything anymore,
      and the implementation from it is dangerous given that it doesn't
      actually unlink a repository from its object pool: it only tries to
      remove a remote named after the pool's project path, which we wouldn't
      ever create in the first place. The RPC has thus been deprecated in
      release v14.3.
      
      Remove the RPC and its backing code.
      
      Changelog: removed
      2f2a001d
  13. Oct 26, 2021
  14. Oct 25, 2021
    • Patrick Steinhardt's avatar
      objectpool: Drop UnlikRepostioryFromObjectPool RPC · ffadcf56
      Patrick Steinhardt authored
      The UnlikRepostioryFromObjectPool RPC is not used by anything anymore,
      and the implementation from it is dangerous given that it doesn't
      actually unlink a repository from its object pool: it only tries to
      remove a remote named after the pool's project path, which we wouldn't
      ever create in the first place. The RPC has thus been deprecated in
      release v14.3.
      
      Remove the RPC and its backing code.
      
      Changelog: removed
      ffadcf56
  15. Oct 11, 2021
    • Patrick Steinhardt's avatar
      global: Always use extended file locking · 9dcc25c6
      Patrick Steinhardt authored
      With 159948cc (Merge branch 'pks-tx-extended-file-locking' into
      'master', 2021-09-20), we have introduced extended file locking such
      that modifications to on-disk files will always lock, vote on and then
      commit the change in a racefree way when transactions are enabled. The
      code has been enabled in production now without any observed issues and
      can thus be deemed stable.
      
      Remove the feature flag such that we always use extended file locking.
      
      Changelog: fixed
      9dcc25c6
  16. Sep 20, 2021
    • Patrick Steinhardt's avatar
      global: Replace trivial cases of deprecated `ioutil.ReadDir()` · 4d63c8f0
      Patrick Steinhardt authored
      With Go 1.16, the ioutil package was deprecated. In addition to being
      moved into the os package, `ioutil.ReadDir()` was also changed to not
      stat(3P) all dir entries anymore. As a result, the caller now has to
      do so manually. This is a performance improvement in some cases where
      the caller didn't require any of the file information, but really only
      wanted to read the directory's entries.
      
      Adapt trivial usecases of `ioutil.ReadDir()` which do not require this
      information with usage of `os.ReadDir()`. This leaves a few callsites of
      the old `ioutil.ReadDir()` function for future conversion.
      4d63c8f0
    • Patrick Steinhardt's avatar
      global: Replace deprecated usage of `ioutil.TempFile()` · af5cd856
      Patrick Steinhardt authored
      With Go 1.16, the ioutil package was deprecated. Replace our usage of
      `ioutil.TempFile()` with `os.CreateTemp()` to adapt accordingly.
      af5cd856
  17. Sep 17, 2021
    • Patrick Steinhardt's avatar
      objectpool: Convert `Link()` to write alternates transactionally · 8ed273f5
      Patrick Steinhardt authored
      When linking a repository into an object pool, then we must write the
      alternates file to point to the object pool's object storage such that
      it reuses objects from it. This is currently done in a non-transactional
      way and can thus easily race with concurrent updates of the same
      gitconfig.
      
      Convert it to use our new locking file writer and voting infrastructure
      to make this update race-free.
      
      Changelog: fixed
      8ed273f5
  18. Aug 09, 2021
    • Pavlo Strokov's avatar
      lint: Handle error returned by os.Remove · fc6d12cd
      Pavlo Strokov authored
      The change removes os.Remove from lint rules exclusion
      and fixes all the places where it causes the issue.
      If removal ends up with an error we try to log it if the
      logger is accessible. Otherwise the error is just omitted.
      We don't return it back to the caller because we don't want
      functional changes here.
      fc6d12cd
  19. May 27, 2021
    • Pavlo Strokov's avatar
      Create module v14 gitaly version · 12e0bf3a
      Pavlo Strokov authored
      The new "v14" version of the Gitaly module is named to match
      the next GitLab release. The module versioning is needed in
      order to pull gitaly as a dependency in other projects. The
      change updates all imports to include v14 version. The go.mod
      file was modified as well after go mod tidy execution. And
      the changes in dependency licenses are reflected in the NOTICE
      file.
      
      Part of: #3177
      12e0bf3a
  20. Feb 17, 2021
    • Patrick Steinhardt's avatar
      remote: Convert users and remove package · 13cac15f
      Patrick Steinhardt authored
      Now that our localrepo interface provides all functionality of the
      remote package, let's convert callsites to use the localrepo interface
      instead and remove the now-unused remote package.
      
      While at it, this commit also moves over a testcase for deletion of
      local branches to ensure that test coverage doesn't decrease because of
      the package deletion.
      13cac15f
  21. Feb 16, 2021
    • Pavlo Strokov's avatar
      Removal of the git.NewCommand · 25923c6b
      Pavlo Strokov authored
      The change replaces all usages of the git.NewCommand
      function with git.CommandFactory.New method.
      The New method of the ExecCommandFactory changed to
      use newCommand method to execute git commands.
      The new parameter was added to some functions to pass
      git.CommandFactory down to the place where git.NewCommand
      was used before.
      
      Part of: #2699
      25923c6b
  22. Jan 25, 2021
    • Patrick Steinhardt's avatar
      git: Add support for options which always get injected · 504af412
      Patrick Steinhardt authored
      Traditionally, the gitconfig is not managed by the Gitaly project but by
      the various projects which deploy Gitaly or by the administrator himself
      if he chooses to do a source-based installation. Admins may have to
      potentially update the gitconfig on each upgrade of Gitaly while we as
      the Gitaly team need to ensure that all projects which deploy Gitaly
      (Docker, CNG, Omnibus and the likes) have the same gitconfig. Going by
      Murphy's law, this means that the git configuration for ways of
      installing Gitaly is going to diverge or has already diverged.
      
      Running in an environment where we don't know that a set of config
      entries is set can be dangerous and may lead to incorrect results,
      failures or even data loss in the case of "core.fsyncObjectFiles" and
      "gc.auto".
      
      This commit thus moves the global configuration into the Gitaly project
      to avoid any inconsistent environments. This allows us to ensure that
      required options are set while being quicker to iterate in case we need
      to change the configuration.
      
      The injection is done via the `git.ConfigPair{}` mechanism, which will
      put all config entries on the command line via `git -c <key>=<value>`
      pairs. While it has the downside of being verbose, the current set of
      entries we inject is quite limited and contains only five different
      options, so we shouldn't be near any command line limits. Furthermore,
      we're currently in the process of upstreaming a new way of injecting
      config entries via the environment, which we can migrate to as soon as
      this git version has been released.
      
      The list of injected configuration entries is taken from our source
      installation instructions.
      504af412
  23. Jan 14, 2021
    • Patrick Steinhardt's avatar
      git: Rename SafeCmd() to NewCommand() · 8b2315f8
      Patrick Steinhardt authored
      Back when we introduced the Git DSL, we still had conflicting sets of
      safe and unsafe functions. Because of this legacy, our safe set of
      functions is still has the "Safe" prefix.
      
      This commit now ends that chapter and renames `SafeCmd()`. In alginment
      with the preceding renames, this is being renamed to `NewCommand()`.
      8b2315f8
  24. Dec 16, 2020
    • Pavlo Strokov's avatar
      Break dependency of the catfile on global config.Config · ef065f1e
      Pavlo Strokov authored
      The catfile package directly or indirectly uses config.Config
      value. And as we are about to remove config.Config from the
      code we need to break this dependency and substitute with
      other abstractions.
      
      For now those are storage.Locator and alternates.Env instead
      of the old alternates.PathAndEnv that depends on the global var.
      
      It was decided to extend constructor function of the catfile
      to accept storage.Locator as a dependency as it is already
      injected in most of the "server"s which handle requests.
      The other option could be creation of the "factory", but it will
      require a new dependency to be injected into all existing
      services which is a lot more changes and adds currently unnecessary
      abstraction that needs to be managed.
      
      Part of: #2699
      ef065f1e
  25. Dec 14, 2020
  26. Dec 11, 2020
    • Stan Hu's avatar
      Support cloning with an object pool in CreateFork · 80770247
      Stan Hu authored
      This is needed to support fast forking. When an object pool is provided,
      forking can cheap in terms of disk space and time since the clone only
      needs to fetch the references.
      
      The `git clone` can take in a `--reference <repository>` flag and will
      output an alternates file with absolute paths. To maintain the use of
      relative paths, after the fork is successful we recreate the alternates
      file.
      
      Relates to gitlab#24523
      80770247
  27. Nov 12, 2020
    • Paul Okstad's avatar
      Add and remove ref hooks as needed · a12e30fa
      Paul Okstad authored
      Once the cache middleware was removed, it created new failures in CI
      that alert us to where ref hook options are needed. This updates those
      places and removes places where it is no longer needed due to updates in
      internal/git/subcommand.go
      a12e30fa
  28. Oct 28, 2020
  29. Apr 22, 2020
    • Patrick Steinhardt's avatar
      objectpool: Update stale comment for unlinking repos · 712fcc95
      Patrick Steinhardt authored
      Previous to commit c02b9ad1 (UnlinkRepositoryFromObjectPool: stop
      removing objects/info/alternates, 2019-03-29), we removed the
      "objects/info/alternates" file for a repository when unlinking it from
      its object pool. While the commit changed it to not delete the file
      anymore, it forgot to update a comment that claims we still do.
      
      Update the comment to avoid confusion.
      712fcc95
  30. Jan 09, 2020
  31. Nov 13, 2019
    • John Cai's avatar
      Add GetObjectPool RPC · 6a32f1cc
      John Cai authored
      Adds an RPC to get a repository's object pool. Also added a method under
      internal/git/objectpool to get an object pool of a repository.
      6a32f1cc
  32. Oct 10, 2019
  33. Jul 31, 2019
  34. Jul 16, 2019
  35. Jul 08, 2019
Loading