Skip to content
Commits on Source (261)
......@@ -56,5 +56,5 @@ For S1 or S2 issues, please follow https://about.gitlab.com/handbook/engineering
- [ ] Severity realistically set
- [ ] Clearly articulated what is needed from the Gitaly team to support your request by filling out the _What specifically do you need from the Gitaly team_
/label ~"Gitaly Customer Issue" ~"group::gitaly" ~"devops::systems" ~"section::enablement" ~"workflow::problem validation"
/cc @mjwood @andrashorvath @john.mcdonnell
/label ~"Gitaly Customer Issue" ~"customer" ~"group::gitaly" ~"devops::systems" ~"section::enablement" ~"workflow::problem validation"
/cc @mjwood @andrashorvath @jcaigitlab @john.mcdonnell
......@@ -34,6 +34,7 @@ linters:
- thelper
- unconvert
- unused
- gitaly-linters
# These linters don't currently support Go 1.18+. Please refer to
# https://github.com/golangci/golangci-lint/issues/2649 for further
# information.
......@@ -135,6 +136,19 @@ linters-settings:
begin: false
tb:
begin: false
custom:
gitaly-linters:
path: ./_build/tools/gitaly-linters.so
description: A collection of linters tailored for Gitaly
original-url: gitlab.com/gitlab-org/gitaly
settings:
string_interpolation_quote:
included-functions:
- fmt.*
error_wrap:
included-functions:
- fmt.Errorf
- gitlab.com/gitlab-org/gitaly/v15/internal/structerr.*
issues:
exclude-use-default: false
......
# Gitaly changelog
## 15.10.0 (2023-03-21)
### Added (4 changes)
- [catfile: Add encoding information to GitCommit](gitlab-org/gitaly@cf24112527408773814956ae6aaf94a8f4a41ad9) by @arkn98 ([merge request](gitlab-org/gitaly!5506))
- [Cgroups: add cpu_quota_us limit](gitlab-org/gitaly@80ce55b0fb2388d11f0016d0b470a3778b00e057) ([merge request](gitlab-org/gitaly!5422))
- [housekeeping: Implement logic to write cruft packs](gitlab-org/gitaly@b18ae51043f470a170e815bf40750f65da0dcbc7) ([merge request](gitlab-org/gitaly!5454))
- [gitaly-backup: Add -remove-all-repositories flag](gitlab-org/gitaly@d7db18aebaae231014257e545f21eb4934be2df1) ([merge request](gitlab-org/gitaly!5410))
### Fixed (6 changes)
- [housekeeping: Don't write cruft packs in pool repositories](gitlab-org/gitaly@548aa80a683a330319d115cd7750862b42b6ddf6) ([merge request](gitlab-org/gitaly!5488))
- [housekeeping: Clean packfile locks left behind by fetches and pushes](gitlab-org/gitaly@7562da2cfbc0f98916c610e744515b24366fdfd4) ([merge request](gitlab-org/gitaly!5479))
- [repository: Handle empty root trees in GetArchive](gitlab-org/gitaly@60d2b90d4d5d93602a4092bc993ded2dbfab1514) ([merge request](gitlab-org/gitaly!5455))
- [catfile: Don't return readers to cache when their queues are in use](gitlab-org/gitaly@21579e11482d2cdb9eccd0d4dcce26e314703a0b) ([merge request](gitlab-org/gitaly!5439))
- [Fix positional argument error when file starts with `-`](gitlab-org/gitaly@0ab47cf61488ed6c40630cf0564852d9b5a0cf75) ([merge request](gitlab-org/gitaly!5445))
- [Makefile: Don't override PKG_CONFIG_PATH](gitlab-org/gitaly@0232d01c91458dc4c1530b21d6d9dcaee1494335) ([merge request](gitlab-org/gitaly!5433))
### Changed (3 changes)
- [housekeeping: Unconditionally enable writing multi-pack indices](gitlab-org/gitaly@ea924d434a5485dcacb6bc2d4e89dca98fdaa2fc) ([merge request](gitlab-org/gitaly!5426))
- [git: Bump Git distribution's default version to v2.39](gitlab-org/gitaly@7511b286aa0a0db89aea3b6870ad92aa5b556de6) ([merge request](gitlab-org/gitaly!5425))
- [git: Always use Git v2.39](gitlab-org/gitaly@45fa9e34dd953cdb9e92dfad725af9130fe6efa8) ([merge request](gitlab-org/gitaly!5425))
### Deprecated (1 change)
- [proto: Deprecate `root_oid` field in the TreeEntry message](gitlab-org/gitaly@9eba19b5b2efd3ef97a5abda98cc230a36b92d18) ([merge request](gitlab-org/gitaly!5509))
### Other (1 change)
- [Simplify streamcache.Cache for efficient null implementation](gitlab-org/gitaly@de53fe2f5c773a059b599674ccaf459b5daf4b49) ([merge request](gitlab-org/gitaly!5444))
## 15.9.3 (2023-03-09)
No changes.
## 15.9.2 (2023-03-02)
No changes.
## 15.9.1 (2023-02-23)
No changes.
## 15.9.0 (2023-02-21)
### Added (13 changes)
- [remote: Have FindRemoteRepository detect repos using SHA256](gitlab-org/gitaly@be3e14641ca8771cd21d62f5c2cb75991af6de58) ([merge request](gitlab-org/gitaly!5368))
- [gitaly-backup: Inject GITALY_SERVERS env var](gitlab-org/gitaly@01930dff6714cd9b904163a50658609c9ad24d75) ([merge request](gitlab-org/gitaly!5384))
- [Implement RemoveAll RPC](gitlab-org/gitaly@5010f808b5ea7e2509d9fbf6b9c77c0d28984af6) ([merge request](gitlab-org/gitaly!5291))
- [git: Wire up Git v2.39 execution environment](gitlab-org/gitaly@b9890e7265593266d5cc157df9227e9a477f6721) ([merge request](gitlab-org/gitaly!5339))
- [remoterepo: Handle SHA256 object hash in `ResolveRevision()`](gitlab-org/gitaly@72f03b1fd0c12eab20a3a559676751273e5b3da6) ([merge request](gitlab-org/gitaly!5317))
- [repository: Implement new RPC to detect the object format](gitlab-org/gitaly@c58728b9b3afd82690d617fd029d43733ecd668f) ([merge request](gitlab-org/gitaly!5317))
- [proto: CommitDiffRequest: Introduce `WhitespaceChanges`](gitlab-org/gitaly@900fabc05e316a073e4be04509d248d4be9ae52f) ([merge request](gitlab-org/gitaly!5288))
- [git/stats: Discern different packfile types](gitlab-org/gitaly@47af9fbaa390da9cfc0de8f27fd7cfa6e21118f5) ([merge request](gitlab-org/gitaly!5310))
- [housekeeping: Report existence of bitmap extensions](gitlab-org/gitaly@beac70afe618377a48f9eabd49a4d3a621e2583f) ([merge request](gitlab-org/gitaly!5303))
- [git/stats: Surface information about reverse indices](gitlab-org/gitaly@a30997dda7e0b9fbe5ecb2e56c5f857a162471c2) ([merge request](gitlab-org/gitaly!5299))
- [housekeeping: Add Prometheus metrics for on-disk repository states](gitlab-org/gitaly@512aed451d7e2c0c45629a13c1b12d86dece3131) ([merge request](gitlab-org/gitaly!5276))
- [git/stats: Read bitmap info when deriving packfile info](gitlab-org/gitaly@0c0dcecd903c7295f0d2e9f83c525b4a921bc57d) ([merge request](gitlab-org/gitaly!5269))
- [benchmarking: Add a README](gitlab-org/gitaly@c39695ac9154de55917f044f3932b14ca91ec0fd) ([merge request](gitlab-org/gitaly!5235))
### Fixed (17 changes)
- [dnsresolver: Support DNS scheme in client dial functions](gitlab-org/gitaly@21e5a03bdd8c8b3845091d28a77ed890d23ed1ab) ([merge request](gitlab-org/gitaly!5367))
- [go: Fix dependencies which have been inadvertently downgraded](gitlab-org/gitaly@a8568f1c8f8dc5f13c8e59a5633e480118c5bf6c) ([merge request](gitlab-org/gitaly!5354))
- [updateref: Fix handling of context cancellation errors](gitlab-org/gitaly@02e4696c9236bd434170a25cb938f3839b89e12d) ([merge request](gitlab-org/gitaly!5345))
- [supervisor: Wait for notifiers to shut down](gitlab-org/gitaly@7868f591aa494e4a2a578040e735d39e3913cedd) ([merge request](gitlab-org/gitaly!5346))
- [git: Fix race between cancelling git-version(1) and reading its output](gitlab-org/gitaly@e83475a5e80fc0daec8096bf0853c51cf64a98d0) ([merge request](gitlab-org/gitaly!5344))
- [updateref: return proper error when writing to closed process](gitlab-org/gitaly@17a74ef76c46706427326b5652d77504dc65e228) ([merge request](gitlab-org/gitaly!5320))
- [objectpool: Detect object format to use for parsing object IDs](gitlab-org/gitaly@04f2c468612b40f41b2e02ada0ad3bcfdf72df60) ([merge request](gitlab-org/gitaly!5318))
- [objectpool: Fix object format when creating from empty repo](gitlab-org/gitaly@c6fd56b6e66ffcb48c7e3fa78c4683b66d3dfec3) ([merge request](gitlab-org/gitaly!5318))
- [git: Allow wildcards in config keys](gitlab-org/gitaly@a841a2caf11f0fe39ebeaeb1222f9f8fb221d136) ([merge request](gitlab-org/gitaly!5314))
- [housekeeping: Don't write bitmaps with MIDX in pooled repos](gitlab-org/gitaly@c74a4b7386f170c5629ab786aae4afbc3a61bcd4) ([merge request](gitlab-org/gitaly!5312))
- [repository: Ignore licenses without a name](gitlab-org/gitaly@529cba3212366a92f21bc99a5328cf2e62f8a477) ([merge request](gitlab-org/gitaly!5285))
- [Makefile: Install protogem dependencies](gitlab-org/gitaly@e917b3d3a8d4e680be8899dc974b12357616b8cf) ([merge request](gitlab-org/gitaly!5296))
- [conflicts: Fix nil pointer access when conflict has no ancestor](gitlab-org/gitaly@98b0027d4791b4e9d3c87f36e80436dc65245de9) ([merge request](gitlab-org/gitaly!5283))
- [conflicts: Fix nil pointer access when conflict has no ancestor](gitlab-org/gitaly@169dc65a41588477a983c3b17e638bd26d25bc9e) ([merge request](gitlab-org/gitaly!5283))
- [git: Upgrade Git to correctly sync packed-refs to disk](gitlab-org/gitaly@266921418ebf81aa71d64d8f417f13eacc10aff4) ([merge request](gitlab-org/gitaly!5282))
- [nodes: Set connection backoff MaxDelay to 1 second](gitlab-org/gitaly@6c4afcd90cf3fe95231bcb12bc6003854cd256ac) ([merge request](gitlab-org/gitaly!5255))
- [conflicts: Fix memory corruption in resolving conflicts](gitlab-org/gitaly@cefe00a120e9b68f32ac7cd9c15da5eb204c70f6) ([merge request](gitlab-org/gitaly!5258))
### Changed (7 changes)
- [housekeeping: Enable writing MIDX by default](gitlab-org/gitaly@2111df234b1a053c159683c4446953d0952e8ece) ([merge request](gitlab-org/gitaly!5397))
- [gitaly-backup: Use pointer layout by default](gitlab-org/gitaly@c60ec202b0510b22748633f62901c5aaee0d0d98) ([merge request](gitlab-org/gitaly!5378))
- [gitaly/config: Unify Git config validation](gitlab-org/gitaly@c320eb9355cca6061fc8fea2529eb84717349e0b) ([merge request](gitlab-org/gitaly!5331))
- [housekeeping: Unconditionally write bitmap lookup tables](gitlab-org/gitaly@150d4a95f7cae901a8b1fb7add89cd235f783be8) ([merge request](gitlab-org/gitaly!5343))
- [Bump Nokogiri to v1.14.1](gitlab-org/gitaly@7a59c3272287ad4164f8dfee5b1a2d1c12376d2a) ([merge request](gitlab-org/gitaly!5327))
- [Default enable invalid metadata deletion in Praefect](gitlab-org/gitaly@02e6584586ae7aed4407bffd1b5f5b3495fd15af) ([merge request](gitlab-org/gitaly!5321))
- [Use Yamux configuration in Praefect by default](gitlab-org/gitaly@2dba1c8e4ce19f4cdce3b0f7c406a4f1d663e087) ([merge request](gitlab-org/gitaly!5215))
### Deprecated (1 change)
- [Deprecate FullPath and SetFulPath RPCs](gitlab-org/gitaly@27abba61db658e7c9e6befc3ba550d3885b1e591) ([merge request](gitlab-org/gitaly!5293))
### Removed (2 changes)
- [git: Rely on `repack.updateServerInfo` for repacks instead of `-n`](gitlab-org/gitaly@42147604283a2c1e387ac918bbcf7b40ee532872) ([merge request](gitlab-org/gitaly!5264))
- [housekeeping: Always enable commit-graph generation data](gitlab-org/gitaly@a15c34925479e82e089b46641bcd9e46996b2704) ([merge request](gitlab-org/gitaly!5260))
### Security (1 change)
- [git: Upgrade to Git security release v2.38.4.gl1 and v2.39.2](gitlab-org/gitaly@cd869bcc60f97c19332f93cbf1c0a8b52607df8f)
### Performance (4 changes)
- [git: Replace regexp in ValidateHex](gitlab-org/gitaly@6f5c8288c7447e0dc382024467b8f1e5a7239fb5) ([merge request](gitlab-org/gitaly!5308))
- [git/stats: Don't stat(3P) all pack directory entries](gitlab-org/gitaly@a440158795b509632fea820ce12973022bba9604) ([merge request](gitlab-org/gitaly!5299))
- [housekeeping: Enable writing of multi-pack-indices](gitlab-org/gitaly@ef10b419db050e8a43d2c345bce167bc0a1d15ce) ([merge request](gitlab-org/gitaly!5275))
- [housekeeping: Enable writing bitmap lookup tables](gitlab-org/gitaly@73505638cebbbd47aa70b9f587f411559abc1f08) ([merge request](gitlab-org/gitaly!5278))
## 15.8.4 (2023-03-02)
No changes.
## 15.8.3 (2023-02-15)
No changes.
......@@ -54,6 +170,10 @@ No changes.
- [repository: Disable misleading dangling checks in git-fsck(1)](gitlab-org/gitaly@fb830f5f01311609c01590ddf04a22a3d33cd36d) ([merge request](gitlab-org/gitaly!5230))
## 15.7.8 (2023-03-02)
No changes.
## 15.7.7 (2023-02-10)
### Security (1 change)
......
......@@ -145,7 +145,7 @@ SKIP_OVERRIDING_GIT_VERSION ?=
# The default version is used in case the caller does not set the variable or
# if it is either set to the empty string or "default".
ifeq (${GIT_VERSION:default=},)
override GIT_VERSION := ${GIT_VERSION_2_38}
override GIT_VERSION := ${GIT_VERSION_2_39}
else
# Support both vX.Y.Z and X.Y.Z version patterns, since callers across GitLab
# use both.
......@@ -274,7 +274,7 @@ GOCACHE_MAX_SIZE_KB ?= 5000000
export GOCACHE ?= ${BUILD_DIR}/cache
export GOPROXY ?= https://proxy.golang.org
export PATH := ${BUILD_DIR}/bin:${PATH}
export PKG_CONFIG_PATH := ${LIBGIT2_INSTALL_DIR}/lib/pkgconfig
export PKG_CONFIG_PATH := ${LIBGIT2_INSTALL_DIR}/lib/pkgconfig:${PKG_CONFIG_PATH}
# Allow the linker flag -D_THREAD_SAFE as libgit2 is compiled with it on FreeBSD
export CGO_LDFLAGS_ALLOW = -D_THREAD_SAFE
......@@ -364,11 +364,18 @@ prepare-test-repos: ${TEST_REPO}
.PHONY: test
## Run Go and Ruby tests.
test: test-go test-ruby
test: test-go test-ruby test-gitaly-linters
.PHONY: test-ruby
test-ruby: rspec
.PHONY: test-gitaly-linters
## Test Go tests in tools/golangci-lint/gitaly folder
## That folder has its own go.mod file. Hence tests must run the context of that folder.
test-gitaly-linters: TEST_PACKAGES := .
test-gitaly-linters:
${Q}cd ${SOURCE_DIR}/tools/golangci-lint/gitaly && $(call run_go_tests)
.PHONY: test-go
## Run Go tests.
test-go: prepare-tests
......@@ -417,14 +424,17 @@ check-mod-tidy:
${Q}go mod tidy -compat=1.17
${Q}${GIT} diff --quiet --exit-code go.mod go.sum || (echo "error: uncommitted changes in go.mod or go.sum" && exit 1)
${TOOLS_DIR}/gitaly-linters.so: ${SOURCE_DIR}/tools/golangci-lint/go.sum $(wildcard ${SOURCE_DIR}/tools/golangci-lint/gitaly/*.go)
${Q}go build -buildmode=plugin -o '$@' -modfile ${SOURCE_DIR}/tools/golangci-lint/go.mod $(filter-out $<,$^)
.PHONY: lint
## Run Go linter.
lint: ${GOLANGCI_LINT} libgit2 ${GITALY_PACKED_EXECUTABLES}
lint: ${GOLANGCI_LINT} libgit2 ${GITALY_PACKED_EXECUTABLES} ${TOOLS_DIR}/gitaly-linters.so lint-gitaly-linters
${Q}${GOLANGCI_LINT} run --build-tags "${SERVER_BUILD_TAGS},${GIT2GO_BUILD_TAGS}" --out-format tab --config ${GOLANGCI_LINT_CONFIG} ${GOLANGCI_LINT_OPTIONS}
.PHONY: lint-fix
## Run Go linter and write back fixes to the files (not supported by all linters).
lint-fix: ${GOLANGCI_LINT} libgit2 ${GITALY_PACKED_EXECUTABLES}
lint-fix: ${GOLANGCI_LINT} libgit2 ${GITALY_PACKED_EXECUTABLES} ${TOOLS_DIR}/gitaly-linters.so
${Q}${GOLANGCI_LINT} run --fix --build-tags "${SERVER_BUILD_TAGS},${GIT2GO_BUILD_TAGS}" --out-format tab --config ${GOLANGCI_LINT_CONFIG} ${GOLANGCI_LINT_OPTIONS}
.PHONY: lint-docs
......@@ -432,6 +442,13 @@ lint-fix: ${GOLANGCI_LINT} libgit2 ${GITALY_PACKED_EXECUTABLES}
lint-docs:
${Q}markdownlint-cli2-config .markdownlint.yml README.md REVIEWING.md STYLE.md **/*.md || (echo "error: markdownlint-cli2 not found!")
.PHONY: lint-gitaly-linters
## Test Go tests in tools/golangci-lint/gitaly folder
## That folder has its own go.mod file. Hence linter must run the context of that folder.
lint-gitaly-linters: ${GOLANGCI_LINT} ${TOOLS_DIR}/gitaly-linters.so
${Q}cd ${SOURCE_DIR}/tools/golangci-lint/gitaly
${Q}${GOLANGCI_LINT} run --out-format tab --config ${GOLANGCI_LINT_CONFIG} .
.PHONY: format
## Run Go formatter and adjust imports.
format: ${GOIMPORTS} ${GOFUMPT}
......@@ -513,7 +530,7 @@ no-changes:
.PHONY: no-proto-changes
no-proto-changes: PROTO_DEST_DIR := ${BUILD_DIR}/proto-changes
no-proto-changes: proto | ${BUILD_DIR}
${Q}${GIT} diff --no-index --exit-code -- "${PROTO_DEST_DIR}" "${SOURCE_DIR}/proto/go"
${Q}${GIT} diff --no-index --exit-code -- "${SOURCE_DIR}/proto/go" "${PROTO_DEST_DIR}"
.PHONY: dump-database-schema
## Dump the clean database schema of Praefect into a file.
......
......@@ -7028,6 +7028,30 @@ Copyright 2018 CoreOS, Inc
This product includes software developed at CoreOS, Inc.
(http://www.coreos.com/).
 
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
LICENSE.md - github.com/cpuguy83/go-md2man/v2/md2man
The MIT License (MIT)
Copyright (c) 2014 Brian Goff
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
LICENSE - github.com/davecgh/go-spew/spew
ISC License
......@@ -7806,15 +7830,27 @@ OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
 
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
LICENSE - github.com/getsentry/sentry-go
Copyright (c) 2019 Sentry (https://sentry.io) and individual contributors.
All rights reserved.
MIT License
 
Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
Copyright (c) 2019 Functional Software, Inc. dba Sentry
 
* Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
 
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
 
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
LICENSE.md - github.com/git-lfs/git-lfs/v3
......@@ -19735,7 +19771,7 @@ LICENSE - github.com/jcmturner/rpc/v2
LICENSE - github.com/jdkato/prose
MIT License
 
Copyright (c) 2017 Joseph Kato
Copyright (c) 2017 -2018 Joseph Kato
 
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
......@@ -20706,36 +20742,35 @@ Copyright 2012 Matt T. Proud (matt.proud@gmail.com)
 
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
LICENSE - github.com/miekg/dns
Copyright (c) 2009 The Go Authors. All rights reserved.
BSD 3-Clause License
Copyright (c) 2009, The Go Authors. Extensions copyright (c) 2011, Miek Gieben.
All rights reserved.
 
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are
met:
modification, are permitted provided that the following conditions are met:
 
* Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above
copyright notice, this list of conditions and the following disclaimer
in the documentation and/or other materials provided with the
distribution.
* Neither the name of Google Inc. nor the names of its
contributors may be used to endorse or promote products derived from
this software without specific prior written permission.
1. Redistributions of source code must retain the above copyright notice, this
list of conditions and the following disclaimer.
 
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
2. Redistributions in binary form must reproduce the above copyright notice,
this list of conditions and the following disclaimer in the documentation
and/or other materials provided with the distribution.
 
As this is fork of the official Go code the same license applies.
Extensions of the original work are copyright (c) 2011 Miek Gieben
3. Neither the name of the copyright holder nor the names of its
contributors may be used to endorse or promote products derived from
this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
LICENSE - github.com/mitchellh/go-homedir
......@@ -23445,6 +23480,30 @@ LICENSE - github.com/uber/jaeger-lib/metrics
See the License for the specific language governing permissions and
limitations under the License.
 
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
LICENSE - github.com/urfave/cli/v2
MIT License
Copyright (c) 2022 urfave/cli maintainers
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
LICENSE - github.com/xanzy/ssh-agent
Apache License
......@@ -23650,6 +23709,30 @@ LICENSE - github.com/xanzy/ssh-agent
limitations under the License.
 
 
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
LICENSE - github.com/xrash/smetrics
Copyright (C) 2016 Felipe da Cunha Gonçalves
All Rights Reserved.
MIT LICENSE
Permission is hereby granted, free of charge, to any person obtaining a copy of
this software and associated documentation files (the "Software"), to deal in
the Software without restriction, including without limitation the rights to
use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of
the Software, and to permit persons to whom the Software is furnished to do so,
subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER
IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
LICENSE - gitlab.com/gitlab-org/labkit
The MIT License (MIT)
15.9.0-rc4
\ No newline at end of file
15.10.0
\ No newline at end of file
/hosts.ini
/terraform/*
!/terraform/main.tf
!/terraform/locals.tf
/group_vars/all.yml
/results/*
config.yml
terraform 1.3.9
python 3.11.2
......@@ -4,19 +4,20 @@
An Ansible script for running RPC-level benchmarks against Gitaly.
## Required tools
**Note**: You must be a member of the `gitaly-benchmark-0150d6cf` GCP group.
The following programs must be installed locally to run the script:
- Ansible 2.14 or above
- Terraform 1.2 or above
- gcloud
## Steps for use
You must be a member of the `gitaly-benchmark-0150d6cf` GCP group.
### 1. Setup your environment
## Steps for use
1. Ensure that [`gcloud`](https://cloud.google.com/sdk/docs/install) is installed and available on your path.
1. Ensure that `python` and `terraform` are installed, or use [`asdf`](https://asdf-vm.com/guide/getting-started.html) to install them (recommended).
1. Create a new Python virtualenv: `python3 -m venv env`
1. Activate the virtualenv: `source env/bin/activate`
1. Install Ansible: `python3 -m pip install -r requirements.txt`
1. **Optional**: Copy `config.yml.example` to `config.yml` to customize the machine type uses for benchmarking
### 1. Create instance
### 2. Create instance
```shell
./create-benchmark-instance
......@@ -32,7 +33,7 @@ Use the `gitaly_bench` user to SSH into the instance if desired:
ssh gitaly_bench@<INSTANCE_ADDRESS>
```
### 2. Configure instance
### 3. Configure instance
```shell
./configure-benchmark-instance
......@@ -42,7 +43,7 @@ Build and install Gitaly from source with from desired reference and install
profiling tools like `perf` and `libbpf-tools`. A disk image containing the
test repositories will be mounted to `/mnt/git-repositories` on the Gitaly node.
### 3. Run benchmarks
### 4. Run benchmarks
```shell
./run-benchmarks
......@@ -86,7 +87,7 @@ When profiling is enabled, the following are also present:
This uses `--call-graph=fp` for accurate stack traces for Golang.
- `page-cachestat.txt` - Kernel page cache hit rate.
### 4. Destroy instance
### 5. Destroy instance
```shell
./destroy-benchmark-instance
......
......@@ -3,3 +3,4 @@ pipelining = True
[defaults]
callbacks_enabled = profile_tasks
interpreter_python = auto_silent
---
project: "gitaly-benchmark-0150d6cf"
gitaly_machine_type: "t2d-standard-4"
client_machine_type: "n1-standard-1"
os_image: "ubuntu-os-cloud/ubuntu-2204-lts"
repository_disk_type: "pd-balanced"
boot_disk_size: 20
benchmark_region: "us-central1"
benchmark_zone: "us-central1-a"
# Enable to use regional persistent disk https://cloud.google.com/compute/docs/disks/regional-persistent-disk
# Regional PD is supported on only E2, N1, N2, and N2D machine type VMs.
use_regional_disk: false
# One of these zones must be the zone for benchmark_zone
regional_disk_replica_zones: ["us-central1-a", "us-central1-b"]
#!/bin/sh
exec ansible-playbook -l localhost create.yml "$@"
exec ansible-playbook -i localhost, create.yml "$@"
---
- name: Set user-defined variables
hosts: localhost
connection: local
vars_prompt:
- name: gitaly_revision
prompt: "Enter Gitaly revision to build"
......
ansible==7.2.0
ansible-core==2.14.2
cffi==1.15.1
cryptography==39.0.1
Jinja2==3.1.2
MarkupSafe==2.1.2
packaging==23.0
pycparser==2.21
PyYAML==6.0
resolvelib==0.8.1
---
# needed for Ansible > 4.0 https://github.com/ansible/ansible/issues/74830
- name: Install acl
apt:
name:
- acl
state: present
- name: Create 'git' user
user:
name: git
......
......@@ -4,6 +4,13 @@
user: root
key: "{{ hostvars[groups['client'][0]]['client_ssh_key'] | b64decode }}"
# needed for Ansible > 4.0 https://github.com/ansible/ansible/issues/74830
- name: Install acl
apt:
name:
- acl
state: present
# Set high uid to ensure it's not squashed by another service
- name: Create 'git' user
user:
......
......@@ -12,3 +12,7 @@
dest: /usr/local
remote_src: true
creates: /usr/local/go/bin/go
retries: 3
delay: 3
register: result
until: result is success
locals {
config = try(yamldecode(file("../${path.root}/config.yml")), yamldecode(file("../${path.root}/config.yml.example")))
}
variable "project" { default = "gitaly-benchmark-0150d6cf" }
variable "benchmark_region" { default = "us-central1" }
variable "benchmark_zone" { default = "us-central1-a" }
variable "gitaly_benchmarking_instance_name" { }
variable "ssh_pubkey" { }
variable "os_image" { default = "ubuntu-os-cloud/ubuntu-2204-lts" }
variable "gitaly_benchmarking_instance_name" {}
variable "ssh_pubkey" {}
variable "startup_script" {
default = <<EOF
set -e
if [ -d /src/gitaly ] ; then exit; fi
EOF
}
variable "gitaly_machine_type" { default = "t2d-standard-4" }
variable "client_machine_type" { default = "n1-standard-1" }
variable "boot_disk_size" { default = "20" }
provider "google" {
project = var.project
region = var.benchmark_region
zone = var.benchmark_zone
project = local.config.project
region = local.config.benchmark_region
zone = local.config.benchmark_zone
}
data "google_compute_disk" "repository-disk" {
name = "git-repos"
project = "gitaly-benchmark-0150d6cf"
name = "git-repos"
project = local.config.project
}
resource "google_compute_disk" "repository-disk" {
name = format("%s-repository-disk", var.gitaly_benchmarking_instance_name)
type = "pd-balanced"
image = format("projects/%s/global/images/git-repositories", var.project)
name = format("%s-repository-disk", var.gitaly_benchmarking_instance_name)
type = local.config.repository_disk_type
image = format("projects/%s/global/images/git-repositories", local.config.project)
}
resource "google_compute_region_disk" "repository-region-disk" {
count = local.config.use_regional_disk ? 1 : 0
name = format("%s-repository-region-disk", var.gitaly_benchmarking_instance_name)
type = local.config.repository_disk_type
snapshot = google_compute_snapshot.repository-disk[0].id
replica_zones = local.config.regional_disk_replica_zones
}
resource "google_compute_snapshot" "repository-disk" {
count = local.config.use_regional_disk ? 1 : 0
name = format("%s-repository-snapshot", var.gitaly_benchmarking_instance_name)
source_disk = google_compute_disk.repository-disk.name
zone = local.config.benchmark_zone
}
resource "google_compute_instance" "gitaly" {
name = format("%s-gitaly", var.gitaly_benchmarking_instance_name)
machine_type = var.gitaly_machine_type
machine_type = local.config.gitaly_machine_type
boot_disk {
initialize_params {
image = var.os_image
size = var.boot_disk_size
image = local.config.os_image
size = local.config.boot_disk_size
}
}
attached_disk {
source = google_compute_disk.repository-disk.name
source = local.config.use_regional_disk ? google_compute_region_disk.repository-region-disk[0].self_link : google_compute_disk.repository-disk.self_link
device_name = "repository-disk"
}
network_interface {
network = "default"
network = "default"
subnetwork = "default"
access_config {}
}
metadata = {
ssh-keys = format("gitaly_bench:%s", var.ssh_pubkey)
ssh-keys = format("gitaly_bench:%s", var.ssh_pubkey)
startup-script = <<EOF
${var.startup_script}
EOF
......@@ -69,12 +77,12 @@ resource "google_compute_instance" "gitaly" {
resource "google_compute_instance" "client" {
name = format("%s-client", var.gitaly_benchmarking_instance_name)
machine_type = var.client_machine_type
machine_type = local.config.client_machine_type
boot_disk {
initialize_params {
image = var.os_image
size = var.boot_disk_size
image = local.config.os_image
size = local.config.boot_disk_size
}
}
......@@ -84,7 +92,7 @@ resource "google_compute_instance" "client" {
}
metadata = {
ssh-keys = format("gitaly_bench:%s", var.ssh_pubkey)
ssh-keys = format("gitaly_bench:%s", var.ssh_pubkey)
startup-script = <<EOF
${var.startup_script}
EOF
......