Skip to content
Snippets Groups Projects
Commit ffab3310 authored by Amy Qualls's avatar Amy Qualls :leopard: Committed by Igor Drozdov
Browse files

docs: Truncate pages, point users to GitLab repo

parent 977a7e80
No related branches found
No related tags found
1 merge request!705docs: Truncate pages, point users to GitLab repo
## Releasing a new version
# GitLab Shell process
GitLab Shell is versioned by git tags, and the version used by the Rails
application is stored in
[`GITLAB_SHELL_VERSION`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/GITLAB_SHELL_VERSION).
For each version, there is a raw version and a tag version:
- The **raw version** is the version number. For instance, `15.2.8`.
- The **tag version** is the raw version prefixed with `v`. For instance, `v15.2.8`.
To release a new version of GitLab Shell and have that version available to the
Rails application:
1. Create a merge request to update the [`CHANGELOG`](CHANGELOG) with the
**tag version** and the [`VERSION`](VERSION) file with the **raw version**.
2. Ask a maintainer to review and merge the merge request. If you're already a
maintainer, second maintainer review is not required.
3. Add a new git tag with the **tag version**.
4. Update `GITLAB_SHELL_VERSION` in the Rails application to the **raw
version**. (Note: this can be done as a separate MR to that, or in and MR
that will make use of the latest GitLab Shell changes.)
## Security releases
GitLab Shell is included in the packages we create for GitLab, and each version of GitLab specifies the version of GitLab Shell it uses in the `GITLAB_SHELL_VERSION` file, so security fixes in GitLab Shell are tightly coupled to the [GitLab security release](https://about.gitlab.com/handbook/engineering/workflow/#security-issues) workflow.
For a security fix in GitLab Shell, two sets of merge requests are required:
* The fix itself, in the `gitlab-org/security/gitlab-shell` repository and its backports to the previous versions of GitLab Shell
* Merge requests to change the versions of GitLab Shell included in the GitLab security release, in the `gitlab-org/security/gitlab` repository
The first step could be to create a merge request with a fix targeting `main` in `gitlab-org/security/gitlab-shell`. When the merge request is approved by maintainers, backports targeting previous 3 versions of GitLab Shell must be created. The stable branches for those versions may not exist, feel free to ask a maintainer to create ones. The stable branches must be created out of the GitLab Shell tags/version used by the 3 previous GitLab releases. In order to find out the GitLab Shell version that is used on a particular GitLab stable release, the following steps may be helpful:
```shell
git fetch security 13-9-stable-ee
git show refs/remotes/security/13-9-stable-ee:GITLAB_SHELL_VERSION
```
These steps display the version that is used by `13.9` version of GitLab.
Close to the GitLab security release, a maintainer should merge the fix and backports and cut all the necessary GitLab Shell versions. This allows bumping the `GITLAB_SHELL_VERSION` for `gitlab-org/security/gitlab`. The GitLab merge request will be handled by the general GitLab security release process.
Once the security release is done, a GitLab Shell maintainer is responsible for syncing tags and `main` to the `gitlab-org/gitlab-shell` repository.
This page [has moved into the `gitlab` repository](https://docs.gitlab.com/ee/development/gitlab_shell/process.html).
......@@ -4,114 +4,10 @@ group: Source Code
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
---
# GitLab Shell
# GitLab Shell documentation has moved
[![pipeline status](https://gitlab.com/gitlab-org/gitlab-shell/badges/main/pipeline.svg)](https://gitlab.com/gitlab-org/gitlab-shell/-/pipelines?ref=main)
[![coverage report](https://gitlab.com/gitlab-org/gitlab-shell/badges/main/coverage.svg)](https://gitlab.com/gitlab-org/gitlab-shell/-/pipelines?ref=main)
[![Code Climate](https://codeclimate.com/github/gitlabhq/gitlab-shell.svg)](https://codeclimate.com/github/gitlabhq/gitlab-shell)
GitLab Shell handles git SSH sessions for GitLab and modifies the list of authorized keys.
GitLab Shell is not a Unix shell nor a replacement for Bash or Zsh.
GitLab supports Git LFS authentication through SSH.
## Requirements
GitLab Shell is written in Go, and needs a Go compiler to build. It still requires
Ruby to build and test, but not to run.
GitLab Shell runs on `port 22` on an Omnibus installation. To use a regular SSH
service, configure it on an alternative port.
Download and install the current version of Go from [golang.org](https://golang.org/dl/)
We follow the [Golang Release Policy](https://golang.org/doc/devel/release.html#policy)
of supporting the current stable version and the previous two major versions.
## How GitLab Shell works
When you access the GitLab server over SSH then GitLab Shell will:
1. Limit you to predefined git commands (`git push`, `git pull`, `git fetch`).
1. Call the GitLab Rails API to check if you are authorized, and what Gitaly server your repository is on
1. Copy data back and forth between the SSH client and the Gitaly server
If you access a GitLab server over HTTP(S) you end up in [gitlab-workhorse](https://gitlab.com/gitlab-org/gitlab/tree/master/workhorse).
### `git pull` over SSH
1. git pull over SSH -> gitlab-shell -> API call to gitlab-rails (Authorization) -> accept or decline -> establish Gitaly session
### `git push` over SSH
1. git push over SSH -> gitlab-shell (git command is not executed yet) -> establish Gitaly session -> (in Gitaly) gitlab-shell pre-receive hook -> API call to gitlab-rails (authorization) -> accept or decline push
[Full feature list](doc/features.md)
### Modifies `authorized_keys`
GitLab Shell modifies the `authorized_keys` file on the client machine.
## Rate Limiting
GitLab Shell performs rate-limiting by user account and project for git operations. GitLab Shell accepts git operation requests and then makes a call to the Rails rate-limiter (backed by Redis). If the `user + project` exceeds the rate limit then GitLab Shell will then drop further connection requests for that `user + project`.
The rate-limiter is applied at the git command (plumbing) level. Each command has a rate limit of 600/minute. For example, `git push` has 600/minute and `git pull` has another 600/minute.
Because they are using the same plumbing command `git-upload-pack`, `git pull` and `git clone` are in effect the same command for the purposes of rate-limiting.
There is also a rate-limiter in place in Gitaly, but the calls will never be made to Gitaly if the rate limit is exceeded in Gitlab Shell (Rails).
## GitLab SaaS
A diagram of the flow of `gitlab-shell` on GitLab.com:
```mermaid
graph LR
a2 --> b2
a2 --> b3
a2 --> b4
b2 --> c1
b3 --> c1
b4 --> c1
c2 --> d1
c2 --> d2
c2 --> d3
d1 --> e1
d2 --> e1
d3 --> e1
a1[Cloudflare] --> a2[TCP<br/> load balancer]
e1[Git]
subgraph HAProxy Fleet
b2[HAProxy]
b3[HAProxy]
b4[HAProxy]
end
subgraph GKE
c1[Internal TCP<br/> load balancer<br/>port 2222] --> c2[GitLab-shell<br/> pods]
end
subgraph Gitaly
d1[Gitaly]
d2[Gitaly]
d3[Gitaly]
end
```
## Releasing
See [PROCESS.md](./PROCESS.md)
## Contributing
- See [CONTRIBUTING.md](./CONTRIBUTING.md).
- See the [beginner's guide](doc/beginners_guide.md).
## License
See [LICENSE](./LICENSE).
## Related topics
- [Using the GitLab Shell chart](https://docs.gitlab.com/charts/charts/gitlab/gitlab-shell/#using-the-gitlab-shell-chart)
The documentation for GitLab Shell [has moved into the `gitlab` repository](https://docs.gitlab.com/ee/development/gitlab_shell/).
---
stage: Create
group: Source Code
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
---
# GitLab Shell architecture
```mermaid
sequenceDiagram
participant Git on client
participant SSH server
participant AuthorizedKeysCommand
participant GitLab Shell
participant Rails
participant Gitaly
participant Git on server
Note left of Git on client: git fetch
Git on client->>+SSH server: ssh git fetch-pack request
SSH server->>+AuthorizedKeysCommand: gitlab-shell-authorized-keys-check git AAAA...
AuthorizedKeysCommand->>+Rails: GET /internal/api/authorized_keys?key=AAAA...
Note right of Rails: Lookup key ID
Rails-->>-AuthorizedKeysCommand: 200 OK, command="gitlab-shell upload-pack key_id=1"
AuthorizedKeysCommand-->>-SSH server: command="gitlab-shell upload-pack key_id=1"
SSH server->>+GitLab Shell: gitlab-shell upload-pack key_id=1
GitLab Shell->>+Rails: GET /internal/api/allowed?action=upload_pack&key_id=1
Note right of Rails: Auth check
Rails-->>-GitLab Shell: 200 OK, { gitaly: ... }
GitLab Shell->>+Gitaly: SSHService.SSHUploadPack request
Gitaly->>+Git on server: git upload-pack request
Note over Git on client,Git on server: Bidirectional communication between Git client and server
Git on server-->>-Gitaly: git upload-pack response
Gitaly -->>-GitLab Shell: SSHService.SSHUploadPack response
GitLab Shell-->>-SSH server: gitlab-shell upload-pack response
SSH server-->>-Git on client: ssh git fetch-pack response
```
The diagram on this page
[has moved into the `gitlab` repository](https://docs.gitlab.com/ee/development/gitlab_shell/#gitlab-shell-architecture).
---
stage: Create
group: Source Code
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments
---
# Beginner's guide to GitLab Shell contributions
## Check
Checks if GitLab API access and Redis via internal API can be reached:
```shell
make check
```
## Compile
Builds the `gitlab-shell` binaries, placing them into `bin/`.
```shell
make compile
```
## Install
Builds the `gitlab-shell` binaries and installs them onto the file system. The
default location is `/usr/local`, but you can change the location by setting the `PREFIX`
and `DESTDIR` environment variables.
```shell
make install
```
## Setup
This command is intended for use when installing GitLab from source on a single
machine. It compiles the GitLab Shell binaries, and ensures that
various paths on the file system exist with the correct permissions. Do not run
this command unless your installation method documentation instructs you to.
```shell
make setup
```
## Testing
Run tests:
```shell
bundle install
make test
```
Run Gofmt:
```shell
make verify
```
Run both test and verify (the default Makefile target):
```shell
bundle install
make validate
```
## Gitaly
Some tests need a Gitaly server. The
[`docker-compose.yml`](../docker-compose.yml) file runs Gitaly on port 8075.
To tell the tests where Gitaly is, set `GITALY_CONNECTION_INFO`:
```plaintext
export GITALY_CONNECTION_INFO='{"address": "tcp://localhost:8075", "storage": "default"}'
make test
```
If no `GITALY_CONNECTION_INFO` is set, the test suite still runs, but any
tests requiring Gitaly are skipped. The tests always run in the CI environment.
## Logging Guidelines
In general, you can determine the structure, but not content, of a GitLab Shell
or `gitlab-sshd` session by inspecting the logs. Some guidelines:
- We use [`gitlab.com/gitlab-org/labkit/log`](https://pkg.go.dev/gitlab.com/gitlab-org/labkit/log)
for logging.
- Always include a correlation ID.
- Log messages should be invariant and unique. Include accessory information in
fields, using `log.WithField`, `log.WithFields`, or `log.WithError`.
- Log both success cases and error cases.
- Logging too much is better than not logging enough. If a message seems too
verbose, consider reducing the log level before removing the message.
The information on this page
[has moved into the `gitlab` repository](https://docs.gitlab.com/ee/development/gitlab_shell/#contribute-to-gitlab-shell).
---
stage: Create
group: Source Code
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments
---
# Feature list
## Discover
Allows users to identify themselves on an instance via SSH. The command helps to
confirm quickly whether a user has SSH access to the instance:
```shell
ssh git@<hostname>
PTY allocation request failed on channel 0
Welcome to GitLab, @username!
Connection to staging.gitlab.com closed.
```
When permission is denied, it returns:
```shell
ssh git@<hostname>
git@<hostname>: Permission denied (publickey).
```
## Git operations
GitLab Shell provides support for Git operations over SSH by processing
`git-upload-pack`, `git-receive-pack` and `git-upload-archive` SSH commands.
It limits the set of commands to predefined Git commands:
- `git archive`
- `git clone`
- `git pull`
- `git push`
## Generate new 2FA recovery codes
Enables users to
[generate new 2FA recovery codes](https://docs.gitlab.com/ee/user/profile/account/two_factor_authentication.html#generate-new-recovery-codes-using-ssh):
```plaintext
ssh git@<hostname> 2fa_recovery_codes
Are you sure you want to generate new two-factor recovery codes?
Any existing recovery codes you saved will be invalidated. (yes/no)
yes
Your two-factor authentication recovery codes are:
...
```
## Verify 2FA OTP
Allows users to verify their
[2FA one-time password (OTP)](https://docs.gitlab.com/ee/security/two_factor_authentication.html#2fa-for-git-over-ssh-operations):
```shell
ssh git@<hostname> 2fa_verify
OTP: 347419
OTP validation failed.
```
## LFS authentication
Enables users to generate credentials for LFS authentication:
```shell
ssh git@<hostname> git-lfs-authenticate <project-path> <upload/download>
{"header":{"Authorization":"Basic ..."},"href":"https://gitlab.com/user/project.git/info/lfs","expires_in":7200}
```
## Personal access token
Enables users to use personal access tokens via SSH:
```shell
ssh git@<hostname> personal_access_token <name> <scope1[,scope2,...]> [ttl_days]
Token: glpat-...
Scopes: api
Expires: 2022-02-05
```
This page
[has moved into the `gitlab` repository](https://docs.gitlab.com/ee/development/gitlab_shell/features.html).
---
stage: Create
group: Source Code
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
---
# gitlab-sshd
`gitlab-sshd` is a binary in [`gitlab-shell`](https://gitlab.com/gitlab-org/gitlab-shell)
which runs as a persistent SSH daemon. It will replace `OpenSSH` on GitLab SaaS,
and eventually other cloud-native environments. Instead of running an `sshd` process,
we run a `gitlab-sshd` process that does the same job, in a more focused manner:
```mermaid
sequenceDiagram
participant Git on client
participant GitLab SSHD
participant Rails
participant Gitaly
participant Git on server
Note left of Git on client: git fetch
Git on client->>+GitLab SSHD: ssh git fetch-pack request
GitLab SSHD->>+Rails: GET /internal/api/authorized_keys?key=AAAA...
Note right of Rails: Lookup key ID
Rails-->>-GitLab SSHD: 200 OK, command="gitlab-shell upload-pack key_id=1"
GitLab SSHD->>+Rails: GET /internal/api/allowed?action=upload_pack&key_id=1
Note right of Rails: Auth check
Rails-->>-GitLab SSHD: 200 OK, { gitaly: ... }
GitLab SSHD->>+Gitaly: SSHService.SSHUploadPack request
Gitaly->>+Git on server: git upload-pack request
Note over Git on client,Git on server: Bidirectional communication between Git client and server
Git on server-->>-Gitaly: git upload-pack response
Gitaly -->>-GitLab SSHD: SSHService.SSHUploadPack response
GitLab SSHD-->>-Git on client: ssh git fetch-pack response
```
This page
[has moved into the `gitlab` repository](https://docs.gitlab.com/ee/development/gitlab_shell/gitlab_sshd.html).
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment