index.md 4.55 KB
Newer Older
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
# Troubleshooting

## UPGRADE FAILED: "$name" has no deployed releases

This error will occur on your second install/upgrade if your initial
install failed.

If your initial install completely failed, and GitLab was never operational, you
should first purge the failed install before installing again.

```
helm delete --purge <release-name>
```

If instead, the initial install command timed out, but GitLab still came up successfully,
you can add the `--force` flag to the `helm upgrade` command to ignore the error
and attempt to update the release.

Otherwise, if you received this error after having previously had successful deploys
of the GitLab chart, then you are encountering a bug. Please open an issue on our
21 22
[issue tracker](https://gitlab.com/gitlab-org/charts/gitlab/issues), and also check out
[issue #630](https://gitlab.com/gitlab-org/charts/gitlab/issues/630) where we recovered our
23
CI server from this problem.
24

25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40
## Error: this command needs 2 arguments: release name, chart path

An error like this could occur when you run `helm upgrade`
and there are some spaces in the parameters. In the following
example, `Test Username` is the culprit:

```sh
helm upgrade gitlab gitlab/gitlab --timeout 600 --set global.email.display_name=Test Username ...
```

To fix it, pass the parameters in single quotes:

```sh
helm upgrade gitlab gitlab/gitlab --timeout 600 --set global.email.display_name='Test Username' ...
```

41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79
## Application containers constantly initializing

If you experience Sidekiq, Unicorn, or other Rails based containers in a constant
state of Initializing, you're likely waiting on the `dependencies` container to
pass.

If you check the logs of a given Pod specifically for the `dependencies` container,
you may see the following repeated:

```
Checking database connection and schema version
WARNING: This version of GitLab depends on gitlab-shell 8.7.1, ...
Database Schema
Current version: 0
Codebase version: 20190301182457
```

This is an indication that the `migrations` Job has not yet completed. The purpose
of this Job is to both ensure that the database is seeded, as well as all
relevant migrations are in place. The application containers are attempting to
wait for the database to be at or above their expected database version. This is
to ensure that the application does not malfunction to the schema not matching
expectations of the codebase.

1. Find the `migrations` Job. `kubectl get job -lapp=migrations`
1. Find the Pod being run by the Job. `kubectl get pod -ljob-name=<job-name>`
1. Examine the output, checking the `STATUS` column.

If the `STATUS` is `Running`, continue. If the `STATUS` is `Completed`, the application conainers should start shortly after the next check passes.

Examine the logs from this pod. `kubectl logs <pod-name>`

Any failures during the run of this job should be addressed. These will block
the use of the application until resolved. Possible problems are:

- Unreachable or failed authentication to the configured PostgreSQL database
- Unreachable or failed authentication to the configured Redis services
- Failure to reach a Gitaly instance

80 81 82 83 84 85 86
## Included GitLab Runner failing to register

This can happen when the runner registration token has been changed in GitLab. (This often happens after you have restored a backup)

1. Find the new shared runner token located on the `admin/runners` webpage of your GitLab installation.
1. Find the name of existing runner token Secret stored in Kubernetes

87 88 89
   ```
   kubectl get secrets | grep gitlab-runner-secret
   ```
90 91 92

1. Delete the existing secret

93 94 95
   ```
   kubectl delete secret <runner-secret-name>
   ```
96 97 98

1. Create the new secret with two keys, (`runner-regisration-token` with your shared token, and an empty `runner-token`)

99 100 101
   ```
   kubectl create secret generic <runner-secret-name> --from-literal=runner-registration-token=<new-shared-runner-token> --from-literal=runner-token=""
   ```
102 103 104 105 106

## Too many redirects

This can happen when you have TLS termination before the nginx ingress, and the tls-secrets are specified in the configuration.

107 108
1. Update your values to set `global.ingress.annotations."nginx.ingress.kubernetes.io/ssl-redirect": "false"`

109
   Via a values file:
110 111 112 113 114 115 116 117 118

   ```yml
   # values.yml
   global:
     ingress:
       annotations:
         "nginx.ingress.kubernetes.io/ssl-redirect": "false"
   ```

119
   Via the helm CLI:
120 121 122 123 124

   ```sh
   helm ... --set-string global.ingress.annotations."nginx.ingress.kubernetes.io/ssl-redirect"=false
   ```

125 126 127 128 129 130
1. Apply the change

>>>
**NOTE:**
When using an external service for SSL termination, that service is responsible for redirecting to https (if so desired).
>>>