Azure External Object Storage Research
Summary
I am working with a customer whose requirement involves using Azure Storage for external object storage. We kept having issues, even after several calls and communication within the team. So I decided to reproduce the use-case.
Steps to reproduce
- I created a cluster on GKE following the directions in the docs
- Created a storage account on Azure and containers for lfs, uploads, backups, etc.
- I created a Minio-Azure gateway using the guide: https://withblue.ink/2017/10/29/how-to-use-s3cmd-and-any-other-amazon-s3-compatible-app-with-azure-blob-storage.html
- Created Secrets as required for external storage
- Deployed GitLab
But I used configurations not documented, see below...
Configuration used
s3cmd.config:
[default]
# Setup endpoint using minio-azure gateway
host_base = abuangominio.azurewebsites.net
host_bucket = abuangominio.azurewebsites.net
# Leave as default
bucket_location = us-east-1
use_https = True
# Setup access keys
# Access Key = Azure Storage Account name
access_key = abubakargitlabtest
# Secret Key = Azure Storage Account Key
secret_key = OUm4Ab6zvWN2tn7MuYOSj+fqS6FgG/MCb46tZI/#############==
# Use S3 v4 signature APIs
signature_v2 = False
Object Storage Connection:
provider: AWS
region: us-east-1
aws_access_key_id: abubakargitlabtest
aws_secret_access_key: OUm4Ab6zvWN2tn7MuYOSj+fqS6FgG/MCb46tZI/############
endpoint: core.windows.net
Registry Object Storage (registry.azure.yaml
):
azure:
accountname: abubakargitlabtest
accountkey: OUm4Ab6zvWN2tn7MuYOSj+fqS6FgG/###########==
container: gitlab-registry
Secrets were created:
➜ kubectl create secret generic gitlab-rails-storage --from-file=connection=connection.yaml
secret "gitlab-rails-storage" created
➜ kubectl create secret generic s3cmd-config --from-file=config=s3cmd.config
secret "s3cmd-config" created
➜ kubectl create secret generic s3access --from-literal=accesskey="abubakargitlabtest" --from-literal=secretkey="OUm4Ab6zvWN2tn7MuYOSj+fqS6FgG/MCb46tZI/#########=="
secret "s3access" created
➜ kubectl create secret generic registry-storage --from-file=config=registry.azure.yaml
secret "registry-storage" created
My values.yaml
file (Edited the customer's copy)
# Default values for gitlab-chart.
# This is a YAML-formatted file.
global:
operator:
enabled: false
# gitlabVersion: master
application:
create: false
links: []
hosts:
domain: git.cn.abuango.me
https: true
externalIP: 35.189.120.48
ssh: ~
ingress:
configureCertmanager: true
annotations: {}
enabled: true
#tls:
# enabled: false
initialRootPassword: {}
redis:
password: {}
gitaly:
authToken: {}
internal:
names: ['default']
external: []
minio:
enabled: false
appConfig:
enableUsagePing: true
defaultCanCreateGroup: true
usernameChangingEnabled: true
issueClosingPattern:
defaultTheme:
defaultProjectsFeatures:
issues: true
mergeRequests: true
wiki: true
snippets: true
builds: true
webhookTimeout:
gravatar:
plainUrl:
sslUrl:
extra:
googleAnalyticsId:
piwikUrl:
piwikSiteId:
lfs:
bucket: gitlab-lfs
connection:
secret: gitlab-rails-storage
key: connection
artifacts:
bucket: gitlab-artifacts
connection:
secret: gitlab-rails-storage
key: connection
uploads:
bucket: gitlab-uploads
connection:
secret: gitlab-rails-storage
key: connection
packages:
bucket: gitlab-uploads
connection:
secret: gitlab-rails-storage
key: connection
backups:
bucket: gitlab-backup
connection:
secret: gitlab-rails-storage
key: connection
tmpBucket: tmp
incomingEmail:
enabled: false
address: ""
host: "imap.gmail.com"
port: 993
ssl: true
startTls: false
user: ""
password:
secret: ""
key: password
mailbox: inbox
idleTimeout: 60
ldap:
servers: {}
# 'main' is the GitLab 'provider ID' of this LDAP server
# main:
# label: 'LDAP'
# host: '_your_ldap_server'
# port: 636
# uid: 'sAMAccountName'
# bind_dn: '_the_full_dn_of_the_user_you_will_bind_with'
# password: '_the_password_of_the_bind_user'
# encryption: 'plain'
omniauth:
enabled: false
autoSignInWithProvider:
syncProfileFromProvider: []
syncProfileAttributes: ['email']
allowSingleSignOn: ['saml']
blockAutoCreatedUsers: true
autoLinkLdapUser: false
autoLinkSamlUser: false
externalProviders: []
providers: []
# - secret: gitlab-google-oauth2
# key: provider
pseudonymizer:
configMap:
bucket: gitlab-pseudo
connection: {}
shell:
authToken: {}
hostKeys: {}
railsSecrets: {}
registry:
bucket: gitlab-registry
certificate: {}
httpSecret: {}
# runner:
# registrationToken: "QoymCJ5guG4W8ODh1lA6GsqteVW8#######"
# Outgoing email server settings
smtp:
enabled: false
address: smtp.mailgun.org
port: 2525
user_name: ""
password:
secret: ""
key: password
# domain:
authentication: "plain"
starttls_auto: false
openssl_verify_mode: "peer"
# Email persona used in email sent by GitLab
email:
from: ''
display_name: GitLab
reply_to: ''
subject_suffix: ''
time_zone: UTC
service:
annotations: {}
antiAffinity: soft
workhorse: {}
# configuration of certificates container & custom CA injection
certificates:
image:
repository: registry.gitlab.com/gitlab-org/build/cng/alpine-certificates
tag: 20171114-r3
customCAs: []
# - secret: custom-CA
# - secret: more-custom-CAs
# Settings to for the Let's Encrypt ACME Issuer
certmanager-issuer:
# The email address to register certificates requested from Let's Encrypt. Required if using Let's Encrypt.
email: h4xx2xxxx@gmail.com
certmanager:
# Install cert-manager chart. Set to false if you already have cert-manager
# installed or if you are not using cert-manager.
install: true
external_url: https://gitlab.git.cn.abuango.me
# Other cert-manager configurations from upstream
# See https://github.com/kubernetes/charts/tree/master/stable/cert-manager#configuration
rbac:
create: true
nginx-ingress:
enabled: true
tcpExternalConfig: "true"
controller:
config:
hsts-include-subdomains: "false"
server-name-hash-bucket-size: "256"
enable-vts-status: "true"
use-http2: "false"
ssl-ciphers: "ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-AES256-SHA:ECDHE-RSA-AES128-SHA:AES256-GCM-SHA384:AES128-GCM-SHA256:AES256-SHA256:AES128-SHA256:AES256-SHA:AES128-SHA:!aNULL:!eNULL:!EXPORT:!DES:!MD5:!PSK:!RC4"
ssl-protocols: "TLSv1.1 TLSv1.2"
server-tokens: "false"
extraArgs:
force-namespace-isolation: ""
service:
externalTrafficPolicy: "Local"
resources:
requests:
cpu: 100m
memory: 100Mi
publishService:
enabled: true
replicaCount: 3
minAvailable: 2
scope:
enabled: true
stats:
enabled: true
metrics:
enabled: true
service:
annotations:
prometheus.io/scrape: "true"
prometheus.io/port: "10254"
defaultBackend:
minAvailable: 1
replicaCount: 2
resources:
requests:
cpu: 5m
memory: 5Mi
rbac:
create: true
serviceAccount:
create: true
prometheus:
install: true
rbac:
create: true
alertmanager:
enabled: false
alertmanagerFiles:
alertmanager.yml: {}
kubeStateMetrics:
enabled: false
nodeExporter:
enabled: false
pushgateway:
enabled: false
redis-ha:
nameOverride: redis
enabled: false
postgresql:
install: true
#postgresUser: gitlab
#postgresDatabase: gitlabhq_production
#imageTag: 9.6.8
#usePasswordFile: true
#existingSecret: 'secret'
#metrics:
# enabled: true
## Optionally define additional custom metrics
## ref: https://github.com/wrouesnel/postgres_exporter#adding-new-metrics-via-a-config-file
registry:
enabled: false
storage:
secret: registry-storage
key: config
#
# redis:
# enabled: false
shared-secrets:
enabled: true
rbac:
create: true
gitlab-runner:
install: true
gitlabUrl: https://gitlab.git.cn.abuango.me
# registrationToken: QoymCJ5guG4W8ODh1lA6Gsq########
rbac:
create: true
runners:
cache:
cacheType: s3
s3BucketName: gitlab-runner
cacheShared: true
s3BucketLocation: us-east-1
s3CachePath: gitlab-runner
s3CacheInsecure: false
s3ServerAddress: abuangominio.azurewebsites.net #minio-azure gateway
secretName: s3access
gitlab:
task-runner:
backups:
objectStorage:
config:
secret: s3cmd-config
key: config
# migrations:
# enabled: false
# unicorn:
# enabled: false
# sidekiq:
# enabled: false
# gitaly:
# enabled: false
# gitlab-shell:
# enabled: false
Installation command: helm upgrade --install -f values.yaml gitlab gitlab/gitlab
Everything was deployed successfully:
NAME READY STATUS RESTARTS AGE
gitlab-certmanager-6fb6f6944f-fztfs 1/1 Running 0 9m
gitlab-gitaly-0 1/1 Running 0 9m
gitlab-gitlab-runner-868d86fcc5-mnfq7 1/1 Running 6 9m
gitlab-gitlab-shell-74f98896cf-4wd4d 1/1 Running 0 9m
gitlab-gitlab-shell-74f98896cf-mshl6 1/1 Running 0 8m
gitlab-issuer.1-89qvm 0/1 Completed 0 9m
gitlab-migrations.1-4j2z5 0/1 Completed 0 9m
gitlab-nginx-ingress-controller-86c9649666-blcn5 1/1 Running 0 9m
gitlab-nginx-ingress-controller-86c9649666-rr8t6 1/1 Running 0 9m
gitlab-nginx-ingress-controller-86c9649666-wxrj9 1/1 Running 0 9m
gitlab-nginx-ingress-default-backend-b75f49dff-88r8x 1/1 Running 0 9m
gitlab-nginx-ingress-default-backend-b75f49dff-nzs49 1/1 Running 0 9m
gitlab-postgresql-65bd954977-jrsgs 2/2 Running 0 9m
gitlab-prometheus-server-69d46f5d6c-tdl79 2/2 Running 0 9m
gitlab-redis-7b9d4587f8-kf92q 2/2 Running 0 9m
gitlab-sidekiq-all-in-1-54856655d4-vrtk9 1/1 Running 0 9m
gitlab-task-runner-79b54b5fbc-fzc4z 1/1 Running 0 9m
gitlab-unicorn-58fbd86d6-qx5bx 2/2 Running 0 8m
gitlab-unicorn-58fbd86d6-v89ff 2/2 Running 0 9m
Though while testing, I got the following error while running a pipeline, but I guess this is another issue about runner configuation:
Checking for jobs... received job=4 repo_url=https://gitlab.git.cn.abuango.me/root/test-proj.git runner=nFKRpYUm
ERROR: Job failed (system failure): Secret "runner-nFKRpYUm-project-1-concurrent-0s275n" is invalid: [metadata.generateName: Invalid value: "runner-nFKRpYUm-project-1-concurrent-0": a DNS-1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*'), metadata.name: Invalid value: "runner-nFKRpYUm-project-1-concurrent-0s275n": a DNS-1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex usedfor validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*')] duration=8.434188ms job=4 project=1 runner=nFKRpYUm
ERROR: Error cleaning up secrets: resource name may not be empty job=4 project=1 runner=nFKRpYUm
Or did anything I did above cause the error?
Versions
- Chart:
NAME REVISION UPDATED STATUS CHART APP VERSION NAMESPACE
gitlab 1 Tue Jan 8 16:42:33 2019 DEPLOYED gitlab-1.4.2 11.6.3 default
-
Platform:
GKE with External Object Storage on Azure
-
Kubernetes: (
kubectl version
)
Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.7", GitCommit:"0c38c362511b20a098d7cd855f1314dad92c2780", GitTreeState:"clean", BuildDate:"2018-08-20T10:09:03Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"11+", GitVersion:"v1.11.5-gke.5", GitCommit:"9aba9c1237d9d2347bef28652b93b1cba3aca6d8", GitTreeState:"clean", BuildDate:"2018-12-11T02:36:50Z", GoVersion:"go1.10.3b4", Compiler:"gc", Platform:"linux/amd64"}
- Helm: (
helm version
)
Client: &version.Version{SemVer:"v2.11.0", GitCommit:"2e55dbe1fdb5fdb96b75ff144a339489417b146b", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.11.0", GitCommit:"2e55dbe1fdb5fdb96b75ff144a339489417b146b", GitTreeState:"clean"}
What do you think? What are the things I got wrong? Is this worth recommending to a customer? Is there a better approach?