Fleet management v2.0

In order to meet the coming demand and scale for GitLab.com, as well as to provide a truly flexible and scalable architecture, we are changing the way we deploy and build our infrastructure.

You are encouraged to read (and contribute to) the detailed plan as this meta-issue is just the actionable plan and assumes the knowledge of some definitions in it.

Bah. I don't have the time to read all that. What are we going to do?

• Clearly define classes of machines (cattle, pets, snowflakes)
• Leverage build and deployment tools to make all 'cattle' machines ephemeral
• Move all Secret Storage into a solution agnostic provider

Taking these steps allows us to being early stage usage of 'canary' deployments as well as align us with a smooth transition into Kubernetes containerisation and deployment.

We accomplish this by using packer and chef-solo for all our cattle. Everything, including pets and snowflakes, will be managed by terraform. All of our secrets will be stored in Vault, which will allow us to open source our chef-repo entirely.

The use of tagged, unambiguous images developed through this pipeline will enable us to do canary deployments and quickly promote code from testing, to stage, and on into production in a continuous manner.

So what do we need to do?

• Enable our chef repository to build vm and container images - https://gitlab.com/gitlab-com/infrastructure/issues/1211
• Use service discovery to configure the fleet in realtime instead of using chef-client - https://gitlab.com/gitlab-com/infrastructure/issues/1214
• Move our secrets from chef-vault to an agnostic provider solution - https://gitlab.com/gitlab-com/infrastructure/issues/1212
• Automate the lifecycle of staging environments - https://gitlab.com/gitlab-com/infrastructure/issues/1040
• Automate db migration in a safe, isolated environment - https://gitlab.com/gitlab-com/infrastructure/issues/1215
• Deploy feature branches in staging using review apps - https://gitlab.com/gitlab-com/infrastructure/issues/1035
• Automate the production lifecycle - https://gitlab.com/gitlab-com/infrastructure/issues/1216

Explain cattle, pets, snowflakes

• cattle: disposable machines - web workers, sidekiq workers, (redis?) etc.
• pets: persistent machines - storage, db.
• snowflakes: specific one off machines that can be persistent but do not have any particular scaling need.

For ex.

@omame

Everything else? , including pets and snowflakes

Another plus of using vault is that we will have a tighter management of secrets, and in many cases it will permit doing a self service for rotating ssh keys for ex.

added critical deploy labels

mentioned in issue #1216 (closed)

mentioned in issue #365 (closed)

LGTM

removed critical label

mentioned in issue #1268 (closed)

mentioned in issue #115 (closed)

mentioned in issue gitlab-org/omnibus-gitlab#1521 (closed)

mentioned in issue #1363 (closed)

mentioned in issue #1040 (closed)

Even though #1040 (closed) is completed we can't tick the box here as it depends on #1211 (closed) in this context.

@omame let's split the point in the issue then

Why? We just need to get to it. It's only the testing phase that's missing there. There's no need to split that out.

@omame I just realize I hadn't seen the google doc before. The doc is not so long :-) Can you update here, perhaps open new issues if needed, and deprecate the doc?

closed

I'm closing this issue in favor of https://gitlab.com/gitlab-com/infrastructure/issues/1504

The latter one is much more comprehensive and provides much more detail over this one.

removed milestone

mentioned in issue #3523 (closed)