Many databases can be configured in Omnibus

Following our guidelines many databases are supported by our Rails stack: https://docs.gitlab.com/ee/development/database/multiple_databases.html#multiple-databases.

Current

Currently config/database.yml as generated is in a form of:

development:
  adapter: postgresql
  encoding: unicode
  database: gitlabhq_development
  host: /path/to/gdk/postgresql
  pool: 10
  prepared_statements: false
  variables:
    statement_timeout: 120s

Proposed change

Perform transformation of config/database.yml to do the following:

  1. A database configuration is transformed into many-database syntax using main as a name of main database
  2. An additional database for CI can be configured

1. Transformed into many-databases

This is backward/forward-compatible change, that does not impact application after the gitlab!65054 (merged) is merged.

This requires that the config/database.yml config looks the following:

development:
  main:
    adapter: postgresql
    encoding: unicode
    database: gitlabhq_development
    host: /path/to/gdk/postgresql
    pool: 10
    prepared_statements: false
    variables:
      statement_timeout: 120s

2. Allow additional databases to be configured

We expect to have many additional databases used over time, with the CI being the first.

We need a scalable design to allow us to define all parameters required for such change. A configured Omnibus allows to enable ci: database and define all parameters of such database, initially pointing to exactly the same database server, but different database name.

We don't fully understand yet how we gonna run provisioning of such database, so, we expect in initial phase to just be able to emit config/database.yml with the CI entry added, but without any other automated provisioning unless done automatically by commands like db:create:

development:
  main:
    adapter: postgresql
    encoding: unicode
    database: gitlabhq_development
    host: /path/to/gdk/postgresql
    pool: 10
    prepared_statements: false
    variables:
      statement_timeout: 120s
  ci:
    adapter: postgresql
    encoding: unicode
    database: gitlabhq_development_ci
    ...
    migration_paths: db/ci_migrate

The Omnibus ideally should be configurable with set off gitlab.rb settings. Probably in a similar form:

### GitLab database settings
###! Docs: https://docs.gitlab.com/omnibus/settings/database.html
###! **Only needed if you use an external database.**
# gitlab_rails['db_adapter'] = "postgresql"
# gitlab_rails['db_encoding'] = "unicode"
# gitlab_rails['db_collation'] = nil
# gitlab_rails['db_database'] = "gitlabhq_production"
# gitlab_rails['db_username'] = "gitlab"
# ...

gitlab_rails['db_ci']['enable'] = true
gitlab_rails['db_ci']['db_database'] = "gitlabhq_production_ci"
gitlab_rails['db_ci']['db_adapter'] = # inherits gitlab_rails['db_adapter']
... and so on as for the inheritance of parameters
... we assume that in most cases only a `db_database` will be different

Tradeoffs

  1. We are aiming at exposing only configuration aspect for purpose of using that on GitLab.com
  2. We don't expect all our automation to work in this BETA phase for on-premise customers, the follow-ups should be created to handle all supported workflows related to migration unless they are covered automatically by Rails
Edited by Kamil Trzciński