Local GDK development should be configured with 2 write connections by default like phase 7

Once we ship phase 7 in &6160 (closed) we will have 2 separate write connections pointing to two DBs. We should do the same for local dev by default so we can catch issues earlier.~ We should start preparing the development environment for most developers to run like phase 7 (in which GitLab has 2 separate connections to 2 separate databases) to catch any potential issues earlier

Proposal

1. Encourage group of beta testers to use distinct ci: database => #356353 (closed)
2. Gather feedback ⏳
3. Enable multiple databases by default in GDK => gitlab-development-kit!2496 (merged)
4. Write guide on how to switch back from two databases to single database => !85374 (closed)
5. Enable multiple databases by default in GCK => gitlab-compose-kit!206 (merged)

FAQ / Troubleshooting

1. I would like to stay on single database.

   • Answer : You have two options:
   • Use single database, but two different connections (preferred)

   gdk config set gitlab.rails.databases.ci.use_main_database true
   gdk reconfigure

   • Use single database only (less preferred)

   gdk config set gitlab.rails.databases.ci.enabled false
   gdk reconfigure

2. Can I pin my config on single database prior to the switch ?

   • Answer : Yes, set the above configurations. GDK will respect your config

3. What Slack channel is best to ask questions about this ?

   • Answer : The #g_sharding channel

4. I can't see any CI data anymore when I switched back to a single database.

   • Answer: The data in the separate ci: database will remain there. You will not be able to see CI data
   • For now, we recommend you create a new GDK instance to test with a single database.

5. What are the differences in multiple databases ?

   • Answer: It's based on Rail's multiple database support. Most things will be the same.
   • Certain commands like bundle exec rails db:migrate:down will need to be run like this now:
     • bundle exec rails db:migrate:down:main
     • bundle exec rails db:migrate:down:ci
   • Read https://guides.rubyonrails.org/active_record_multiple_databases.html for reference

6. What is the plan ?

   • Answer: Assuming this test goes well, we plan to update GDK to:
     1. !82090 (merged) Enable multiple databases on CI (DONE)
     2. gitlab-development-kit!2464 (merged) use multiple database connections, but same underlying database (gitlab.rails.databases.ci.enabled to true, gitlab.rails.databases.ci.use_main_database to true). (DONE)
     3. use multiple database connections, with distinct underlying databases (gitlab.rails.databases.ci.enabled to true, gitlab.rails.databases.ci.use_main_database to false). Target date: 2022-04-20
     4. Note in next major milestone %16.0 we plan to make multiple databases mandatory for all GitLab instances.