Research and discussion: Provisioning Container Registry Database for Omnibus
Before implementing the Omnibus change, we'll need to develop a plan. Discussions have been happening in gitlab-org/quality/reference-architectures#246 and gitlab-org/distribution/team-tasks#606 (closed) but we don't have an actionable implementation path.
Objectives of this issue:
- Continue discussions and make a decision on how to begin the Omnibus implementation
- Break down Omnibus changes into iterative tasks for scheduling in subsequent milestones.
Designs
- Show closed items
Relates to
Activity
-
Newest first Oldest first
-
Show all activity Show comments only Show history only
added Category:Container Registry backend devopspackage golang sectionops labels
- Crystal Poole marked this issue as related to #1074 (closed)
marked this issue as related to #1074 (closed)
- 🤖 GitLab Bot 🤖 added sectionci label and removed sectionops label
added sectionci label and removed sectionops label
- Crystal Poole changed milestone to %16.4
changed milestone to %16.4
- Crystal Poole mentioned in issue #1074 (closed)
mentioned in issue #1074 (closed)
- Maintainer
@crystalpoole - please see the following guidance and update this issue.1 Error Please add typebug typefeature, typemaintenance and a subtype label to this issue. If you do not feel the purpose of this issue matches one of the types, you may apply the typeignore label to exclude it from type tracking metrics and future prompts.
This message was generated automatically. You're welcome to improve it.
- Crystal Poole added typefeature label
added typefeature label
- Maintainer
Setting label(s) groupcontainer registry based on Category:Container Registry.
- 🤖 GitLab Bot 🤖 added groupcontainer registry label
added groupcontainer registry label
- Hayley Swimelar added headway-self-managed-registry-db label
added headway-self-managed-registry-db label
- Jaime Martinez assigned to @jaime
assigned to @jaime
- Jaime Martinez added workflowin dev label
added workflowin dev label
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#13859 (closed)
mentioned in issue gitlab-org/quality/triage-reports#13859 (closed)
- Maintainer
Notes
I've been learning about Omnibus and reading a bit through the code as well. I'll add some notes as I find things out in this comment.
There are 3 scenarios that apply when installing GitLab using Omnibus:
- Single node installation
- Multiple node installation for replication and failover
- Geo installations
-
Looking at the Reference architectures, we recommend for installations of up to 2k users.
-
For 3k+ users, we always recommend using at least 3 nodes for the Database component:
-
Geo is a special use case https://docs.gitlab.com/ee/administration/geo/ and so far it seems like it would need to be its own epic given the complexity around replication and the Geo tracking database.
As for Container Registry: Using a database to store m... (gitlab-org/distribution/team-tasks#606 - closed), I see that we have Enable multi-database creation in the database_... (omnibus-gitlab#6638 - closed) with Automatically create additional databases and r... (omnibus-gitlab!5492 - merged) in the review process.
From that MR, creating additional DBs on the same PostgreSQL instance is possible, and perhaps the easiest way to move forward. There are some unknowns so I've left a question in there. It also looks like the MR has stalled so we might need to look into our own method for the registry DB.
@hswimelar seems to me that there are various pieces moving but nothing concrete exists yet. I see a few scenarios moving forward:
- a) Encourage beta users to provision their own PostgreSQL instance. We need Container Registry: expose database configurati... (#1071 - closed) for this.
- b) Add a more formal step-by-step guide to the beta program issue, based on the PoC findings and Create walkthroughs/demos for Experimental inst... (#1068 - closed) to enable the DB. We also need #1071 (closed).
- c) Wait for omnibus-gitlab#6638 (closed) and related issues that formally support this. I think this would be ideal for going to GA but not for the current beta phase.
Perhaps a good middle step that will help for scenarios a) and b) would be to add a subcommand to
gitlab-ctl
to deal with the registry migrations. This could probably be as simple as doing some aliases as Omnibus should already have access to the registry binary and the configuration files. - Maintainer
add a subcommand to
gitlab-ctl
to deal with the registry migrationsThis does seem the right way to go, and likely how migration will then be automatically applied later on (related to omnibus-gitlab!5492 (comment 1533189226)). This command is already used to wrap the registry binary for running offline GC.
As for options A, B and C, I'd favor A, as we're aiming for technically advanced beta testers.
1 Collapse replies - Maintainer
@jaime here's the current implementation of offline garbage collection in omnibus: https://gitlab.com/gitlab-org/omnibus-gitlab/-/blob/master/files/gitlab-ctl-commands/registry_garbage_collect.rb It should be relatively easy to create a command for migrations, as well as a basic migration command with downtime.
I also favor A, we should get user feedback before investing too much time into refining this part of the experience. As João mentioned, we're looking for advanced beta testers. We're not sure how much extra workload will go into supporting the very first stages of this beta program. As we improve the user experience, or if we find that our workload isn't increasing due to supporting beta users, we can start to try reaching a broader audience.
1 - Maintainer
Thanks for the context! I've opened Add registry migration subcommand to Omnibus' g... (#1108 - closed) with some rough details for now.
- Jaime Martinez added to epic &5521
added to epic &5521
- Jaime Martinez changed epic to &10757
changed epic to &10757
- Jaime Martinez marked this issue as related to #1108 (closed)
marked this issue as related to #1108 (closed)
- Jaime Martinez mentioned in issue #1108 (closed)
mentioned in issue #1108 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#13978 (closed)
mentioned in issue gitlab-org/quality/triage-reports#13978 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#14076 (closed)
mentioned in issue gitlab-org/quality/triage-reports#14076 (closed)
- Hayley Swimelar mentioned in epic &5521
mentioned in epic &5521
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#14145 (closed)
mentioned in issue gitlab-org/quality/triage-reports#14145 (closed)
- Maintainer
Status Update
From the previous notes we decided to get users in the beta program to provision their own Postgres database. We have steps documented to perform schema migrations, and to migrate content from filesystem to the metadata database. For now, this is enough to get feedback from beta users to understand if/when we will need to provision the database for them in future iterations.
Based on this, I think this issue will be ongoing until we have a concrete set of steps to make the provisioning happen.
As of now, we still don't have a clear path forward on how to provision a database for the registry using Omnibus. The main unknowns are:
- Provision a single logical database that uses the same main Postgres instance as Rails, this would be similar to Omnibus provisions additional databases configured (omnibus-gitlab#6193 - closed), but:
- Running schema migrations for the registry database follows a different process than the main and CI databases.
- Backing up and restoring mechanisms could be similar if they don't need to know any of the registry specifics.
- Provision a separate Postgres instance. This has a lot of more unknowns:
- Will we support replication? If so,
- How to configure and manage PgBouncer?
- Can we just copy the process for the main DB and adapt it to the registry needs?
- Same as above in regards to back up, restoring and running migrations.
I'm sure there are plenty more unknowns we don't know yet.
@crystalpoole @trizzi these are some of the questions we need to answer in order to get #1074 (closed) across the line, so I don't think we are ready to tackle that issue in %16.5. @twk3 Who can we ask to help us answer these questions?
cc @hswimelar
1 - Provision a single logical database that uses the same main Postgres instance as Rails, this would be similar to Omnibus provisions additional databases configured (omnibus-gitlab#6193 - closed), but:
Collapse replies - Author Developer
Thank you for the update @jaime. Sounds like we need to wait for Omnibus provisions additional databases configured (omnibus-gitlab#6193 - closed) to be completed in 16.6 before starting on Provision a Container Registry database for Omn... (#1074 - closed).
Thanks, I moved #1074 (closed) to the backlog and marked it as blocked by omnibus-gitlab#6193 (closed).
(I also updated the milestone planning issue)
1 1
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#14239 (closed)
mentioned in issue gitlab-org/quality/triage-reports#14239 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#14353 (closed)
mentioned in issue gitlab-org/quality/triage-reports#14353 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#14445 (closed)
mentioned in issue gitlab-org/quality/triage-reports#14445 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#14516 (closed)
mentioned in issue gitlab-org/quality/triage-reports#14516 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#14611 (closed)
mentioned in issue gitlab-org/quality/triage-reports#14611 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#14677 (closed)
mentioned in issue gitlab-org/quality/triage-reports#14677 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#14795 (closed)
mentioned in issue gitlab-org/quality/triage-reports#14795 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#14890 (closed)
mentioned in issue gitlab-org/quality/triage-reports#14890 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#14961 (closed)
mentioned in issue gitlab-org/quality/triage-reports#14961 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#15057 (closed)
mentioned in issue gitlab-org/quality/triage-reports#15057 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#15171 (closed)
mentioned in issue gitlab-org/quality/triage-reports#15171 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#15266 (closed)
mentioned in issue gitlab-org/quality/triage-reports#15266 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#15386 (closed)
mentioned in issue gitlab-org/quality/triage-reports#15386 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#15482 (closed)
mentioned in issue gitlab-org/quality/triage-reports#15482 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#15549 (closed)
mentioned in issue gitlab-org/quality/triage-reports#15549 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#15690 (closed)
mentioned in issue gitlab-org/quality/triage-reports#15690 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#15814 (closed)
mentioned in issue gitlab-org/quality/triage-reports#15814 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#15887 (closed)
mentioned in issue gitlab-org/quality/triage-reports#15887 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#15997 (closed)
mentioned in issue gitlab-org/quality/triage-reports#15997 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#16167 (closed)
mentioned in issue gitlab-org/quality/triage-reports#16167 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#16270 (closed)
mentioned in issue gitlab-org/quality/triage-reports#16270 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#16353 (closed)
mentioned in issue gitlab-org/quality/triage-reports#16353 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#16459 (closed)
mentioned in issue gitlab-org/quality/triage-reports#16459 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#16628 (closed)
mentioned in issue gitlab-org/quality/triage-reports#16628 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#16729 (closed)
mentioned in issue gitlab-org/quality/triage-reports#16729 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#16812 (closed)
mentioned in issue gitlab-org/quality/triage-reports#16812 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#16918 (closed)
mentioned in issue gitlab-org/quality/triage-reports#16918 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#16997 (closed)
mentioned in issue gitlab-org/quality/triage-reports#16997 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#17190 (closed)
mentioned in issue gitlab-org/quality/triage-reports#17190 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#17270 (closed)
mentioned in issue gitlab-org/quality/triage-reports#17270 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#17346 (closed)
mentioned in issue gitlab-org/quality/triage-reports#17346 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#17445 (closed)
mentioned in issue gitlab-org/quality/triage-reports#17445 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#17612 (closed)
mentioned in issue gitlab-org/quality/triage-reports#17612 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#17715 (closed)
mentioned in issue gitlab-org/quality/triage-reports#17715 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#17797 (closed)
mentioned in issue gitlab-org/quality/triage-reports#17797 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#17903 (closed)
mentioned in issue gitlab-org/quality/triage-reports#17903 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#18068 (closed)
mentioned in issue gitlab-org/quality/triage-reports#18068 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#18167 (closed)
mentioned in issue gitlab-org/quality/triage-reports#18167 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#18251 (closed)
mentioned in issue gitlab-org/quality/triage-reports#18251 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#18359 (closed)
mentioned in issue gitlab-org/quality/triage-reports#18359 (closed)
- 🤖 GitLab Bot 🤖 mentioned in issue gitlab-org/quality/triage-reports#18440 (closed)
mentioned in issue gitlab-org/quality/triage-reports#18440 (closed)
- Maintainer
I have not been actively working on this. I will unassign myself and put it back as workflowready for development
- Jaime Martinez changed milestone to %Backlog
changed milestone to %Backlog
- Jaime Martinez added workflowready for development label and removed workflowin dev label
added workflowready for development label and removed workflowin dev label
- Jaime Martinez unassigned @jaime
unassigned @jaime
- Hayley Swimelar mentioned in issue gitlab#480742 (closed)
mentioned in issue gitlab#480742 (closed)
- Maintainer
For everyone following this issue, I'm trying to consolidate discussion into this issue as much as possible: Include a Default Database for the Container Re... (gitlab#480742 - closed) • Hayley Swimelar • 17.8 • At risk
Collapse replies - Maintainer
Discussion in the above issue has concluded so this issue should be closed as well.
- Hayley Swimelar closed
closed