Source Code Backend: Cells and Protocells Support

Workstream: Cells and Protocells Support

Description

This issue serves as a container track all Source Code Backend issues related to delivering Cells and Protocells, as part of the overall cells program. The issue will continue to be updated as issues progress.

Parent Issues

Cells [Epic]: Protocells (gitlab-com/gl-infra&1616)
Protocell Interlock [Epic]: Protocells Code Yellow Interlock - Migrate Firs... (gitlab-org&18384)
Source Code Backend Planning (1.0): (Size: L) Create:Source Code - Cells 1.0 Support (gitlab-org&13627)

High-Level Overview

Revise all workstreams and update deliverable high confidence target dates
Update all Source Code Backend Tables to support sharding
Ensure Unique Indexes across sharded tables
SSH Routing Service

Protocells Development Work

title: Protocells Issues
display: table
fields: state, status, title, epic, assignee, labels("Protocells::Sharding"), weight, milestone, updated
sort: closed asc
limit: 50
query: project = "gitlab-org/gitlab"
  and type = Issue
  and state = all
  and label = ("group::source code", "Protocells::*")
  and label != ("Protocells::Unique Indexes")

1. Protocells Workstreams: Table Sharding

1a. Table Sharding Priorities

priority1 push_rules
priority2 gpg_key_subkeys
priority2lfs_objects_projects (blocks lfs_objects)
priority3 x509_issuers
priority3 lfs_objects
priority4 pool_repositories
priority4 x509_certificates
priority4 snippet_repositories

1b. Table Sharding Workstreams

Group	Workstream	Status‎‎	Issue	Target	POC
Push Rules Phase 2	Complete `group_push_rules` migration	Done	Group push rules: extract shared logic from `PushRules` into a module	-	@jtapiab
		Done	Group push rules: execute read operations from `group_push_rules` table	18.5	@jtapiab
		Done	Group push rules: stop updating `push_rules` table	18.5	@jtapiab
	Complete `organization_push_rules` migration	Done	Organization push rules: extract shared logic from `PushRules` into a module	-	@jtapiab
		Done	Organization push rules: execute read operations from `organization_push_rules` table	18.4	@olaoluro
		In Progress	Organization push rules: stop updating `push_rules` table	18.5	@olaoluro
	Convert `push_rules` into Project push rules table	Done	Project push rules: Ensure that `regexp_uses_re2` column from `push_rules` is always `true`	-	@nav-j
		Done	Project push rules: Move push rules inheritance logic in services and finders	-	@olaoluro
		In Progress	Project push rules: Remove organization and group records	18.5	@olaoluro
		Not Started	Project push rules: Add missing attributes for `project_id` field	18.5	@olaoluro
				-
GPG Keys	gpg_keys	Done	gitlab-org/gitlab#553458 (closed)	18.4	@olaoluro
	gpg_key_subkeys	Done	gitlab-org/gitlab#553457 (closed)	18.4	@olaoluro
				-
LFS Objects	lfs_objects_projects	In Progress	BBM to populate lfs_objects_projects.oid (gitlab-org/gitlab#546679 - closed) (MR: gitlab-org/gitlab!207128 (merged))	18.5	@kerrizor
	lfs_objects	Not Started	Add sharding key for `lfs_objects` (gitlab-org/gitlab#490482)	18.5	@kerrizor
				-
x509	x509_issuers	In Progress	gitlab-org/gitlab#553481 (Needed additional info)	18.5	@vyaklushin
	x509_certificates	In Progress	gitlab-org/gitlab#553480	18.6	@vyaklushin
				-
Pool Repositories	Schema Updates	Not Started	Add `NOT NULL` constraint to `organization_id` ... (gitlab-org/gitlab#519620)	18.5	@hustewart
	Schema Updates	Done	Add the `organization_id` column to `pool_repos... (gitlab-org/gitlab#519615 - closed)	18.5	@hustewart
	Schema Updates	In Progress	Add sharding key for `pool_repositories` (gitlab-org/gitlab#490484)	18.5	@hustewart
	Backfill	In Progress	Backfill `pool_repositories` records that don't... (gitlab-org/gitlab#519617)	18.5	@hustewart
				-
Misc.	content_blocked_states	Done	gitlab-org/gitlab#553455 (closed)	-
	programming_languages	Done	gitlab-org/gitlab#553468 (closed)	-
	snippets_repositories	In Progress	Define a sharding key for snippet_repositories (gitlab-org/gitlab#550705 - closed) (Blocked by loose foreign key issue here)	18.5	@hustewart

1c. Miscellaneous Issues (Need additional info)

Add sharding key for `push_rules` (gitlab-org/gitlab#490488) • Vasilii Iakliushin • 18.6 (Can this issue be considered complete?)
Push rules: add a unique index by project_id (gitlab-org/gitlab#493391) • Vasilii Iakliushin • 18.6 (MR merged, can this issue be considered complete?)

2. Post Migration Follow Up

Push Rules Cleanup: Follow-ups: Push rule migration (gitlab-org&19227)

3. Unique Indexes for Sharded Tables

Epic: gitlab-org&18840

display: table
title: Source Code Unique Index Tables
description: List of tables that require unique indexes
fields: state, title, assignee, epic, milestone, weight, updated
sort: closed desc
limit: 20
query: 
  project = "gitlab-org/gitlab" AND 
  labels = ("group::source code", "Protocells::Unique Indexes")

4. SSH Routing Service

Estimated delivery date: Q1 2027 - will need to schedule before

Epic: Cells: Cells: SSH Routing for Git Pull and Push (gitlab-org&13532)

Overview

Workstream	Status	Action Item	Issue	Notes
Database Sharding	Done	content_blocked_states	gitlab-org/gitlab#553455 (closed)	This table will be moved to JiHu: gitlab-org/gitlab#560435 (closed)
	Done	programming_languages	gitlab-org/gitlab#553468 (closed)	8/12: Closed in favor of gitlab-org/gitlab#519895
	Done	gpg_keys	gitlab-org/gitlab#553458 (closed)
	Done	gpg_key_subkeys	gitlab-org/gitlab#553457 (closed)
	In Progress	push_rules	gitlab-org/gitlab#490488	[Done] Phase 1, [Not Started] Phase 2
	In Progress	pool_repositories	gitlab-org/gitlab#490484
	In Progress	x509_certificates	gitlab-org/gitlab#553480
	In Progress	lfs_objects_projects	gitlab-org/gitlab#546679 (closed)
	Not Started	lfs_objects	gitlab-org/gitlab#490482
	Not Started	x509_issuers	gitlab-org/gitlab#553481	Additional info needed to finalize approach
	In Progress	snippet_repositories	gitlab-org/gitlab#550705 (closed)

Additional Resources

Sharding Key Dashboard: https://cells-progress-tracker-gitlab-org-tenant-scale-g-f4ad96bf01d25f.gitlab.io/sharding_summary
Cells 1.0 Tracking Issue: Track Cells 1.0 work for Core Development (gitlab-org/gitlab#456762) • Kerri Miller

Edited Oct 21, 2025 by Andre Richards