Tags as cell-local accelerant. Duplicate tag name in source tables

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

Problem statement

The tags table is decided to be cell-local. In other words, Org mover will not move the tags table.

So we will need to store the tag name in other org tables so that we do not lose data when an org is moved.

Old description

@mbobin wrote:

tags table - what's the plan here to set the sharding key? We still have sharding_key_issue_url in the codebase.

gitlabhq_dblab=# \d+ tags
                                                                   Table "public.tags"
     Column     |          Type          | Collation | Nullable |             Default              | Storage  | Compression | Stats target | Description
----------------+------------------------+-----------+----------+----------------------------------+----------+-------------+--------------+-------------
 id             | integer                |           | not null | nextval('tags_id_seq'::regclass) | plain    |             |              |
 name           | character varying(510) |           |          | NULL::character varying          | extended |             |              |
 taggings_count | integer                |           |          | 0                                | plain    |             |              |

Action items

  • Refine and implement #470872 (comment 2382060671)
    1. p_ci_build_tags will be replaced with ci_job_definitions
    2. %18.5: #570215 (closed) We add ci_runner_taggings with name and tag_id. It drops type to reduce space usage compared to current taggings (polymorphic mapping).
      1. %18.5: #570217 (closed) Backfill name column
      2. %18.6: #570754 (closed) Finalize migration
    3. When OrgMoving we do not migrate tag_id, this will be reconstructed later to deduplicate cell-local.
    4. We retain tags as an cell-local acceleration table.
    5. %18.5: #547650 (closed) We drop ci_runners.sharding_key, instead add ci_runners.organization_id.
    6. We indirectly introduce a placeholder support for having organization-wide runners with having ci_runners.organization_id and later introducing additional runner_type=4.
    7. If we transfer project/group we will update ci_runners.organization_id as long as runner cross-join tables (ci_runner_projects and ci_runner_groups) we allow or deny transfer.
    8. %18.6: #578669 The ci_pending_builds will upsert ci_job_tags.name into tags and continue using tag_ids[].
    9. %18.6: #576030 The jobs/request will use ci_runner_taggings.tag_id to match ci_pending_builds.tag_ids[].
    10. %18.7: #578702 Convert FKs into tags from org tables into LFKs
Edited by Pedro Pombeiro