Skip to content

Introduce paper_trail to WorkspaceAgentConfig

zlirequested to merge
zl/479263 into master

Issue - Backend Migration & Model: Introduce versioning... (#479263 - closed)

What does this MR do and why?

This MR introduce paper_trail to remote_development_agent_configs table with all necessary migrations and configs.

MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Screenshots or screen recordings

See updated schema for all changes introduced.

[1] pry(main)> aa=RemoteDevelopment::RemoteDevelopmentAgentConfig.last
  RemoteDevelopment::RemoteDevelopmentAgentConfig Load (2.2ms)  SELECT "remote_development_agent_configs".* FROM "remote_development_agent_configs" ORDER BY "remote_development_agent_configs"."id" DESC LIMIT 1 /*application:console,db_config_name:main,console_hostname:Zhaochens-MacBook-Pro.local,console_username:zhaochen.li,line:(pry):1:in `__pry__'*/
=> #<RemoteDevelopment::RemoteDevelopmentAgentConfig:0x0000000317951400
 id: 16,
 created_at: Thu, 22 Aug 2024 02:36:38.556000000 UTC +00:00,
 updated_at: Fri, 23 Aug 2024 01:36:15.830214000 UTC +00:00,
 cluster_agent_id: 7,
 enabled: true,
 dns_zone: "workspaces.localdev.me",
 network_policy_enabled: true,
 gitlab_workspaces_proxy_namespace: "gitlab-workspaces",
 network_policy_egress:
  [{"allow"=>"0.0.0.0/0", "except"=>["10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16"]}, {"allow"=>"172.16.123.1/32"}],
 default_resources_per_workspace_container: {},
 max_resources_per_workspace: {},
 workspaces_quota: -1,
 workspaces_per_user_quota: -1,
 project_id: 19,
 default_max_hours_before_termination: 29,
 max_hours_before_termination_limit: 120>
[2] pry(main)> aa.versions
  RemoteDevelopment::WorkspaceAgentConfigVersion Load (2.3ms)  SELECT "workspace_agent_config_versions".* FROM "workspace_agent_config_versions" WHERE "workspace_agent_config_versions"."item_id" = 16 AND "workspace_agent_config_versions"."item_type" = 'RemoteDevelopment::RemoteDevelopmentAgentConfig' ORDER BY "workspace_agent_config_versions"."created_at" ASC, "workspace_agent_config_versions"."id" ASC /*application:console,db_config_name:main,console_hostname:Zhaochens-MacBook-Pro.local,console_username:zhaochen.li,line:bin/rails:4:in `<main>'*/
=> []
[3] pry(main)> aa.default_max_hours_before_termination = 99
=> 99
[4] pry(main)> aa.save!
  TRANSACTION (0.2ms)  BEGIN /*application:console,db_config_name:main,console_hostname:Zhaochens-MacBook-Pro.local,console_username:zhaochen.li,line:/lib/gitlab/database/schema_cache_with_renamed_table_legacy.rb:23:in `primary_keys'*/
  Clusters::Agent Load (3.0ms)  SELECT "cluster_agents".* FROM "cluster_agents" WHERE "cluster_agents"."id" = 7 LIMIT 1 /*application:console,db_config_name:main,console_hostname:Zhaochens-MacBook-Pro.local,console_username:zhaochen.li,line:(pry):4:in `__pry__'*/
  RemoteDevelopment::RemoteDevelopmentAgentConfig Update (2.4ms)  UPDATE "remote_development_agent_configs" SET "updated_at" = '2024-08-23 02:31:46.301342', "default_max_hours_before_termination" = 99 WHERE "remote_development_agent_configs"."id" = 16 /*application:console,db_config_name:main,console_hostname:Zhaochens-MacBook-Pro.local,console_username:zhaochen.li,line:(pry):4:in `__pry__'*/
  RemoteDevelopment::WorkspaceAgentConfigVersion Create (16.2ms)  INSERT INTO "workspace_agent_config_versions" ("item_type", "item_id", "event", "object", "created_at") VALUES ('RemoteDevelopment::RemoteDevelopmentAgentConfig', 16, 'update', '{"id":16,"created_at":"2024-08-22T02:36:38.556Z","updated_at":"2024-08-23T01:36:15.830Z","cluster_agent_id":7,"enabled":true,"dns_zone":"workspaces.localdev.me","network_policy_enabled":true,"gitlab_workspaces_proxy_namespace":"gitlab-workspaces","network_policy_egress":[{"allow":"0.0.0.0/0","except":["10.0.0.0/8","172.16.0.0/12","192.168.0.0/16"]},{"allow":"172.16.123.1/32"}],"default_resources_per_workspace_container":{},"max_resources_per_workspace":{},"workspaces_quota":-1,"workspaces_per_user_quota":-1,"project_id":19,"default_max_hours_before_termination":29,"max_hours_before_termination_limit":120}', '2024-08-23 02:31:46.301342') RETURNING "id" /*application:console,db_config_name:main,console_hostname:Zhaochens-MacBook-Pro.local,console_username:zhaochen.li,line:(pry):4:in `__pry__'*/
  TRANSACTION (0.4ms)  COMMIT /*application:console,db_config_name:main,console_hostname:Zhaochens-MacBook-Pro.local,console_username:zhaochen.li,line:/lib/gitlab/database.rb:403:in `commit'*/
=> true
[5] pry(main)> aa.versions
  RemoteDevelopment::WorkspaceAgentConfigVersion Load (0.6ms)  SELECT "workspace_agent_config_versions".* FROM "workspace_agent_config_versions" WHERE "workspace_agent_config_versions"."item_id" = 16 AND "workspace_agent_config_versions"."item_type" = 'RemoteDevelopment::RemoteDevelopmentAgentConfig' ORDER BY "workspace_agent_config_versions"."created_at" ASC, "workspace_agent_config_versions"."id" ASC /*application:console,db_config_name:main,console_hostname:Zhaochens-MacBook-Pro.local,console_username:zhaochen.li,line:bin/rails:4:in `<main>'*/
=> [#<RemoteDevelopment::WorkspaceAgentConfigVersion:0x000000030f55b060
  id: 1,
  item_type: "RemoteDevelopment::RemoteDevelopmentAgentConfig",
  item_id: 16,
  event: "update",
  whodunnit: nil,
  object:
   {"id"=>16,
    "enabled"=>true,
    "dns_zone"=>"workspaces.localdev.me",
    "created_at"=>"2024-08-22T02:36:38.556Z",
    "project_id"=>19,
    "updated_at"=>"2024-08-23T01:36:15.830Z",
    "cluster_agent_id"=>7,
    "workspaces_quota"=>-1,
    "network_policy_egress"=>
     [{"allow"=>"0.0.0.0/0", "except"=>["10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16"]}, {"allow"=>"172.16.123.1/32"}],
    "network_policy_enabled"=>true,
    "workspaces_per_user_quota"=>-1,
    "max_resources_per_workspace"=>{},
    "gitlab_workspaces_proxy_namespace"=>"gitlab-workspaces",
    "max_hours_before_termination_limit"=>120,
    "default_max_hours_before_termination"=>29,
    "default_resources_per_workspace_container"=>{}},
  project_id: 19,
  created_at: Fri, 23 Aug 2024 02:31:46.301342000 UTC +00:00>]

How to set up and validate locally

  1. checkout this branch
  2. run migration bundle exec rails db:migrate
  3. bundle exec rails c
  4. select or create one foo = RemoteDevelopment::RemoteDevelopmentAgentConfig.last
  5. check its versions, and should be empty array
  6. then update any field, and save
  7. check its versions, it should have one version saved with fields before update action, and also check its project_id is the set the same as foo.project_id via trigger

Database Review

Followed the doc here for new table on DB review.

What is the anticipated growth for the new table over the next 3 months, 6 months, 1 year? What assumptions are these based on?

Based on the current remote_development_agent_configs table size, which is around 100 rows, it is likely that this version table will have thousands of rows over next 6 months, and maybe 5 to 10 times more over 1 year.

How many reads and writes per hour would you expect this table to have in 3 months, 6 months, 1 year? Under what circumstances are rows updated? What assumptions are these based on?

For writes, I think there is no more than 100 per hour over 1 year, as this agent_config is more like a one-off configuration, and less likely to be updated very soon. For reads, assume we will have 1000 agent_configs over 1 year, and based on the current reconciliation pattern, we trigger once per second, it means around 4 million reads per hour (1000 * 3600).

Based on the anticipated data volume and access patterns, does the new table pose an availability risk to GitLab.com or self-managed instances? Does the proposed design scale to support the needs of GitLab.com and self-managed customers?

I do not think so, the access pattern of the version table is straight-forward. We insert new rows as needed for writes. For reads it will fetch a list of versions for a given agent_config via item_type and item_id association.

Edited by zli

Merge request reports