Skip to content

Remove plaintext token index from ci_builds

What does this MR do and why?

Related to #339191 (closed)

This was dropped in production via change request gitlab-com/gl-infra/production#7345 (comment 1030499429) and this MR should be a no-op on .com.

The ci_builds token column is no longer used and we're in the process of removing it #339192 (closed), but it has an index attached to it. Since this is a high traffic table, it might be a good idea to drop the index in a separate migration.

List of relations  
 Schema |               Name               | Type  |  Owner  |   Table   |  Size   | Description   
--------+----------------------------------+-------+---------+-----------+---------+-------------  
 public | index_ci_builds_on_token_partial | index | joe_309 | ci_builds | 4421 MB |   
(1 row) 

Screenshots or screen recordings

image

https://thanos.gitlab.net/graph?g0.expr=sum(rate(pg_stat_user_indexes_idx_tup_read%7Benv%3D%22gprd%22%2C%20indexrelname%3D%22index_ci_builds_on_token_partial%22%7D%5B5m%5D))&g0.tab=0&g0.stacked=0&g0.range_input=2w&g0.max_source_resolution=0s&g0.deduplicate=1&g0.partial_response=0&g0.store_matches=%5B%5D&g1.expr=sum(rate(pg_stat_user_indexes_idx_tup_fetch%7Benv%3D%22gprd%22%2C%20indexrelname%3D%22index_ci_builds_on_token_partial%22%7D%5B5m%5D))&g1.tab=0&g1.stacked=0&g1.range_input=2w&g1.max_source_resolution=0s&g1.deduplicate=1&g1.partial_response=0&g1.store_matches=%5B%5D&g2.expr=sum(rate(pg_stat_user_indexes_idx_scan%7Benv%3D%22gprd%22%2C%20indexrelname%3D%22index_ci_builds_on_token_partial%22%7D%5B5m%5D))&g2.tab=0&g2.stacked=0&g2.range_input=2w&g2.max_source_resolution=0s&g2.deduplicate=1&g2.partial_response=0&g2.store_matches=%5B%5D

Migrations output:

marius@roast-carrot ~/W/g/gitlab (339191-remove-plaintext-token-index-from-ci-builds)> VERSION=20220628111752 bin/rails db:migrate:up:ci
ci: == 20220628111752 DropTokenIndexFromCiBuilds: migrating =======================
ci: -- transaction_open?()
ci:    -> 0.0001s
ci: -- indexes(:ci_builds)
ci:    -> 0.0274s
ci: -- execute("SET statement_timeout TO 0")
ci:    -> 0.0003s
ci: -- remove_index(:ci_builds, {:algorithm=>:concurrently, :name=>"index_ci_builds_on_token_partial"})
ci:    -> 0.0091s
ci: -- execute("RESET statement_timeout")
ci:    -> 0.0004s
ci: == 20220628111752 DropTokenIndexFromCiBuilds: migrated (0.0519s) ==============

marius@roast-carrot ~/W/g/gitlab (339191-remove-plaintext-token-index-from-ci-builds)> VERSION=20220628111752 bin/rails db:migrate:up:main
main: == 20220628111752 DropTokenIndexFromCiBuilds: migrating =======================
main: -- transaction_open?()
main:    -> 0.0000s
main: -- indexes(:ci_builds)
main:    -> 0.0200s
main: -- execute("SET statement_timeout TO 0")
main:    -> 0.0003s
main: -- remove_index(:ci_builds, {:algorithm=>:concurrently, :name=>"index_ci_builds_on_token_partial"})
main:    -> 0.0061s
main: -- execute("RESET statement_timeout")
main:    -> 0.0004s
main: == 20220628111752 DropTokenIndexFromCiBuilds: migrated (0.0359s) ==============

marius@roast-carrot ~/W/g/gitlab (339191-remove-plaintext-token-index-from-ci-builds)> VERSION=20220628111752 bin/rails db:migrate:down:ci
ci: == 20220628111752 DropTokenIndexFromCiBuilds: reverting =======================
ci: -- transaction_open?()
ci:    -> 0.0000s
ci: -- index_exists?(:ci_builds, :token, {:unique=>true, :where=>"token IS NOT NULL", :name=>"index_ci_builds_on_token_partial", :algorithm=>:concurrently})
ci:    -> 0.0265s
ci: -- execute("SET statement_timeout TO 0")
ci:    -> 0.0004s
ci: -- add_index(:ci_builds, :token, {:unique=>true, :where=>"token IS NOT NULL", :name=>"index_ci_builds_on_token_partial", :algorithm=>:concurrently})
ci:    -> 0.0164s
ci: -- execute("RESET statement_timeout")
ci:    -> 0.0004s
ci: == 20220628111752 DropTokenIndexFromCiBuilds: reverted (0.0529s) ==============

marius@roast-carrot ~/W/g/gitlab (339191-remove-plaintext-token-index-from-ci-builds)> VERSION=20220628111752 bin/rails db:migrate:down:main
main: == 20220628111752 DropTokenIndexFromCiBuilds: reverting =======================
main: -- transaction_open?()
main:    -> 0.0000s
main: -- index_exists?(:ci_builds, :token, {:unique=>true, :where=>"token IS NOT NULL", :name=>"index_ci_builds_on_token_partial", :algorithm=>:concurrently})
main:    -> 0.0193s
main: -- execute("SET statement_timeout TO 0")
main:    -> 0.0003s
main: -- add_index(:ci_builds, :token, {:unique=>true, :where=>"token IS NOT NULL", :name=>"index_ci_builds_on_token_partial", :algorithm=>:concurrently})
main:    -> 0.0133s
main: -- execute("RESET statement_timeout")
main:    -> 0.0004s
main: == 20220628111752 DropTokenIndexFromCiBuilds: reverted (0.0416s) ==============

Impact

Reduced ci_builds size by 4.4GB

How to set up and validate locally

Numbered steps to set up and validate the change are strongly suggested.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Marius Bobin

Merge request reports