Monday 2024-04-08 04:59 UTC - `gitlab-org/gitlab` broken `master` with update-tests-metadata

added Engineering Productivity master-brokenundetermined master:broken labels

Root Cause Analysis

update-tests-metadata: master-brokenundetermined.

rspec-profiling v0.0.8 added the seed column via https://github.com/procore-oss/rspec_profiling/pull/1, which does not exist in the current database.

In addition to the seed column, it looks like start_memory and end_memory columns were also added. These appear to be floating point numbers:

require 'get_process_mem'
GetProcessMem.new.mb
=> 29.71875

The database is in the gitlab-eng-productivity project.

I logged into the database with credentials from the Production vault and ran:

ALTER TABLE spec_profiling_results
ADD COLUMN seed INTEGER,
ADD COLUMN start_memory DOUBLE PRECISION,
ADD COLUMN end_memory DOUBLE PRECISION;

The schema is now:


                                                                 Table "public.spec_profiling_results"
      Column      |              Type              | Collation | Nullable |                      Default                       | Storage  | Stats target | Description 
------------------+--------------------------------+-----------+----------+----------------------------------------------------+----------+--------------+-------------
 id               | bigint                         |           | not null | nextval('spec_profiling_results_id_seq'::regclass) | plain    |              | 
 branch           | character varying              |           |          |                                                    | extended |              | 
 commit_hash      | character varying              |           |          |                                                    | extended |              | 
 date             | timestamp without time zone    |           |          |                                                    | plain    |              | 
 file             | text                           |           |          |                                                    | extended |              | 
 line_number      | integer                        |           |          |                                                    | plain    |              | 
 description      | text                           |           |          |                                                    | extended |              | 
 time             | numeric                        |           |          |                                                    | main     |              | 
 status           | character varying              |           |          |                                                    | extended |              | 
 exception        | text                           |           |          |                                                    | extended |              | 
 query_count      | integer                        |           |          |                                                    | plain    |              | 
 query_time       | numeric                        |           |          |                                                    | main     |              | 
 request_count    | integer                        |           |          |                                                    | plain    |              | 
 request_time     | numeric                        |           |          |                                                    | main     |              | 
 created_at       | timestamp(6) without time zone |           |          |                                                    | plain    |              | 
 updated_at       | timestamp(6) without time zone |           |          |                                                    | plain    |              | 
 feature_category | text                           |           |          |                                                    | extended |              | 
 seed             | integer                        |           |          |                                                    | plain    |              | 
 start_memory     | double precision               |           |          |                                                    | plain    |              | 
 end_memory       | double precision               |           |          |                                                    | plain    |              | 
Indexes:
    "spec_profiling_results_pkey" PRIMARY KEY, btree (id)
    "index_spec_profiling_results_branch" btree (branch)
    "index_spec_profiling_results_commit" btree (commit_hash)
    "index_spec_profiling_results_date" btree (date)
    "index_spec_profiling_results_file" btree (file)
    "index_spec_profiling_results_file_line_number" btree (file, line_number)
    "index_spec_profiling_results_query_count" btree (query_count)
    "index_spec_profiling_results_query_time" btree (query_time)
    "index_spec_profiling_results_request_count" btree (request_count)
    "index_spec_profiling_results_request_time" btree (request_time)
    "index_spec_profiling_results_time" btree ("time")
Access method: heap

Thanks @stanhu!

Sure! I wish we had a better way of managing migrations for this gem, but it shouldn't get updated that frequently.

https://gitlab-org.gitlab.io/rspec_profiling_stats/ seems to have been updated recently.

I guess we'll have to wait until https://gitlab.com/gitlab-org/gitlab/-/jobs/6575488519 completes to see for sure.

@stanhu It's not very obvious that the CSV collector is related to our database structure

See #5751 (comment 1851637550)

Sure! I wish we had a better way of managing migrations for this gem, but it shouldn't get updated that frequently.

We've seen CSV (or SQL on our side) related changes in 0.0.8 and 0.0.9. It seems the author picked up the update pace again

Perhaps we could tweak the CSV to PSQL ingestion to have more control over which CSV columns are dumped into database columns.

It's not very obvious that the CSV collector is related to our database structure

Yeah, that's because we patch things here: https://gitlab.com/gitlab-org/gitlab/-/blob/32eb2931f6ea457460d1005cd3b4e91b88432ecd/config/initializers/rspec_profiling.rb#L12. Perhaps we should upstream some of those changes, as well as https://gitlab.com/gitlab-org/gitlab/-/blob/master/scripts/insert-rspec-profiling-data.

We've seen CSV (or SQL on our side) related changes in 0.0.8 and 0.0.9. It seems the author picked up the update pace again

Perhaps we could tweak the CSV to PSQL ingestion to have more control over which CSV columns are dumped into database columns.

Yes, we could add a test or put a fixed value in that patch above.