500 error in staging for vscode/settings_sync for gitlab-qa user who has a large amount of recently opened projects

We are noticing 500 errors when syncing web ide settings in staging.gitlab.com for the gitlab-qa user.

This is caused by the gitlab-qa user having many (~5000) recently opened projects stored in the vs_code_settings stable, see: #479912 (comment 2060722985)

Investigation

500 error surfaces because of a constraint around the size of the content: https://gitlab.com/gitlab-org/gitlab/-/blob/d3e724bdb0bd0efd03c68fb99e19baecbc53b62b/db/structure.sql#L19408, with error:

PG::CheckViolation: ERROR:  new row for relation "vs_code_settings" violates check constraint "check_5da3b2910b"
DETAIL:  Failing row contains (1, 1614863, 2024-03-04 10:30:01.887153+00, 2024-08-19 23:46:40.112622+00, globalState, {"storage":{"workbench.panel.markers.hidden":{"version":1,"value..., 1015c269-350f-4268-8ae7-dddd65d5a58a, 1).

/*application:web,correlation_id:fcb8d3a3869a5129d65c8c2bfa33b04f,endpoint_id:POST /api/:version/vscode/settings_sync/v1/resource/:resource_name,db_config_name:main*/ UPDATE "vs_code_settings" SET "updated_at" = $1, "content" = $2, "uuid" = $3 WHERE "vs_code_settings"."id" = $4

The above message is truncated. To figure out why this constraint is being broken we must see what is in this table for the gitlab-qa user.

Note: this is not occurring in production currently: https://log.gprd.gitlab.net/app/r/s/N24cm