Move prompt cache setting to Application Settings JSON column

Shola Quadrirequested to merge
538970/sq/move-prompt-caching-setting into master

What does this MR do and why?

Addresses #538970 (closed)

Follow-up:

@missy-gitlab - Thanks for raising this, would have been great to have this and other related attrs in a single jsonb within application_settings, for all the reason mentioned in the doc. Loading the column cache for application_settings is one of the most expensive queries because of how wide it became and it's called more frequently 😟

This MR migrates the cascading setting model_prompt_cache_enabled from a dedicated database column to the existing code_creation application setting JSON column.

Why

The application_settings table has become increasingly wide, making column cache loading one of the most expensive database operations in the application. By consolidating related settings into JSONB columns, we:

  • Prevent table bloat - avoid adding more individual columns
  • Improve database performance - reduce the cost of loading application settings
  • Group related settings - keep code creation settings together logically

Application Settings Development Docs [link]

Changes made:

Database Schema

  • Migrated existing data from model_prompt_cache_enabled column to code_creation JSON column
  • Added ignore column for the old model_prompt_cache_enabled column for future removal as per doc
  • Preserved all existing values during migration

Application Code

  • Updated JSONB accessor to include model_prompt_cache_enabled with default value: true
  • Added JSON schema validation
  • Maintained cascading behavior, i.e. settings still cascade from Project → Namespace → ApplicationSetting

Steps to Verify Locally (model_prompt_cache_enabled)

1. Verify when no specific project is provided, we use application setting's JSON default value (true):

Request:

curl --request POST \
  --url https://gdk.test:3443/api/v4/code_suggestions/direct_access \
  --header "Authorization: Bearer <YOUR_PAT>" \
  --header "Content-Type: application/json" \
  -d '{}'

Response (X-Gitlab-Model-Prompt-Cache-Enabled should return true):

{"base_url":"http://0.0.0.0:5052",
"token":"ew",
"expires_at":1753197699,
"headers":
{
"x-gitlab-feature-enablement-type":"duo_pro",
"x-gitlab-enabled-feature-flags":"",
"x-gitlab-enabled-instance-verbose-ai-logs":"",
"x-gitlab-host-name":"gdk.test",
"x-gitlab-instance-id":"263f654b-9705-465b-8355-0cef3a419b26",
"x-gitlab-realm":"saas",
"x-gitlab-version":"18.3.0",
"x-gitlab-global-user-id":"LNBtI/YB/Uwii0bQiOdtFsewYbcPow1D6soC268GHI4=",
"x-gitlab-feature-enabled-by-namespace-ids":"1000000",
"X-Gitlab-Saas-Namespace-Ids":"",
"X-Gitlab-Saas-Duo-Pro-Namespace-Ids":"1000000",
"X-Gitlab-Model-Prompt-Cache-Enabled":"true",
"X-Gitlab-Authentication-Type":"oidc"},
"model_details":
{
"model_provider":"vertex-ai",
"model_name":"codestral-2501"
}

2. Verify project-level settings take precedence by setting model_prompt_cache_enabled to False:

In Rails Console

[33] pry(main)> p = Project.where(name: "Gitlab Shell")
[33] pry(main)> p = p.first
[33] pry(main)> ps = p.project_setting
[33] pry(main)> ps.model_prompt_cache_enabled = false
[33] pry(main)> ps.save

Request:

curl --request POST \
  --url https://gdk.test:3443/api/v4/code_suggestions/direct_access \
  --header "Authorization: Bearer <YOUR_PAT>" \
  --header "Content-Type: application/json" \
  -d '{"project_path":"gitlab-org/gitlab-shell"}'

Response (X-Gitlab-Model-Prompt-Cache-Enabled should return false):

{"base_url":"http://0.0.0.0:5052",
"token":"ew",
"expires_at":1753197699,
"headers":
{
"x-gitlab-feature-enablement-type":"duo_pro",
"x-gitlab-enabled-feature-flags":"",
"x-gitlab-enabled-instance-verbose-ai-logs":"",
"x-gitlab-host-name":"gdk.test",
"x-gitlab-instance-id":"263f654b-9705-465b-8355-0cef3a419b26",
"x-gitlab-realm":"saas",
"x-gitlab-version":"18.3.0",
"x-gitlab-global-user-id":"LNBtI/YB/Uwii0bQiOdtFsewYbcPow1D6soC268GHI4=",
"x-gitlab-feature-enabled-by-namespace-ids":"1000000",
"X-Gitlab-Saas-Namespace-Ids":"",
"X-Gitlab-Saas-Duo-Pro-Namespace-Ids":"1000000",
"X-Gitlab-Model-Prompt-Cache-Enabled":"false",
"X-Gitlab-Authentication-Type":"oidc"},
"model_details":
{
"model_provider":"vertex-ai",
"model_name":"codestral-2501"
}

3. Verify when project settings is nil, it cascades to namespace setting:

Rails Console:

[33] pry(main)> ps.model_prompt_cache_enabled = nil
[33] pry(main)> ps.save
[33] pry(main)> g = p.parent
[33] pry(main)> g.parent ==========================================> returns nil
[33] pry(main)> gn = g.namespace_settings
[33] pry(main)> gn.model_prompt_cache_enabled = false
[33] pry(main)> gn.save

Request:

curl --request POST \
  --url https://gdk.test:3443/api/v4/code_suggestions/direct_access \
  --header "Authorization: Bearer <YOUR_PAT>" \
  --header "Content-Type: application/json" \
  -d '{"project_path":"gitlab-org/gitlab-shell"}'

Response (X-Gitlab-Model-Prompt-Cache-Enabled should return false)::

{"base_url":"http://0.0.0.0:5052",
"token":"ew",
"expires_at":1753197699,
"headers":
{
"x-gitlab-feature-enablement-type":"duo_pro",
"x-gitlab-enabled-feature-flags":"",
"x-gitlab-enabled-instance-verbose-ai-logs":"",
"x-gitlab-host-name":"gdk.test",
"x-gitlab-instance-id":"263f654b-9705-465b-8355-0cef3a419b26",
"x-gitlab-realm":"saas",
"x-gitlab-version":"18.3.0",
"x-gitlab-global-user-id":"LNBtI/YB/Uwii0bQiOdtFsewYbcPow1D6soC268GHI4=",
"x-gitlab-feature-enabled-by-namespace-ids":"1000000",
"X-Gitlab-Saas-Namespace-Ids":"",
"X-Gitlab-Saas-Duo-Pro-Namespace-Ids":"1000000",
"X-Gitlab-Model-Prompt-Cache-Enabled":"false",
"X-Gitlab-Authentication-Type":"oidc"},
"model_details":
{
"model_provider":"vertex-ai",
"model_name":"codestral-2501"
}

4. Verify when both project and namespace settings are nil, it cascades to application setting's default value:

Rails Console

[33] pry(main)> gn.model_prompt_cache_enabled = nil
[33] pry(main)> gn.save
[42] pry(main)> ApplicationSetting.first.model_prompt_cache_enabled
=> true

Request:

curl --request POST \
  --url https://gdk.test:3443/api/v4/code_suggestions/direct_access \
  --header "Authorization: Bearer <YOUR_PAT>" \
  --header "Content-Type: application/json" \
  -d '{"project_path":"gitlab-org/gitlab-shell"}'

Response (X-Gitlab-Model-Prompt-Cache-Enabled should return true):

{"base_url":"http://0.0.0.0:5052",
"token":"ew",
"expires_at":1753197699,
"headers":
{
"x-gitlab-feature-enablement-type":"duo_pro",
"x-gitlab-enabled-feature-flags":"",
"x-gitlab-enabled-instance-verbose-ai-logs":"",
"x-gitlab-host-name":"gdk.test",
"x-gitlab-instance-id":"263f654b-9705-465b-8355-0cef3a419b26",
"x-gitlab-realm":"saas",
"x-gitlab-version":"18.3.0",
"x-gitlab-global-user-id":"LNBtI/YB/Uwii0bQiOdtFsewYbcPow1D6soC268GHI4=",
"x-gitlab-feature-enabled-by-namespace-ids":"1000000",
"X-Gitlab-Saas-Namespace-Ids":"",
"X-Gitlab-Saas-Duo-Pro-Namespace-Ids":"1000000",
"X-Gitlab-Model-Prompt-Cache-Enabled":"true",
"X-Gitlab-Authentication-Type":"oidc"},
"model_details":
{
"model_provider":"vertex-ai",
"model_name":"codestral-2501"
}

5. Verify locking behaviour with lock_model_prompt_cache_enabled:

Set up application level lock

app_setting = ApplicationSetting.current
app_setting.update!(model_prompt_cache_enabled: false)
app_setting.update!(lock_model_prompt_cache_enabled: true)

Test project setting cannot change locked value

p = Project.where(name: "Gitlab Shell").first
ps = p.project_setting

# Try to change the value (should get overridden)
ps.model_prompt_cache_enabled = true
ps.model_prompt_cache_enabled (should return false, i.e. locked value is enforced)

ps.model_prompt_cache_enabled_changed?
=> false

ps.save
ps.reload.model_prompt_cache_enabled
=> false (the locked value should still be in use)

Test namespace setting behaviour

g = p.parent
ns = g.namespace_settings

ns.model_prompt_cache_enabled = true
ns.model_prompt_cache_enabled (should return false, i.e. locked value is enforced)

ns.save (the locked value should still be in use)

Verify lock detection methods work

ps.model_prompt_cache_enabled_locked?
=> true

ps.model_prompt_cache_enabled_locked_by_application_setting?
# => true

Verify via API

Request:

curl --request POST \
  --url https://gdk.test:3443/api/v4/code_suggestions/direct_access \
  --header "Authorization: Bearer <YOUR_PAT>" \
  --header "Content-Type: application/json" \
  -d '{"project_path":"gitlab-org/gitlab-shell"}'

Response:

  1. When locked: X-Gitlab-Model-Prompt-Cache-Enabled should return false

Verify unlocking allows changes

app_setting.update!(lock_model_prompt_cache_enabled: false)

ps.reload
ps.model_prompt_cache_enabled = true
ps.model_prompt_cache_enabled
=> true (change should now be allowed)

ps.save
ps.reload.model_prompt_cache_enabled
=> true (the change should persist)

Verify via API

Request:

curl --request POST \
  --url https://gdk.test:3443/api/v4/code_suggestions/direct_access \
  --header "Authorization: Bearer <YOUR_PAT>" \
  --header "Content-Type: application/json" \
  -d '{"project_path":"gitlab-org/gitlab-shell"}'

Response:

  1. When unlocked: X-Gitlab-Model-Prompt-Cache-Enabled should return true

References

  1. !188732 (comment 2470115077)

Screenshots or screen recordings

Before After

How to set up and validate locally

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Shola Quadri

Merge request reports