Some vulnerability reports last updated value not being updated after scans

Summary

While working on a ticket with a customer, we are encountering a scenario where the last updated value of the vulnerability report is still pointing to an older pipeline yet there are more newer pipelines that have been triggered and ran successfully on the default branch master.

Filtering for low severity vulnerabilities as of creating this issue, I can see there are results that were introduced on Sept 4th 2022, but the last updated value is still stale.

I also noticed the Dependency List last updated value is being updated correctly. Same to the license compliance report.

Slack conversation: https://gitlab.slack.com/archives/CV09DAXEW/p1661877413504849

Zendesk Ticket: #319132 - internal only, Ultimate customer

Steps to reproduce

Navigate to https://gitlab.com/cynergyfusion/fintech-hub/fth-drupal-app/-/security/vulnerability_report/ and check for the last updated value on the page.
https://gitlab.com/syniverse1/signaling-engineering/imx/-/security/vulnerability_report is facing this issue as well

Example Project

What is the current bug behavior?

The last updated value of the vulnerability report has not been updated. Currently shows 4 weeks ago.

What is the expected correct behavior?

The last updated value should be updated even if no new vulnerabilities have been introduced in the repository.

Relevant logs and/or screenshots

Kibana Logs:

https://log.gprd.gitlab.net/goto/081eece0-3353-11ed-b86b-d963a1a6788e

Backtrace

[
        "lib/gitlab/database/load_balancing/connection_proxy.rb:100:in `block in read_using_load_balancer'",
        "lib/gitlab/database/load_balancing/load_balancer.rb:115:in `block in read_write'",
        "lib/gitlab/database/load_balancing/load_balancer.rb:191:in `retry_with_backoff'",
        "lib/gitlab/database/load_balancing/load_balancer.rb:111:in `read_write'",
        "lib/gitlab/database/load_balancing/connection_proxy.rb:99:in `read_using_load_balancer'",
        "lib/gitlab/database/load_balancing/connection_proxy.rb:48:in `select_all'",
        "app/models/concerns/each_batch.rb:62:in `each_batch'",
        "ee/app/services/security/ingestion/mark_as_resolved_service.rb:19:in `execute'",
        "ee/app/services/security/ingestion/mark_as_resolved_service.rb:10:in `execute'",
        "ee/app/services/security/ingestion/ingest_reports_service.rb:43:in `mark_resolved_vulnerabilities'",
        "ee/app/services/security/ingestion/ingest_reports_service.rb:31:in `then'",
        "ee/app/services/security/ingestion/ingest_reports_service.rb:31:in `store_reports'",
        "ee/app/services/security/ingestion/ingest_reports_service.rb:17:in `execute'",
        "ee/app/services/security/ingestion/ingest_reports_service.rb:9:in `execute'",
        "ee/app/workers/store_security_reports_worker.rb:21:in `block in perform'",
        "ee/app/workers/store_security_reports_worker.rb:18:in `perform'",
        "lib/gitlab/database/load_balancing/sidekiq_server_middleware.rb:26:in `call'",
        "lib/gitlab/sidekiq_middleware/duplicate_jobs/strategies/until_executing.rb:16:in `perform'",
        "lib/gitlab/sidekiq_middleware/duplicate_jobs/duplicate_job.rb:58:in `perform'",
        "lib/gitlab/sidekiq_middleware/duplicate_jobs/server.rb:8:in `call'",
        "lib/gitlab/sidekiq_middleware/worker_context.rb:9:in `wrap_in_optional_context'",
        "lib/gitlab/sidekiq_middleware/worker_context/server.rb:19:in `block in call'",
        "lib/gitlab/application_context.rb:110:in `block in use'",
        "lib/gitlab/application_context.rb:110:in `use'",
        "lib/gitlab/application_context.rb:52:in `with_context'",
        "lib/gitlab/sidekiq_middleware/worker_context/server.rb:17:in `call'",
        "lib/gitlab/sidekiq_status/server_middleware.rb:7:in `call'",
        "lib/gitlab/sidekiq_versioning/middleware.rb:9:in `call'",
        "lib/gitlab/sidekiq_middleware/query_analyzer.rb:7:in `block in call'",
        "lib/gitlab/database/query_analyzer.rb:37:in `within'",
        "lib/gitlab/sidekiq_middleware/query_analyzer.rb:7:in `call'",
        "lib/gitlab/sidekiq_middleware/admin_mode/server.rb:14:in `call'",
        "lib/gitlab/sidekiq_middleware/instrumentation_logger.rb:9:in `call'",
        "lib/gitlab/sidekiq_middleware/batch_loader.rb:7:in `call'",
        "lib/gitlab/sidekiq_middleware/extra_done_log_metadata.rb:7:in `call'",
        "lib/gitlab/sidekiq_middleware/request_store_middleware.rb:10:in `block in call'",
        "lib/gitlab/with_request_store.rb:17:in `enabling_request_store'",
        "lib/gitlab/with_request_store.rb:10:in `with_request_store'",
        "lib/gitlab/sidekiq_middleware/request_store_middleware.rb:9:in `call'",
        "lib/gitlab/sidekiq_middleware/server_metrics.rb:76:in `block in call'",
        "lib/gitlab/sidekiq_middleware/server_metrics.rb:103:in `block in instrument'",
        "lib/gitlab/metrics/background_transaction.rb:33:in `run'",
        "lib/gitlab/sidekiq_middleware/server_metrics.rb:103:in `instrument'",
        "lib/gitlab/sidekiq_middleware/server_metrics.rb:75:in `call'",
        "lib/gitlab/sidekiq_middleware/monitor.rb:10:in `block in call'",
        "lib/gitlab/sidekiq_daemon/monitor.rb:49:in `within_job'",
        "lib/gitlab/sidekiq_middleware/monitor.rb:9:in `call'",
        "lib/gitlab/sidekiq_middleware/size_limiter/server.rb:13:in `call'",
        "lib/gitlab/sidekiq_logging/structured_logger.rb:21:in `call'"
],

Exception message

PG::QueryCanceled: ERROR:  canceling statement due to statement timeout

exception.sql


"/*application:sidekiq,correlation_id:3cbdc1f0af9abf4804dbaa422a98456e,jid:0c3834f72d98af4c29c5506c,endpoint_id:StoreSecurityReportsWorker,db_config_name:main*/ SELECT \"vulnerabilities\".\"id\" FROM \"vulnerabilities\" WHERE \"vulnerabilities\".\"project_id\" = $1 ORDER BY \"vulnerabilities\".\"id\" ASC LIMIT $2",

Vulnerability report screenshot:

Dependency List

License Compliance

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info

(For installations with omnibus-gitlab package run and paste the output of:
`sudo gitlab-rake gitlab:env:info`)

(For installations from source run and paste the output of:
`sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production`)

Results of GitLab application Check

Expand for output related to the GitLab application check

(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true)
(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true)
(we will only investigate if the tests are passing)

Possible fixes

Edited Sep 13, 2022 by Christopher Mutua