Missing Service Pings for version <= 12.0 starting 16th of February

Problem

Starting from the 15th of February we are receiving less Service Pings than normal in Snowflake. Investigation in https://gitlab.com/gitlab-data/analytics/-/issues/19854#note_1792137458 has shown two things:

  • likely duplicates responsible for the drop on weekdays, since the drop is not present in version app
  • Drop on weekends is actually missing data from versions <= 12.0

Detection

Impact

  • Missing Service pings for Version < 12.0 starting from the weekend of the 16th of February

Additional information

Checklist

  • Assigned severity tags based on this guidance
  • Assigned to PM and EM of groupanalytics instrumentation
  • Posted link to incident in g_analyze_analytics_instrumentation and tagged both PM and EM of the group

<---- TO BE FILLED BY ASSIGNEE / RESOLUTION DRI---->

Summary

Root Cause

Most likely cause is the combination of:

  1. The explicit preferredChain setting and bug in cert-manager caused issuing a certificate with an expired DST Root CA X3 in the chain.
  2. 8th of Feb Let's Encrypt changed Chain of Trust to terminate at a newish ISRG Root X1. More information on the official site
  3. Verification failures for old CA bundle in Gitlab versions <= 12.0 since those use OpenSSL 1.0.2r, older instances wen't able to establish SSL connection with Version app.

Resolution

The issue was resolved by

  1. Removing the explicit preferredChain value https://gitlab.com/gitlab-org/gitlab-services/cluster-management/-/merge_requests/12/diffs#6afc159cc7e8375f4331123d3e50d006e93f4f9b_11_11 (and in production) and upgrading cert-manager.
  2. Deleting the old certificate and re-issuing a new one, without DST Root CA X3:
kubectl cert-manager -n version-prod renew production-auto-deploy-tls
  1. The fix was verified first by sending a request from Gitlab instance v11:
```ruby
> Gitlab::HTTP.post('https://version.gitlab.com/usage_data', body: {foo: "bar"}.to_json, allow_local_requests: true, headers: { 'Content-type' => 'application/json' })
=> #<HTTParty::Response ...<Net::HTTPCreated 201

Then by verifying the new Service Ping payloads in the DW.

Edited by Niko Belokolodov