Generate sitemap.xml in staging and production

Production Change

Change Summary

Our SEO has decreased a lot in the last months. One of the ideas on the table to improve this is to generate a sitemap with all public groups, subgroups, and projects. Nevertheless, that task is quite complex and first we need to validate this assumption.

That's why we implemented gitlab-org/gitlab#270051 (closed). That MR is a POC valid only on .com (staging and prod) that generates a sitemap for the gitlab-org subgroups and projects. This way, we can send it to the crawlers and analyze if the resources listed there get a better SEO.

Since it's a POC and for the gitlab-org the idea is not to make it complex, that's why we didn't implement a background job or cron job to generate it.

We need to generate the sitemap first for staging, validate its format using the online tools https://www.websiteplanet.com/webtools/sitemap-validator/ or https://www.xml-sitemaps.com/validate-xml-sitemap.html and, if everything is ok, remove the generated file (we don't need it in staging).

After testing in staging we can check with prod. We have to do the same steps, generate the sitemap, validate it against the same tools, and leave it there.

Change Details

  1. Services Impacted - Gitlab(Rails) and Postgres
  2. Change Technician - @fjsanpedro
  3. Change Criticality - C4
  4. Change Type - changeunscheduled
  5. Change Reviewer - @craigf
  6. Due Date - November 2
  7. Time tracking - ~ 10 minutes
  8. Downtime Component - None

Detailed steps for the change

Change Steps - steps to take to execute the change

Estimated Time to Complete 10mins

My user in prod and staging does not have enough rights to create a file in the /public dir, so I'll ned support with this.

Post-Change Steps - steps to take to verify the change

*Estimated Time to Complete 1 min

Rollback

Rollback steps - steps to be taken in the event of a need to rollback this change

Estimated Time to Complete 1 min

  • Staging: delete the generated sitemap.xml file in /public dir.
  • Production: delete the generated sitemap.xml file in /public dir

Monitoring

Key metrics to observe

The metrics to observe are in the Google Search Console and they're not immediate.

Summary of infrastructure changes

  • Does this change introduce new compute instances?
  • Does this change re-size any existing compute instances?
  • Does this change introduce any additional usage of tooling like Elastic Search, CDNs, Cloudflare, etc?

Summary of the above

Changes checklist

  • This issue has a criticality label (e.g. C1, C2, C3, C4) and a change-type label (e.g. changeunscheduled, changescheduled) based on the Change Management Criticalities.
  • This issue has the change technician as the assignee.
  • Pre-Change, Change, Post-Change, and Rollback steps and have been filled out and reviewed.
  • Necessary approvals have been completed based on the Change Management Workflow.
  • Change has been tested in staging and resultes noted in a comment on this issue.
  • A dry-run has been conducted and results noted in a comment on this issue.
  • SRE on-call has been informed prior to change being rolled out. (In #production channel, mention @sre-oncall and this issue.)
  • There are currently no active incidents.
Edited by Francisco Javier López