Database goals for April 16th, 2018

@abrandl:

• PGConf.de (one day German PostgreSQL conference in Berlin right next door)
• Reduce impact of malformed project lookups (/api/v4/projects/:id), add validation for format of id in https://gitlab.com/gitlab-org/gitlab-ce/issues/45247
• Atomic ID generation for other models in https://gitlab.com/gitlab-org/gitlab-ce/issues/44259 Moved to next week due to short cycle

@_stark:

• Arrange/plan upgrade to 9.6.8 in gitlab.com #15 (closed)
  • Plan and document upgrade process and migrate knowledge from jtevnan to DB team
    • Continue testing in staging
    • Fix problem with chef running reconfigure unexpectedly
    • Fix problems caused by hard-coded IP addresses in chef
    • Fix problem causing failure to do automatic failover after a switchover
    • Finish whole procedure in staging
  • Have a specific scheduled time for production upgrade -- possibly complete it this week?
    • • Announced Monday on Infrastructure call
    • • May push to Tuesday or Wednesday based on testing results on Monday
• Investigate Production performance problems on Friday
  • Lots of problems uncovered and issues made but no database smoking gun

On the back burner for after the 9.6.8 update (some added during the week)

• Polish up gitlab-org/omnibus-gitlab!2332 (closed)
  • Resolve review comments from richardc
• Polish up https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/17196
  • Resolve review comments from @DylanGriffith
• Change Postgres prefix to "9.6" - gitlab-org/omnibus-gitlab#3346 (closed)
  • Start work on MR
• Investigate high transaction rollback rate in GPRD
  • migration#292 (closed)
• https://gitlab.com/gitlab-com/infrastructure/issues/4020
  • Fix broken Grafana graphs broken by changed node metrics
  • Check for other missing metrics?
  • Investigate adding a Postgres backend to grafana to make this easier to do in future
• Lower sensitivity for alerts with false positives: #67 (closed)
  • Turns out we already lowered sensitivity for the Disk I/O alert but it was ineffective due to the same node metric renames as the grafana dashboard
  • runbooks!551 (merged)
• Add monitoring for slow lock logs (including trigram index bug): #68 (closed)
  • https://dev.gitlab.org/cookbooks/chef-repo/merge_requests/1853
  • gitlab-cookbooks/gitlab-exporters!10 (merged)

@yorickpeterse:

• Merge https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/5312
• Adjust https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/18157 based on feedback
• Start porting a few dashboards over to Prometheus: https://gitlab.com/gitlab-com/infrastructure/issues/1962
  • Blocked until we resolve https://gitlab.com/gitlab-com/infrastructure/issues/1962#note_68292200 somehow.
• #54 (moved)

Last week's goals: #2 (closed)

Next week's goals: #69 (closed)