postgres-dr-archive-01-db-gprd.c.gitlab-production.internal postgres service appears down

Current Status

Received an alert due to PostgreSQL service being down on postgres-dr-archive-01-db-gprd.c.gitlab-production.internal.

After login in, we noticed that the instance had been rebooted, and only had an uptime of 9min:

mchacon@postgres-dr-archive-01-db-gprd.c.gitlab-production.internal:~$ uptime
 21:03:32 up 9 min,  1 user,  load average: 0.00, 0.32, 0.36

We noticed that PostgreSQL wasn't running on the system:

mchacon@postgres-dr-archive-01-db-gprd.c.gitlab-production.internal:~$ sudo ps aux |grep postgres
root      1866  0.0  0.0   4240   744 ?        Ss   20:53   0:00 runsv postgres_exporter
root      2125  0.0  0.0   4384   684 ?        S    20:53   0:00 svlogd -tt /var/log/prometheus/postgres_exporter
root      2138  0.7  0.0 1404788 30920 ?       Sl   20:53   0:03 /opt/prometheus/mtail/mtail -progs /opt/prometheus/mtail/progs -logs /var/log/apt/term.log,/var/log/syslog,/var/log/td-agent/td-agent.log,/var/log/gitlab/postgresql/postgresql.csv,/var/log/wal-g/wal-g.log,/var/log/wal-g/wal-g_backup_push.log,/var/log/gitlab/pgbouncer/pgbouncer.log -logtostderr
prometh+  2154  0.0  0.0 113292 12128 ?        Sl   20:53   0:00 /opt/prometheus/postgres_exporter/postgres_exporter --extend.query-path=/opt/prometheus/postgres_exporter/queries.yaml
root      3112  0.0  0.0   4240   732 ?        Ss   20:54   0:00 runsv postgresql
root      3116  0.0  0.0   4384   660 ?        S    20:54   0:00 svlogd -tt /var/log/gitlab/postgresql
mchacon  12727  0.0  0.0  12944   924 pts/0    S+   21:02   0:00 grep postgres

We then ran sudo chef-client but this didn't bring Postgres online, so proceeded with manually starting Postgres:

mchacon@postgres-dr-archive-01-db-gprd.c.gitlab-production.internal:~$ systemctl status postgresql.service
● postgresql.service - PostgreSQL RDBMS
   Loaded: loaded (/lib/systemd/system/postgresql.service; enabled; vendor preset: enabled)
   Active: active (exited) since Sat 2021-08-07 20:53:57 UTC; 19min ago
 Main PID: 1682 (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/postgresql.service



mchacon@postgres-dr-archive-01-db-gprd.c.gitlab-production.internal:~$ sudo gitlab-ctl status postgresql
down: postgresql: 1658s; run: log: (pid 3116) 1658s
mchacon@postgres-dr-archive-01-db-gprd.c.gitlab-production.internal:~$ sudo gitlab-ctl start postgresql
ok: run: postgresql: (pid 22749) 0s, normally down
mchacon@postgres-dr-archive-01-db-gprd.c.gitlab-production.internal:~$ ps aux |grep postgres
root      1866  0.0  0.0   4240   744 ?        Ss   20:53   0:00 runsv postgres_exporter
root      2125  0.0  0.0   4384   684 ?        S    20:53   0:00 svlogd -tt /var/log/prometheus/postgres_exporter
root      2138  0.3  0.0 1404788 34544 ?       Sl   20:53   0:06 /opt/prometheus/mtail/mtail -progs /opt/prometheus/mtail/progs -logs /var/log/apt/term.log,/var/log/syslog,/var/log/td-agent/td-agent.log,/var/log/gitlab/postgresql/postgresql.csv,/var/log/wal-g/wal-g.log,/var/log/wal-g/wal-g_backup_push.log,/var/log/gitlab/pgbouncer/pgbouncer.log -logtostderr
prometh+  2154  0.0  0.0 113292 12652 ?        Sl   20:53   0:01 /opt/prometheus/postgres_exporter/postgres_exporter --extend.query-path=/opt/prometheus/postgres_exporter/queries.yaml
root      3112  0.0  0.0   4240   732 ?        Ss   20:54   0:00 runsv postgresql
root      3116  0.0  0.0   4384   660 ?        S    20:54   0:00 svlogd -tt /var/log/gitlab/postgresql
gitlab-+ 22749  2.8  0.3 8685384 232384 ?      Ss   21:22   0:00 /opt/gitlab/embedded/bin/postgres -D /var/opt/gitlab/postgresql/data
gitlab-+ 22756 14.0  0.1 8686452 98760 ?       Rs   21:22   0:00 postgres: startup   recovering 000000050005B634000000EA
gitlab-+ 22869  0.0  0.0 8685384 4244 ?        Ss   21:22   0:00 postgres: checkpointer
gitlab-+ 22871  0.0  0.0 8685384 6848 ?        Ss   21:22   0:00 postgres: background writer
mchacon  22950  0.0  0.0  12944   928 pts/0    S+   21:22   0:00 grep postgres
mchacon@postgres-dr-archive-01-db-gprd.c.gitlab-production.internal:~$

After the database was able to initialize we were able to connect:

mchacon@postgres-dr-archive-01-db-gprd.c.gitlab-production.internal:~$ sudo ps aux |grep postgres
root      1866  0.0  0.0   4240   744 ?        Ss   20:53   0:00 runsv postgres_exporter
root      2125  0.0  0.0   4384   684 ?        S    20:53   0:00 svlogd -tt /var/log/prometheus/postgres_exporter
root      2138  0.6  0.0 1552252 32388 ?       Sl   20:53   0:20 /opt/prometheus/mtail/mtail -progs /opt/prometheus/mtail/progs -logs /var/log/apt/term.log,/var/log/syslog,/var/log/td-agent/td-agent.log,/var/log/gitlab/postgresql/postgresql.csv,/var/log/wal-g/wal-g.log,/var/log/wal-g/wal-g_backup_push.log,/var/log/gitlab/pgbouncer/pgbouncer.log -logtostderr
prometh+  2154  2.5  0.0 115980 39012 ?        Sl   20:53   1:21 /opt/prometheus/postgres_exporter/postgres_exporter --extend.query-path=/opt/prometheus/postgres_exporter/queries.yaml
root      3112  0.0  0.0   4240   732 ?        Ss   20:54   0:00 runsv postgresql
root      3116  0.0  0.0   4384   660 ?        S    20:54   0:00 svlogd -tt /var/log/gitlab/postgresql
gitlab-+  4884  8.4 13.9 8724888 8596324 ?     Ss   21:33   1:11 postgres: gitlab-monitor gitlabhq_production 127.0.0.1(35300) idle
gitlab-+  6653 10.7 13.9 8749392 8626724 ?     Ss   21:33   1:28 postgres: gitlab-monitor gitlabhq_production 127.0.0.1(36116) idle
mchacon  12126  0.0  0.0  12944  1084 pts/0    S+   21:47   0:00 grep postgres
gitlab-+ 19661  0.0  0.0  34400  5324 ?        Ss   21:26   0:00 postgres: stats collector
gitlab-+ 19700  2.3  0.7 8740672 458980 ?      Ss   21:26   0:29 postgres: postgres_exporter gitlabhq_production 127.0.0.1(56534) idle
gitlab-+ 22749  0.0  0.3 8685384 232340 ?      Ss   21:22   0:00 /opt/gitlab/embedded/bin/postgres -D /var/opt/gitlab/postgresql/data
gitlab-+ 22756 30.7 13.8 8687780 8569992 ?     Ss   21:22   7:33 postgres: startup   waiting for 000000050005B63F000000EC
gitlab-+ 22869  1.9 13.8 8697872 8535728 ?     Ss   21:22   0:28 postgres: checkpointer
gitlab-+ 22871  1.5 12.6 8686872 7814008 ?     Ss   21:22   0:22 postgres: background writer
gitlab-+ 25993  5.5 13.8 8725192 8591804 ?     Ss   21:43   0:13 postgres: gitlab-monitor gitlabhq_production 127.0.0.1(43838) idle
mchacon@postgres-dr-archive-01-db-gprd.c.gitlab-production.internal:~$ sudo gitlab-psql
psql (12.6)
Type "help" for help.

gitlabhq_production=# \q

More information will be added as we investigate the issue.

Timeline

View recent production deployment and configuration events / gcp events (internal only)

All times UTC.

2021-08-07

  • 21:16 - @mchacon3 declares incident in Slack.

Corrective Actions

Corrective actions should be put here as soon as an incident is mitigated, ensure that all corrective actions mentioned in the notes below are included.

  • ...

Note: In some cases we need to redact information from public view. We only do this in a limited number of documented cases. This might include the summary, timeline or any other bits of information, laid out in out handbook page. Any of this confidential data will be in a linked issue, only visible internally. By default, all information we can share, will be public, in accordance to our transparency value.


Click to expand or collapse the Incident Review section.

Incident Review
  • Ensure that the exec summary is completed at the top of the incident issue, the timeline is updated and relevant graphs are included in the summary
  • If there are any corrective action items mentioned in the notes on the incident, ensure they are listed in the "Corrective Action" section
  • Fill out relevant sections below or link to the meeting review notes that cover these topics

Customer Impact

  1. Who was impacted by this incident? (i.e. external customers, internal customers)
    1. ...
  2. What was the customer experience during the incident? (i.e. preventing them from doing X, incorrect display of Y, ...)
    1. ...
  3. How many customers were affected?
    1. ...
  4. If a precise customer impact number is unknown, what is the estimated impact (number and ratio of failed requests, amount of traffic drop, ...)?
    1. ...

What were the root causes?

  • ...

Incident Response Analysis

  1. How was the incident detected?
    1. ...
  2. How could detection time be improved?
    1. ...
  3. How was the root cause diagnosed?
    1. ...
  4. How could time to diagnosis be improved?
    1. ...
  5. How did we reach the point where we knew how to mitigate the impact?
    1. ...
  6. How could time to mitigation be improved?
    1. ...
  7. What went well?
    1. ...

Post Incident Analysis

  1. Did we have other events in the past with the same root cause?
    1. ...
  2. Do we have existing backlog items that would've prevented or greatly reduced the impact of this incident?
    1. ...
  3. Was this incident triggered by a change (deployment of code or change to infrastructure)? If yes, link the issue.
    1. ...

Lessons Learned

  • ...

Guidelines

Resources

  1. If the Situation Zoom room was utilised, recording will be automatically uploaded to Incident room Google Drive folder (private)
Edited by Marcel Chacon