ANALYZE namespaces table every three hours
As a corrective action we need to execute an analyze twice daily of the following table:
namespaces
Acceptance criteria:
-
create a cronjob to execute an ANALYZE twice daily for the table
namespaces
.
Designs
- Show closed items
Activity
-
Newest first Oldest first
-
Show all activity Show comments only Show history only
- Jose Finotto added Database corrective action workflow-infraReady + 1 deleted label
added Database corrective action workflow-infraReady + 1 deleted label
- Jose Finotto marked this issue as related to #3962 (closed)
marked this issue as related to #3962 (closed)
- Author Developer
As discussed we will execute in an interval of 12 hours.
- Jose Finotto mentioned in merge request gitlab-cookbooks/gitlab-patroni!74 (closed)
mentioned in merge request gitlab-cookbooks/gitlab-patroni!74 (closed)
- Author Developer
- Jose Finotto added docsP1 label
added docsP1 label
- Author Developer
@nnelson could you please review the MR ?
Collapse replies - Contributor
I have reviewed the MR, and it was missing spec tests, so I created a new branch from yours, and implemented new tests. You pre-approved the tests, I have not change the code since, but had to wrestle with kitchen on the CI pipeline for hours today.
Sorry for the delay. The new MR which include the now passing tests has been merged.
Tonight I will create the required role version bumps to the
staging
environment.I doubt I will be able to get any management approval for a
staging
environment database change plan so late in the evening, though.Edited by Nels Nelson - Author Developer
@nnelson thanks for the update, please do not forget to add the involved in your answer.
cc: the manager or I follow up in Europe with the managers in the EMEA times.
thank you.
- Contributor
@Finotto - Here is the
staging
change plan: #3976 (closed)I have marked Alejandro as the reviewer, but his time zone is later than yours, so feel free to review yourself.
- Jose Finotto changed due date to March 16, 2021
changed due date to March 16, 2021
- Jose Finotto changed the description
Compare with previous version changed the description
- Brent Newton added severity1 label
added severity1 label
- Brent Newton changed title from ANALYZE namespaces table daily to ANALYZE namespaces table twice daily
changed title from ANALYZE namespaces table daily to ANALYZE namespaces table twice daily
- Brent Newton changed the description
Compare with previous version changed the description
- 🤖 GitLab Bot 🤖 added review-requested label
added review-requested label
- Nels Nelson mentioned in merge request gitlab-cookbooks/gitlab-patroni!75 (merged)
mentioned in merge request gitlab-cookbooks/gitlab-patroni!75 (merged)
- Author Developer
Meanwhile, we do not roll out this change in production I am executing manually the
analyze
periodically. - Author Developer
Output:
gitlabhq_production=# analyze namespaces; ANALYZE gitlabhq_production=# select now(); now ------------------------------- 2021-03-16 09:44:53.011392+00 (1 row)
- Maintainer
This incident was closed with a review-requested label still applied. Either remove the label or complete a review and add an IncidentReview-Completed label.
Please review the Incident Workflow section on the Incident Management handbook page for more information.
- 🤖 GitLab Bot 🤖 reopened
reopened
- 🤖 GitLab Bot 🤖 added auto updated label
added auto updated label
- Author Developer
I executed another analyze:
gitlabhq_production=# analyze namespaces; select now(); ANALYZE now ------------------------------- 2021-03-16 20:32:35.021017+00 (1 row)
- Author Developer
Analyze executed:
gitlabhq_production=# analyze namespaces; select now(); ANALYZE now ------------------------------ 2021-03-17 09:27:22.62514+00
- Nels Nelson mentioned in issue #3976 (closed)
mentioned in issue #3976 (closed)
- Author Developer
Analyze executed:
gitlabhq_production=# analyze namespaces; select now(); ANALYZE now ------------------------------ 2021-03-17 22:20:23.37416+00 (1 row)
- Author Developer
Analyze executed:
gitlabhq_production=# analyze namespaces; select now(); ANALYZE now ------------------------------- 2021-03-18 14:06:15.734934+00
- Jose Finotto mentioned in issue #4011 (closed)
mentioned in issue #4011 (closed)
- Brent Newton changed title from ANALYZE namespaces table twice daily to ANALYZE namespaces table every three hours
changed title from ANALYZE namespaces table twice daily to ANALYZE namespaces table every three hours
- Contributor
- Nels Nelson mentioned in issue #4015 (closed)
mentioned in issue #4015 (closed)
- Author Developer
Executing another analyze:
gitlabhq_production=# analyze namespaces; select now(); ANALYZE now 2021-03-18 20:22:05.522913+00
- Author Developer
@nnelson in stg seems we are good:
Cgitlab-psql@patroni-01-db-gstg.c.gitlab-staging-1.internal:/home/jfinotto$ crontab -l # Chef Name: GCP snapshot pg_start_backup 0 */6 * * * gitlab-psql -tc 'SELECT pg_is_in_recovery()' | grep 'f' && gitlab-psql -c "SELECT pg_start_backup('GCP snapshot', FALSE, FALSE);" # Chef Name: GCP snapshot pg_stop_backup 20 */6 * * * gitlab-psql -tc 'SELECT pg_is_in_recovery()' | grep 'f' && gitlab-psql -c "SELECT pg_stop_backup(FALSE, FALSE);" # Chef Name: full wal-g backup 0 0 * * * /opt/wal-g/bin/backup.sh >> /var/log/wal-g/wal-g_backup_push.log 2>&1 # Chef Name: analyze_namespaces_table PATH="/usr/local/sbin:/usr/sbin/:/sbin:/usr/local/bin:/usr/bin:/bin" 0 */3 * * * /usr/local/bin/analyze-namespaces-table.sh
Edited by Jose Finotto - Author Developer
Executing another analyze in production:
gitlabhq_production=# analyze namespaces; select now(); ANALYZE now ------------------------------- 2021-03-19 08:36:53.426643+00 (1 row)
- Jose Finotto marked this issue as related to #4015 (closed)
marked this issue as related to #4015 (closed)
- Contributor
Closing this one as a duplicate and we will continue to leverage https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/12843
Edited by Brent Newton - Brent Newton closed
closed
- Maintainer
This incident was closed with a review-requested label still applied. Either remove the label or complete a review and add an IncidentReview-Completed label.
Please review the Incident Workflow section on the Incident Management handbook page for more information.
- 🤖 GitLab Bot 🤖 reopened
reopened
- Brent Newton removed review-requested label
removed review-requested label
- Brent Newton removed severity1 label
removed severity1 label
- Brent Newton removed docsP1 label
removed docsP1 label
- Brent Newton removed corrective action label
removed corrective action label
- Contributor
Closing this one as a duplicate and we will continue to leverage https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/12843
- Brent Newton closed
closed
- Brent Newton removed auto updated label
removed auto updated label
- Brent Newton removed Database label
removed Database label
- Brent Newton removed 1 deleted label
removed 1 deleted label
- Brent Newton removed workflow-infraReady label
removed workflow-infraReady label