recommendations around hugepages for PostgreSQL
Topic to Evaluate
PostgreSQL training end April 2020 included a recommendation about huge pages for databases with a lot of shared buffers.
In previous roles, I've built Linux with hugepages on RHEL5, 6, and 7 for Oracle. It's the defacto setup for Oracle - of any size of database.
Transparent huge pages hurt PostgreSQL, so I think we should also recommend disabling that on GitLab deployments.
The advise on the course around the size of shared buffers was: test it. Similarly for hugepages, we should test that: I understand we have automated QA testing on our 10K architecture.
IMO, I think it probably fits into the category of 'it won't hurt' for any size of deployment. What possible downside can there be of a memory intensive workload having a clean, allocated block of memory which simultaneously also reduces the kernel workload.
For Omnibus, we cannot configure the setting, so it defaults to 'try'. Provide huge pages, and PostgreSQL will use/reserve them.
HugePages_Total: 1024
HugePages_Free: 980
HugePages_Rsvd: 973
HugePages_Surp: 0
Hugepagesize: 2048 kB
Hugepages are allocated at server boot as a contiguous chunk of memory. It's set aside by the kernel, and it's unavailable for use except by processes that can use huge pages. The kernel can't use that memory either.
An Omnibus installation is a busy server with lots of different processes (Sidekiq, Nginx, container registry, puma/unicorn, workhorse, etc., plus PostgreSQL ..) and so there's definitely scope for memory fragmentation. PostgreSQL having a nice clean block of memory could be very beneficial for our customers.
related:
- https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/7819
- https://www.percona.com/blog/2018/12/20/benchmark-postgresql-with-linux-hugepages/
- https://medium.com/@FranckPachot/did-you-forget-to-allocate-huge-pages-on-your-postgresql-server-7a97e7727b03
Tasks to Evaluate
-
Determine feasibility of the feature -
Create issue for implementation or update existing implementation issue description with implementation proposal -
Set weight on implementation issue -
If weight is greater than 5, break issue into smaller issues -
Add task -
Add task
Risks and Implementation Considerations
Team
-
Add workflowplanning breakdown feature and the corresponding ~devops::<stage>
and~group::<group>
labels. -
Ping the PM and EM.