Skip to content

Memory Usage Bounding

Summary

On large OpenBao instances, memory usage will grow from a few tricky places, beyond what we'd hope from proportional usage. Some investigation is discussed in https://github.com/orgs/openbao/discussions/1865:

  • High-cardinality (per-mount) metrics. Depending on collection interval &c, these can exceed the total memory usage of all the other objects (caches, per-mount backends, &c).
  • Cache size is at 131k entries by default. If large items are written (we limit secrets to 10KB but policies may exceed that depending on complexity), this could exceed 512kb per entry, leading to ~62GB of memory usage. We likely want some cache as PostgreSQL is a remote service and thus may have some latency to it. In Runway, we've set this to 2560 entries.
  • Maximum request size / JSON object size. This defaults to 30MB and 30MBk+128KB by default. In Runway, we've limited this to 768kb and 1MB respectively. Outside of policy complexity, we could've potentially bounded this to a lower limit.

Success criteria

  • High cardinality metrics are filtered
  • cache_size is set to a sane default
  • Request sizes are limited.