ActionCable readiness review

Operational Readiness Guide for Infrastructure Services

For new services introduced in our GitLab SaaS infrastructure we want to review them for operational readiness before going live. This checklist should be a guide to help covering all important aspects for an infrastructure readiness review - not all points are mandatory, apply what makes sense. Goal of the readiness review should be to identify gaps, create issues for them, link them in the review and bring them to a solution.

The readiness review should mostly link to design docs or runbooks for referral. It is recommended to write most of the information falling out of the readiness review into the README.md in the services' runbook directory. Runbooks should be the main source of truth. Design docs and readiness reviews tend to be point-in-time snapshots and should not duplicate information in runbooks.

This guide also can be used to retroactively review already existing services.

For readiness review of new product features use this issue template: production_readiness.md

• Infrastructure epic to enable Websockets in Kubernetes as a separate service &355 (closed)
• Development epic for using ActionCable gitlab-org&3056

Summary

• Short overview mentioning purpose of the service, dependencies and owners
• Explain the scope of this review and what is explicitly out of scope.

Architecture

ActionCable will be handled in process by Rails using our existing Puma server.

The discussion on whether we should run ActionCable on the same Puma server as the application is in gitlab-org/gitlab#214061 (closed)

     sequenceDiagram
        participant Client
        participant Workhorse
        participant Rails/AC

      
        Client->>Workhorse: HTTP GET `Upgrade: websocket`
        Workhorse->>Rails/AC: proxy
        Rails/AC->>Rails/AC: open connection
        Rails/AC-->>Workhorse: HTTP 101 Switching Protocols
        Workhorse-->>Client: 200 OK
        Rails/AC->>Client: websocket traffic
        Client->>Rails/AC: websocket traffic

Documentation

• is there a blue print/design doc? https://about.gitlab.com/company/team/structure/working-groups/real-time/design_document.html
• do we have runbooks? (provide links)
• are runbooks up-to-date?
• where else is documentation for this service located?
• is there a service catalog entry? (provide link)
  • is service catalog listing all dependencies?
  • has service catalog links to all existing documentation?
  • is service catalog linking to readiness review?

Performance

• is there a runbook section with performance characteristics? (it should cover following considerations, provide link)
  • current requests/s (min, max, average), latency characteristics, saturation, ...
  • throtteling/limits
  • bottlenecks (cpu-bound, memory-bound, ...)
  • is there documentation on how/why we set certain config options that are affecting performance?

Database

Concerns raised about additional database connections in gitlab-org/gitlab#214061 (comment 357439952)

 With our ActionCable threads, whether in the web Puma or standalone Puma, we increased the number of threads that could use a DB connection. And the connection pool size calculation is based on the worst-case scenario where all of them are using a connection at the same time.

Memory

We had multiple issues with memory consumption when this rolled out previously:

Following the last time it was enabled there was an investigation into memory problems on production tracked in https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/11660 and gitlab-org/gitlab#267542 (closed)

In summary we found nothing conclusive in the investigation and believed there was some added (but not significant) memory pressure but nothing that we shouldn't be able to account for by adding more resources. One issue early on in the rollout was that we were consuming more memory because jemalloc was not enabled in the Kubernetes fleet. When we enabled it the second time we saw increased memory pressure due to the combination of pod scaling and hitting memory limits production#2940 (closed). For this rollout we will be enabling this feature on an isolated node pool and with new request/limits that have been adjusted since the first time it was enabled.

Redis

We are using the Persistent Redis and will monitor the rolout

&228 (comment 414346096)

I'd be OK with going with the persistent Redis at least while this is still controlled by a feature flag. If we see problems we can disable the flag and spin up a separate Redis, but I hope that's unlikely.

Scalability

• is there a runbook section with scalability information? (it should cover following considerations, provide link)
  • expected load in the future
  • how can we scale to the expected load?
  • can it be scaled across availability zones or regions?
  • are there scalability limitations?
  • are we doing performance tests?

Availability

• is there a runbook section covering availability considerations? (it should cover following topics, provide link)
  • failure modes of this service, blast radius, how long does it take to recover?
  • what happens on outage of services we are depending on?
  • Availability Zone (AZ) outage
  • split brain between AZs
  • region outage
  • other external dependencies that could affect availability
  • what other services are affected by an outage of this service?
  • is there an existing Recovery Time Objective (RTO) documented? How do we plan to achieve it?
  • do we have an error budget?
  • are we doing disaster recovery tests?
  • is there a failover procedure? Do we have runbook instructions?

Durability

• is there a runbook section covering durability considerations? (it should cover following topics, provide link)
• possible failure modes and how to recover from them
  • deletion by accident
  • disk failure
  • data corruption
  • GCP outage
  • ...
  • is there an existing Recovery Point Objective (RPO) documented? How do we plan to achieve it?
  • Backups
    • are we testing backup replay?
    • are we monitoring backups?
    • what is the backup retention policy?
    • are backups in a different logical and physical environment?

Security/Compliance

• is there a runbook section covering security considerations? (it should cover following topics, provide link)
  • list of access roles
  • Who has which role?
  • How do we protect access?
  • Auditability of access
  • Which entrypoints need protection?
  • How are we applying security updates? (OS and service)
  • Regulations/Policies applying? (PII, SOX, ...)
  • how do we protect customer data?
    • encryption at rest?
    • could customer data leak in logs?
    • how long do we keep logs?
• is someone from security included for the readiness review?

Monitoring

There is not much additional monitoring needed for this service because we are running Actioncable in Puma. We can use existing Puma metricsand the action cable pool metrics delivery#1210 (comment 414930104)

• is there a runbook section covering monitoring? (it should cover following topics, provide link)
  • list key SLIs. Are we monitoring them?
  • list SLOs. Are we monitoring/alerting on them?
  • list of relevant alerts
  • are alerts actionable and linking to a runbook?
  • do we have a metrics catalog entry for the service? (provide link)
  • list of relevant dashboards
  • list of relevant logs