Skip to content
Snippets Groups Projects
New look: Off

2022-04-06: enforce TLS rate limiting in pages

    • Closed Issue created by ops-gitlab-net @ops-gitlab-net

    Production Change

    Change Summary

    Enforce TLS rate limits in pages.

    For context (quoting https://gitlab.com/gitlab-org/gitlab-pages/-/issues/632):

    We already have rate limits implemented and enabled, but they work on the HTTP level. This means that we first need to finish TLS handshake, then we look into request and decide if we want to handle or reply with 429 Too many requests.

    On a few recent incidents, these rate limits weren't enough to save us for a few reasons:

    1. these limits were too high(we wanted to roll them out quicker, so put too high values). We autoscaled the number of pods quicker than rate limits were reached on any of the pods.
    2. even if rate limits functioned as intended, they would not prevent CPU overload because they work AFTER we terminate TLS, and as you see from the profiling image below, we spend almost 50% CPU time on TLS handshakes.

    Also, we can in theory set TLS connections limit lower than HTTP one because all normal browsers establish only a single HTTP2 connection and then load everything through it. So these TLS limits (per IP and per domain) can reject the attacks quicker.

    We have been experimenting with values in production while enforcing was disabled, and we think we've reached reasonable rate limit thresholds. These are:

        rateLimitTLSSourceIP: 20
    rateLimitTLSSourceIPBurst: 50
    rateLimitTLSDomain: 40
    rateLimitTLSDomainBurst: 100

    Change Details

    1. Services Impacted - ServicePages
    2. Change Technician - @gsgl
    3. Change Reviewer - @f_santos
    4. Time tracking - ~20min
    5. Downtime Component - none

    Detailed steps for the change

    Pre-Change Steps - steps to be completed before execution of the change

    Estimated Time to Complete (mins) - 2m

    Change Steps - steps to take to execute the change

    Estimated Time to Complete (mins) - 2m

    Post-Change Steps - steps to take to verify the change

    Estimated Time to Complete (mins) - 10m

    • Monitor pipeline

    Rollback

    Rollback steps - steps to be taken in the event of a need to rollback this change

    Estimated Time to Complete (mins) - 5m

    Either:

    OR

    Monitoring

    Key metrics to observe

    • Metric: rate limited requests
      • Location: https://log.gprd.gitlab.net/goto/4e5734d0-b54d-11ec-afaf-2bca15dfbf33
      • What changes to this metric should prompt a rollback: rate limiting a concerning number of requests and the rate limiter is either tls_connections_by_domain or tls_connections_by_source_ip (extremely unlikely since we're not seeing any of that currently and we're doubling the thresholds)

    Summary of infrastructure changes

    • Does this change introduce new compute instances?
    • Does this change re-size any existing compute instances?
    • Does this change introduce any additional usage of tooling like Elastic Search, CDNs, Cloudflare, etc?

    Change Reviewer checklist

    C4 C3 C2 C1:

    • The scheduled day and time of execution of the change is appropriate.
    • The change plan is technically accurate.
    • The change plan includes estimated timing values based on previous testing.
    • The change plan includes a viable rollback plan.
    • The specified metrics/monitoring dashboards provide sufficient visibility for the change.

    C2 C1:

    • The complexity of the plan is appropriate for the corresponding risk of the change. (i.e. the plan contains clear details).
    • The change plan includes success measures for all steps/milestones during the execution.
    • The change adequately minimizes risk within the environment/service.
    • The performance implications of executing the change are well-understood and documented.
    • The specified metrics/monitoring dashboards provide sufficient visibility for the change. - If not, is it possible (or necessary) to make changes to observability platforms for added visibility?
    • The change has a primary and secondary SRE with knowledge of the details available during the change window.

    Change Technician checklist

    • This issue has a criticality label (e.g. C1, C2, C3, C4) and a change-type label (e.g. changeunscheduled, changescheduled) based on the Change Management Criticalities.
    • This issue has the change technician as the assignee.
    • Pre-Change, Change, Post-Change, and Rollback steps and have been filled out and reviewed.
    • This Change Issue is linked to the appropriate Issue and/or Epic
    • Necessary approvals have been completed based on the Change Management Workflow.
    • Change has been tested in staging and results noted in a comment on this issue.
    • A dry-run has been conducted and results noted in a comment on this issue.
    • SRE on-call has been informed prior to change being rolled out. (In #production channel, mention @sre-oncall and this issue and await their acknowledgement.)
    • Release managers have been informed (If needed! Cases include DB change) prior to change being rolled out. (In #production channel, mention @release-managers and this issue and await their acknowledgment.)
    • There are currently no active incidents.
    8 of 30 checklist items completed · Edited

    Designs

      Child items ...

      2. Activity

      Please register or sign in to reply