Skip to content
GitLab
Next
    • GitLab: the DevOps platform
    • Explore GitLab
    • Install GitLab
    • How GitLab compares
    • Get started
    • GitLab docs
    • GitLab Learn
  • Pricing
  • Talk to an expert
  • /
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
    • Contribute to GitLab
    Projects Groups Topics Snippets
  • Register
  • Sign in
  • gitlab-runner gitlab-runner
  • Project information
    • Project information
    • Activity
    • Labels
    • Members
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributor statistics
    • Graph
    • Compare revisions
    • Locked files
  • Issues 3.5k
    • Issues 3.5k
    • List
    • Boards
    • Service Desk
    • Milestones
    • Iterations
    • Requirements
  • Merge requests 102
    • Merge requests 102
  • CI/CD
    • CI/CD
    • Pipelines
    • Jobs
    • Artifacts
    • Schedules
    • Test cases
  • Deployments
    • Deployments
    • Environments
    • Releases
  • Packages and registries
    • Packages and registries
    • Container Registry
    • Model experiments
  • Monitor
    • Monitor
    • Incidents
  • Analytics
    • Analytics
    • Value stream
    • CI/CD
    • Code review
    • Insights
    • Issue
    • Repository
  • Activity
  • Graph
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
Collapse sidebar
  • GitLab.orgGitLab.org
  • gitlab-runnergitlab-runner
  • Issues
  • #28630
Closed
Open
Issue created Nov 02, 2021 by Arran Walker@ajwalkerMaintainer

Specially crafted docker images can exhaust resources on managers

Summary

For the Docker executor, when we run a job's main container, we copy the containers output directly to the trace log implementation. This implementation has a limit on how much data is stored.

!2534 (merged) introduced a user package that executes id inside of the user's provided container. This is used to fetch the containers uid and gid and allows us to chown files created by our helper image, whose files will also be created with uid 0.

Unfortunately, no such limit is placed on the output from id, so a specially crafted docker image that replaces how id works can have it return an excessive amount of data to fill and exhaust a limitless buffer on the Runner manager.

This problem only exists when FF_DISABLE_UMASK_FOR_DOCKER_EXECUTOR is enabled, however, it can be enabled at the job level. A simple protection is to ensure FF_DISABLE_UMASK_FOR_DOCKER_EXECUTOR cannot be enabled until the problem is resolved.

A similar problem exists for Docker's service output, which fails its 30s health check. However, whilst there's no limit here (and probably should be), this command reads the Docker's log file output via a different API endpoint. This appears to have an internal limit of 1MB, because we don't follow the output.

Solution

  • Disable FF_DISABLE_UMASK_FOR_DOCKER_EXECUTOR on the Runner fleet to protect GitLab hosted Runner Managers: https://gitlab.com/gitlab-com/gl-infra/chef-repo/-/merge_requests/805/
  • Explicitly set sane limits on any command/data returned from containers that are not directly fed into the trace log.

Closes #28630 (closed)

Edited Dec 02, 2021 by Georgi N. Georgiev
Assignee
Assign to
Time tracking