Create a retention policy for job logs

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

Manual Cleanup Options

Self Managed Only - https://docs.gitlab.com/administration/cicd/job_logs/#how-to-remove-job-logs

Self Managed Only - https://docs.gitlab.com/administration/cicd/job_artifacts_troubleshooting/#delete-old-job-logs-and-artifacts-for-a-project

API Endpoint - The primary method mentioned is the Erase a Job API, which allows deletion of individual job logs through the API.

What did we learn?

Results

📋 Summary

  • Respondents primarily access job logs for debugging failed jobs, with 83% of respondents indicating this reason. 54% of respondents require access to job logs for up to 30 days. 
  • The most common challenge faced in managing artifacts is storage. Storage is affected by the number of artifacts generated from pipeline runs, not the individual artifact file sizes. 
  • Users expressed a need for improved job log interfaces, including timestamps and auto-collapsing sections for better usability.

💡 Suggestions

  • A mechanism to bulk delete job logs after a designated retention period would significantly free up storage space. Here are some ideas:
    • For job logs generate in the future - Automated process
      • Provide an option for users to opt into automated removal, running cleanup at specified intervals.
      • Add job logs to the existing expire_in keyword to set a designated retention period.
    • For job logs generated in the past - Bulk deletion and archive
      • Provide a "delete all" button for job logs from a specific day forward.
      • Enable bulk deletion through the API.
      • Develop a better search and multi-select mechanism in the UI for easier management of job logs.
  • Implement auto-collapsing for script sections in the job log to help users navigate more easily when there is a lot of output.

💬 Detailed Findings and Respondents Background

Primary Workflows for Job Logs:

  • The primary reason for accessing job logs is to debug when jobs fail (10 responses).
  • Other reasons include seeing test results, checking build speeds, ensuring reproducibility, auditing specific tests, and assessing project health (1 response each).

Duration of Job Log Access:

  • 54% of respondents (7 people) need access to job logs for 30 days.
  • Other durations include (1 response each):
    • Custom retention policy
    • One week
    • Indefinite for tagged releases, 30 days for per-commit builds
    • 3 months
    • 6 months
  • 1 respondent was unsure about the required access duration.

Challenges in Managing Artifact Storage:

  • Storage is affected by many teams running numerous pipelines.
  • Challenges are more related to Docker images rather than artifacts/logs.
  • Issues with cleanup policies from runners and specific retention policies were highlighted.

Additional Needs and Improvements:

  • Respondents look forward to having timestamps on job logs as a default feature.
  • The feature flag FF_SCRIPT_SECTIONS has significantly improved the user experience and is suggested to become default.
  • Respondent suggests an interface with auto-collapsing sections (e.g. script sections from the bottom) for better usability.

Compliance Requirements:

  • 54% (7 respondents) work under internal requirements.
  • 38% (5 respondents) need to comply with GDPR.
  • 15% (2 respondents each) are either not working with any of the listed requirements, unsure, or complying with PCI-DSS and FDA standards.

Background:

This research is part of https://gitlab.com/gitlab-org/ux-research/-/issues/2971+, conducted in May 2024. It targeted Dev team leads, platform engineers and software developers from a mix of SMB and enterprise-size customers. The goal was to determine specific use cases for restricting download access to artifacts and to understand how job logs are used and how long they need to be retained.

The survey ran for a month, yielding 12 valid responses. Of these, 53% were from technology companies involved in creating pipelines, writing code, and running tests. Additionally, 54% of respondents use Self-Managed GitLab, while 23% use GitLab.com.

Proposal (v1 scope - draft) - Updated Sept 21 2026

  1. Introduce a job log retention period for new projects only - Following this period, the job logs will be permanently deleleted
    1. Job logs will be retained for maximum 6 months on GitLab.com (exact period TBD to be confirmed with Legal/Compliance)
    2. An administrator on a Self Managed/Dedicated instance can override this value
  2. For retroactive job log cleanup - we can choose one of the following options
    1. Option 1: Add an opt-in admin task. For customers who want the policy applied to already-stored logs, we propose a separate administrator-triggered task that stamps expiry dates onto existing logs in batches, with deletion still handled by the existing cleanup path.
    2. Option 2: Updated API for bulk removal action. This solution is detailed in Introduce additional parameters to bulk delete ... (#14495)

Later phases

  1. Group/Project owners select a lower value than set by instance admin

Links

Edited by Rutvik Shah