GitLab Runner Feature Request: Container Image Usage Tracking for Intelligent Pruning
Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.
Release notes
Problem to solve
Container registries can grow in size over time as many images or tags are added, which slows down retrieval of available tags/images and consumes substantial storage space. Currently, GitLab Runner doesn't provide a native mechanism to track which container images are actually being used by CI/CD jobs. This makes it difficult to implement intelligent pruning strategies like Least Recently Used (LRU) for container images without manually creating custom tracking systems.
Intended users
User experience goal
To create a "set-and-forget" image management system that eliminates the cognitive burden of container image management from DevOps teams while optimizing resource usage transparently.
Proposal
Usage Tracking: Record which images are used by each job, including:
- Image name and tag
- Timestamp of usage
- Job ID and project reference
- Runner ID
API Endpoints: Create API endpoints to query image usage statistics, such as:
- Most/least frequently used images
- Most/least recently used images
- Images that haven't been used in X days/weeks
Pruning Commands: Add GitLab Runner commands to prune images based on usage data
Metrics and Reporting: Expose Prometheus metrics for container image usage to track:
- Total size of cached images
- Number of cached images
- Usage frequency metrics
- Pruning efficiency stats
Further details
The implementation could use:
- A local database (SQLite) to store image usage data within the GitLab Runner's data directory
- Hooks in the Docker executor workflow to record when images are used
- Integration with the existing Docker executor's image handling code
- Prometheus metrics for monitoring and visualization
Benefits This feature would:
- Reduce disk space usage by intelligently pruning unused images
- Improve runner performance by maintaining an optimized image cache
- Provide insights into which images are actually being used
- Reduce manual intervention needed for runner maintenance
- Allow for more efficient resource utilization in CI/CD infrastructure
Permissions and Security
Documentation
Availability & Testing
Available Tier
Feature Usage Metrics
- Implementation Rate: Percentage of eligible GitLab Runner installations that enable the image tracking feature
- Configuration Customization: Percentage of users who modify default settings versus those who use defaults
- Strategy Distribution: Breakdown of which pruning strategies (LRU, time-based, size-based) are most commonly used
- Enterprise vs. Small Team Adoption: Adoption rates across different organization sizes and types
What does success look like, and how can we measure that?
Storage Efficiency
- Disk Space Reduction: Average reduction in disk space usage by 30-50% after implementing intelligent pruning compared to before implementation
- Storage Growth Rate: Decreased rate of storage growth over time (e.g., 75% slower growth rate)
- Image Retention Accuracy: 90%+ retention of actively used images while successfully pruning unused ones
Performance Metrics
- Build Time Impact: No more than 1-2% increase in average job execution time due to tracking overhead
- Image Pull Frequency: 25-40% reduction in redundant image pulls
- Cache Hit Rate: Increased cache hit rate by at least 20% for frequently used images
-->