Skip to content

Add batch support while querying downloadable artifacts

David Wainaina requested to merge docs-add-batch-when-querying-downloadable-artifacts into master

What does this MR do and why?

I have a customer whose job was killed whenever they ran project.builds.with_downloadable_artifacts. The solution is to query in small batches to reduce memory utilization.

I have seen several tickets with the same error.

Screenshots or screen recordings

Screenshots are required for UI changes, and strongly recommended for all other merge requests.

Before After

How to set up and validate locally

Run the following script in the rails console.

# Get debug output
Rails.logger.level = Logger::DEBUG
project = Project.find_by_full_path('path/to/project')
builds_with_artifacts =  project.builds.with_downloadable_artifacts

After running the script, the job is killed

In the dmesg logs

 Killed process 5648 (ruby) total-vm:5757928kB, anon-rss:3977872kB, file-rss:0kB, shmem-rss:0kB

After changing the script and adding batches, the scripts run without getting killed.

# Get debug output
Rails.logger.level = Logger::DEBUG
project = Project.find_by_full_path('path/to/project')
builds_with_artifacts =  project.builds.with_downloadable_artifacts.find_each(batch_size: 1000)

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by David Wainaina

Merge request reports