Fix status export limitations
Problem to solve
Currently the export only exports the last 100 rows, this is a hardcoded limitation to get around potential performance issues.
We need to allow larger extracts to be performed, without limits, so that all the data can be retrieved.
Proposal
Paginate the results and retrieve all the data.
Implementation approach
Replace the hardcoded 100-row limit in the finder with a paginated fetch that iterates through all relevant records in batches. Integrate this with CsvBuilder::Stream to stream the results efficiently to the CSV, while limiting total rows/size to prevent excessively large exports.
We can make use of [updated_at, id] for the keyset as we already have the data ordered by those two columns by default and a composite index that matches, but we need to be cautious that other orderings will impact which indexes can be used and will impact performance.
No DB schema changes required, however, we need to be cautious of what sorting and ordering we are allowing as that will impact the efficient use of indexes.
- Update
ee/app/finders/compliance_management/compliance_framework/project_requirement_status_finder.rbto remove the limit and paginate the wider dataset(to the upper limit of our export). - Modify
ee/app/services/compliance_management/compliance_framework/project_requirement_statuses/export_service.rbto batch through the keyset results. - Use
CsvBuilder::Streamwith an upper limit to protect from huge exports.
- No frontend changes proposed.
- No additional documentation required.