perf(api): GET /tasks/ pagination COUNT re-runs every annotation over every row (~80% of the request's DB time)
Split out of #2807 (closed), which carries the full EXPLAIN (ANALYZE, BUFFERS) evidence.
PageNumberPagination calls .count() on the queryset it is handed, and
TaskViewSet.get_queryset() hands it the fully annotated queryset from
annotate_tasks_queryset. Django cannot count an aggregated queryset directly, so
it wraps the whole thing:
SELECT COUNT(*) FROM (
SELECT (EXISTS(SELECT 1 FROM projects_task c WHERE … lquery …)) AS "is_summary",
… every RawSQL / Exists / Count annotation …
FROM projects_task … GROUP BY projects_task.id, 1, 2, 3, 4
) subqueryEvery annotation is computed for every row in the project, purely to arrive at a number that only needs the rows counted. Measured on a 4,000-task project, best of 3 runs:
count query: 457–503 ms
page query: 115 msSo ~80% of the request's database time is the pagination count, and it is constant across pages — the Schedule's all-pages fetch pays it once per page request.
Fix sketch
Count on the filtered-but-unannotated queryset. TaskViewSet.get_queryset() already
has a clean seam: every _filter_tasks_by_* call is pure filtering and
annotate_tasks_queryset is the single final step, so the pre-annotation queryset
can be kept and used for the count.
Two things to get right, and they are why this is filed rather than done inline:
- Join fan-out. Some filters join to-many relations (
_filter_tasks_by_labels,_filter_tasks_mine). The grouped count collapses duplicates for free; an unannotated count would over-report. Count over.values("pk").distinct(), and test each filter that can multiply rows. filter_querysetruns afterget_queryset.SearchFilter(?search=) andOrderingFilterboth narrow/order the annotated queryset, so the count queryset has to go through the same backends orcountandresultswill disagree.
A test that asserts the count query does not contain lquery (i.e. the annotations
are gone from it) pins this cheaply and non-brittlely.
Related: #2814 removes the GROUP BY, which changes the shape of this query but does
not by itself make the count cheap — the correlated RawSQL subqueries would
still be evaluated per row inside the wrapped count. The two fixes compose.
Refs #2807 (closed), #2767 (closed).