fix: the sidecar join is on every query including COUNT(*) — 126-202x, and a latent wrong obs_count
_availability builds join_clause once and appends it to all three query
shapes, COUNT(*) included:
count_sql = f"SELECT COUNT(*) FROM {main}{join_clause}"The join exists so a componentID naming a sidecar attribute (e.g. GEO_LEVEL)
can be resolved. It is applied whether or not the query references a sidecar
column.
Cost
Measured on cenprof-2021 (519,785,416 observations, 70,039-row sidecar),
across duckdb 1.5.5 and 1.5.6:
| measured range | |
|---|---|
COUNT(*) without the join |
4.5 - 51 ms |
COUNT(*) with the sidecar join |
3.9 - 6.4 s |
| ratio | 126x - 202x (882x on one cold run) |
| whole availability request, as shipped vs join elided | 7.0 s vs 3.1 s (~2.2x) |
Trust the ratio, not the absolute — see #2 (closed).
DuckDB elides the join for the per-dimension DISTINCTs, which project one
column. It cannot for COUNT(*): a LEFT JOIN whose right side has duplicate
keys legitimately changes the count. So the entire delta is the count, and the
reason it is not elidable is also the reason it is a correctness risk.
The correctness half
On this bundle the sidecar is exactly 1:1 — 70,039 rows, 70,039 distinct
(REF_AREA, GEO_LEVEL) — so the joined count returns the true 519,785,416.
Nothing in our code guarantees that. A bundle whose sidecar carries duplicate
join keys would silently inflate the obs_count we hand .Stat for its
"(N observations)" badge. Silent, because every other field of the response stays
correct.
This is sdmxlib #388
Their tests/bench/test_bundle_observations.py pins the same defect at 3814x
on a 1.14 B partition, and their query compiler elides a join no referenced
column needs. A compiler can decide that per query; an f-string built above all
three shapes cannot. That asymmetry is the general argument behind #6.
Fix
Join only when a column the query actually references lives on the sidecar —
never for COUNT(*). #6 makes it structural rather than a rule to remember, but
this is worth fixing on its own first: it is self-contained, it is the largest
single win available, and it does not wait on the migration.