fix: the sidecar join is on every query including COUNT(*) — 126-202x, and a latent wrong obs_count

_availability builds join_clause once and appends it to all three query shapes, COUNT(*) included:

count_sql = f"SELECT COUNT(*) FROM {main}{join_clause}"

The join exists so a componentID naming a sidecar attribute (e.g. GEO_LEVEL) can be resolved. It is applied whether or not the query references a sidecar column.

Cost

Measured on cenprof-2021 (519,785,416 observations, 70,039-row sidecar), across duckdb 1.5.5 and 1.5.6:

measured range
COUNT(*) without the join 4.5 - 51 ms
COUNT(*) with the sidecar join 3.9 - 6.4 s
ratio 126x - 202x (882x on one cold run)
whole availability request, as shipped vs join elided 7.0 s vs 3.1 s (~2.2x)

Trust the ratio, not the absolute — see #2 (closed).

DuckDB elides the join for the per-dimension DISTINCTs, which project one column. It cannot for COUNT(*): a LEFT JOIN whose right side has duplicate keys legitimately changes the count. So the entire delta is the count, and the reason it is not elidable is also the reason it is a correctness risk.

The correctness half

On this bundle the sidecar is exactly 1:1 — 70,039 rows, 70,039 distinct (REF_AREA, GEO_LEVEL) — so the joined count returns the true 519,785,416. Nothing in our code guarantees that. A bundle whose sidecar carries duplicate join keys would silently inflate the obs_count we hand .Stat for its "(N observations)" badge. Silent, because every other field of the response stays correct.

This is sdmxlib #388

Their tests/bench/test_bundle_observations.py pins the same defect at 3814x on a 1.14 B partition, and their query compiler elides a join no referenced column needs. A compiler can decide that per query; an f-string built above all three shapes cannot. That asymmetry is the general argument behind #6.

Fix

Join only when a column the query actually references lives on the sidecar — never for COUNT(*). #6 makes it structural rather than a rule to remember, but this is worth fixing on its own first: it is self-contained, it is the largest single win available, and it does not wait on the migration.