Add per-kind reference limits for AI Catalog flows

What does this MR do and why?

Adds Ai::Catalog::Flows::ReferenceLimits, which encodes the limits for AI Catalog references in a custom flow, such as how many sub-agents or depth of nesting. This has been explored in #628319 (closed), and we are generally pointing in this direction:

• There should be a count limit for each kind of reference. Each kind has its own performance cost: a sub-agent reference adds a prompt of up to 10,000 characters to the CI job environment, and adds about 275 tokens to every coordinator LLM call.
• Kinds of references this MR does not implement, such as MCP servers and nested flows, will have costs of their own. We cannot measure those costs until they are implemented. Keeping one limit per kind means each future kind gets its own number instead of inheriting one chosen for agents. DWS structures its ceilings the same way, one per kind, in its CatalogItems registry.
• Agent references can start at a limit of 20.
• A size limit of 112 KiB on the serialized agent list. The list is written into one CI job environment variable, and Linux allows at most 128 KiB in a single environment variable. The count limit cannot protect the variable on its own: 20 agents at maximum prompt size measure about 219 KiB.

There is no per-field cap on an agent's prompt, by design: a standalone agent in Duo Chat has no prompt limit, and for sub-agents the payload byte limit is what binds. The sizing tests state 10,000 as the planning assumption and also cover the true worst case, one agent whose prompt fills the whole 80 KiB definition limit.

The limits are frozen constants checked at save time, and by surrounding code convention, we are not providing an application setting for this. MergeRequestBlock::MAX_BLOCKS_COUNT is the precedent for this approach, and can be changed in a single line MR.

How this fits into &23526

This MR is the decision from #628319 (closed) expressed as code, published first so the numbers are reviewable on their own. The consumers are already planned:

Epic step	What it calls
#628153 (closed) MR 3 (save-time checks)	count_exceeded? on the include list and payload_size_exceeded? on the serialized resolved agents, in CreateService#validate_before_save and UpdateService#validate_item
#628315 (run time)	payload_size_exceeded? before StartWorkflowService writes the agent list into the CI job variable
Later kinds of references in &21832 (MCP servers, flows)	one new entry each in MAX_REFERENCES_PER_KIND

Nothing reads the module yet, so no feature flag is needed.

Test evidence

The spec has two layers: boundary tests for the check methods, and an executable version of the sizing model so a reviewer can verify the 112 KiB ceiling instead of trusting the arithmetic in the cost analysis.

Sizing model: measured payload sizes at the real field ceilings

The payload shape is the run-time contract from #628315: one entry per agent with item_id, version, name, description, system_prompt, toolset. Field ceilings are the real ones: Ai::Catalog::Item caps description at 1,024 characters, and 10,000 is the prompt size the sizing analysis assumed, with the true worst case tested separately.

Agents	Prompt size	Serialized size	Verdict
1	10,000 (max)	10.95 KiB	matches the ~11 KB estimate in the cost model
10	10,000 (max)	109.5 KiB	fits under 112 KiB
11	10,000 (max)	120.5 KiB	rejected
20	10,000 (max)	219.1 KiB	rejected. This is why the count limit alone cannot protect the variable
20	2,000 (typical)	~47 KiB	fits with room to spare

The last test also proves the ceiling plus the environment variable name stays under the 128 KiB Linux limit.

Boundary tests

• limit_for returns the configured limit and raises KeyError for an unknown kind, so a typo in a future caller raises an error instead of skipping the check.
• count_exceeded? at the limit (false), one above (true), zero (false).
• payload_size_exceeded? at the ceiling (false), one byte above (true), empty (false).
• The kind hash is frozen and the agent limit is 20.
• MAX_AGENT_PAYLOAD_SIZE is under 128 KiB.

References

• Decision issue: #628319 (closed)
• Consumer: #628153 (closed)
• Parent epic: &23526

Screenshots or screen recordings

No UI change.

How to set up and validate locally

Nothing calls this module yet, so the spec is the full validation:

bundle exec rspec ee/spec/services/ai/catalog/flows/reference_limits_spec.rb

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.