Write OAuth applications to IAM immediately on commit

Daniele Braccianirequested to merge
db/606195/introduce-best-effort-immediate-write-to-iam-data-replication into master

What does this MR do and why?

This MR adds Layer 2 to IAM data replication for OAuth applications. Today, an outbox row is written in the same transaction as every create/update/destroy (Layer 1), and DrainWorker drains it to IAM about once a minute (Layer 3). That leaves up to a minute where IAM doesn't see the change yet.

Layer 2 closes that gap: right after the outbox row commits, an after_commit hook makes a best-effort synchronous call to IAM. Best-effort is by design per ADR-DATA-01: it can't block or roll back the transaction, and on failure it just logs and drops, leaving the row for DrainWorker to retry. On success it marks the row delivered.

Most of the diff is not new logic: it relocates the delivery code that used to live in DrainWorker into a shared OauthApplicationReplicator#deliver, so both Layer 2 and Layer 3 call the same method and the field set sent to IAM can't drift between them. The actual new code is the Layer 2 write itself (the after_commit hook). Log lines are tagged "Layer 2"/"Layer 3".

Owner mirroring keeps sending the placeholder text (An administrator / An anonymous service) via owner_label. This is intentionally temporary: it will be removed once IAM owns the owner label, tracked in #623571 (closed).

Heads up: every gRPC failure (including expected Layer 2 ones) reports to Sentry, pre-existing behavior. Layer 2 also runs on the Puma thread and makes two sequential gRPC calls for an upsert (no update RPC in IAM yet, 616947) — worst case adds a few seconds to the request. Accepted tradeoff.

Related to #606195 (closed)

References

Screenshots or screen recordings

Not applicable, no UI change.

How to set up and validate locally

Enable the feature flag in the Rails console:

Feature.enable(:iam_data_replication)

Happy flow (end to end)

  1. Create an OAuth application under http://gdk.test:3000/oauth/applications, redirect URI http://gdk.test:3000/callback, scope api.
  2. Unlike !248091 (merged), don't wait ~1 minute — Layer 2 delivers immediately. Confirm right away in the Rails console: 
    Authn::IamOutbox.last.l0_delivered_at # => present immediately, l0_attempts == 0
  3. Initiate a consent flow against IAM to confirm it's really there (substitute your app's uid as client_id): 
    http://gdk.test:8084/oauth2/authorize?client_id=<CLIENT_ID>&redirect_uri=http://gdk.test:3000/callback&response_type=code&scope=api&state=my-random-state&code_challenge=<CODE_CHALLENGE>&code_challenge_method=S256

Manual check (Rails console)

app = Authn::OauthApplication.create!(name: "Test", redirect_uri: "http://gdk.test:3000/callback")
row = Authn::IamOutbox.where(entity_id: app.id).last
row.l0_delivered_at # => present, set by Layer 2, not the drain worker
row.l0_attempts     # => 0, DrainWorker never had to run

With the flag disabled (Feature.disable(:iam_data_replication)), confirm no outbox row is written and Layer 2 is a no-op.

Debugging 
row.attributes.slice('l0_delivered_at', 'l0_attempts', 'l0_last_error')

l0_last_error holds the gRPC reason for upstream failures, or the error class name for errors raised before the RPC. Also check the Rails log for a line with message: "IAM immediate write failed" and layer: 2.

Query plan

Danger flagged several iam_outbox operations for database review, but the majority is code relocation: the delivery logic (the l0_undelivered_lookup read and the update_all that marks rows delivered / records the error) moved out of DrainWorker into the shared OauthApplicationReplicator#deliver, so Danger re-triggers on the moved lines. These are the exact same queries already analyzed in the Layer 3 query plan, unchanged in scope and WHERE clause. All three shapes (the l0_undelivered_lookup scope, the bounded pending read, and the mark-delivered / error-path update_all + update_counters) resolve through index_iam_outbox_l0_undelivered_lookup with no sequential scan and no new index. That analysis applies here as-is.

The only genuinely new write in this MR is the Layer 2 mark-delivered at outboxable.rb#L124:

UPDATE "iam_outbox" SET "l0_delivered_at" = NOW(), "updated_at" = NOW() WHERE "iam_outbox"."id" = <id>;

It is a single-row update keyed on the primary key, so it needs no query plan: the lookup is a primary-key index hit with no scan to analyze.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist.

Edited by Daniele Bracciani

Merge request reports