Skip to content
Back to Blog

Why Not Just Use a Database Transaction for Agent Refunds?

Published Editorial policy & corrections

Share:XBSMRedditHNEmail

Two agents try to refund the same order. A senior engineer looks at the design and asks the right question: this is a race condition. Put SELECT ... FOR UPDATE on the order row, check the ceiling, write, commit. Why put a gateway in front of the refund service?

For one important case, the engineer is right. If the effect is a write to a database you own, and every caller goes through that one service, use the transaction. It is simpler, faster and stronger than anything placed in front of the service. Keep it even if you add a gateway.

A refund is usually not that case. The money moves at the payment provider, in a database you cannot enlist in your transaction. This post runs the textbook row-lock design and a tool-boundary ledger against the same faults, then gives a decision table for when each one belongs.

Source and scope: the comparison is the runnable examples/why-not-a-row-lock in the ContextOS runtime, shipped in v0.1.0-alpha.2. Both sides use synthetic loopback services. The full argument, with prior art, is in the runtime’s transaction guide. The runtime is alpha; nothing here is a production performance claim.

The baseline is written carefully, and it wins the race

The baseline refund_service.py holds an INR 1,000 ceiling per order. It takes the write lock (SQLite’s BEGIN IMMEDIATE, playing the role of SELECT ... FOR UPDATE), checks refunds against the ceiling and writes atomically. It is shown with the provider call in both usual places: inside the lock, and after committing a reservation, with the amount either released or kept when the call fails. Two agents race to refund INR 800 each. One is applied and one is refused. Correct.

Then the faults that agents make common:

Part 1: a refund service with a row lock, ceiling INR 1,000 per order
  race: two agents, INR 800 each                     1 applied, 1 refused. Correct.
  lost response, lock held across the call           ledger INR 800, provider refunded INR 1,600. Over the limit.
  lost response, reserve, call, release on failure   ledger INR 800, provider refunded INR 1,600. Over the limit.
  stalled call, reserve and keep on failure          ledger INR 800, provider refunded INR 0. Owed refunds refused until someone edits the row.
  refund + store credit, two databases               each within its own ceiling; customer paid INR 1,600. Over the limit.
  Every lock held. The order's limit did not, because the money moves outside the database.

Every row the service locked stayed consistent. The order’s limit broke anyway, because the limit is about money at the provider, not about the row.

A timeout has two meanings the caller cannot tell apart

Your database and the payment provider cannot commit atomically. No payment provider joins a two-phase commit with your database. Every design therefore picks an order of operations, and a timeout tests that choice:

DesignLost response: applied, no answerStalled call: not applied, no answer
Lock across the call, roll back on errorRollback forgets an applied refund; the next one passes. Over the limit.Correct
Reserve and commit, call, release on errorSame. Over the limit.Correct
Reserve and commit, call, keep on errorCorrectCapacity stranded; refunds the customer is owed are refused indefinitely

Any design that resolves a timeout by itself is wrong in one of the two columns. Holding the lock across the call adds a second cost: provider latency becomes database contention, with a connection and a lock held for the provider’s timeout on every refund.

The correct design adds a third state: held until evidence resolves it. That state takes more than one extra column to build well.

The checklist for “held until resolved”

Use this as a review checklist for any service that triggers an external effect under a limit:

  1. Attempt identity. A retry is either the same attempt or a deliberately new one, never an accidental new one. The operation’s identity is separate from its transport attempts.
  2. No retry of an unknown before reconciliation. Otherwise the retry is the double refund. Idempotency keys do not cover this when the retry comes from another agent or run with a different key.
  3. Fencing. A worker that pauses after reserving (garbage collection, a partition, a slow pod) must not dispatch after recovery has given up on its attempt. Dispatch needs a one-time activation of that exact attempt.
  4. Reconciliation that never re-sends. Resolve by querying the provider, or by an attributed operator assertion that cites its evidence.
  5. Sealing. The first accepted definitive fact settles the attempt. A later contradictory report is an incident with evidence, not an edit to the balance.
  6. No TTL. Elapsed time does not prove an effect was not applied. Restarts, expiry and run completion do not release capacity.
  7. Evidence. Record who reserved, who asserted the outcome, on what evidence, and why a later call was refused.

This is a reservation (escrow) ledger, the structure behind card authorization holds. The real choice is not “row lock or gateway”. It is build this ledger into every service that triggers an external effect, or run it once at the tool boundary. For one service and one provider, building it into the service is reasonable.

The tool-boundary ledger is not an escape from transactions either. In the ContextOS runtime it is a compare-and-swap on a protected store, PostgreSQL in production. It sits in front of the dispatch and records the dispatch’s state in that transaction.

The same faults at the tool boundary

The example’s second part sends the same faults through the ContextOS gateway, with one guarantee over both the refund and the store-credit tools:

[[guarantees]]
id = "compensation-per-order"
capability_ids = ["adp_payments.issue_refund", "adp_loyalty.grant_store_credit"]
resource_pointer = "/order_id"
amount_pointer = "/amount_minor"
limit = 100000
unit = "INR_minor"
Part 2: the same faults through ContextOS, one guarantee over both tools
  lost response, ord_881                             unknown: INR 800 held, INR 200 available
  second agent, refund INR 800                       withheld before dispatch (shared_guarantee_exceeded)
  third agent, store credit INR 800                  withheld before dispatch: another system, one limit
  reconciled from the provider ledger                confirmed: INR 800 committed
  stalled call, ord_882                              held; reconciled not applied, released; retry refunds INR 500
  store credit INR 200                               applied; customer paid INR 1,000 of INR 1,000

The lost response stays held, so neither the second refund nor the store credit can spend the INR 800 that may already have left. The stalled call is also held, but because reconciliation establishes it was not applied, its capacity comes back and the owed refund goes through. The gateway’s signed audit log then verifies offline.

Limits that span systems, and tools that are not yours

Two more situations have no row to lock at all.

One limit, several systems. “Total compensation on an order may not exceed INR 1,000” covers refunds at the payment provider and store credit in the loyalty system. Each team can lock its own row; none can see the others’ rows. The options are a new compensation service every team must call first (the ledger above, now shared by several teams), a distributed transaction across teams and a provider (not available in practice), or one guarantee over all the capabilities, as in the configuration above.

Third-party tools. Agents increasingly call MCP servers and SaaS APIs directly. The provider still enforces its own invariants, such as refusing refunds above the captured amount; keep that, it is the strongest backstop. But “agents may refund at most INR 1,000 per order without a human” is your policy, not the provider’s, and the call path is the only place to enforce it.

Agents also change the retry model. After a timeout an agent often rephrases the call, starts a new run or hands the ticket to another agent. That produces a new idempotency key with the same business intent. A ceiling only helps if it counts unknown outcomes and charges the business resource, not the run or the key.

Decision table

SituationUse
The effect is a write to a database you own, and every caller goes through one serviceThe transaction. A gateway adds nothing to this limit.
One service you own calls one external provider, and you will build attempts, fencing and reconciliationBuild it in the service. A gateway is optional.
One limit spans several services or teamsOne guarantee over all of their capabilities at the tool boundary
Agents call third-party tools directlyA gateway on the call path; keep the provider’s own checks
Several agent frameworks or runtimes call the same toolsA gateway, so the rule lives in one place rather than in each framework

Run both when both apply. A service’s own invariant stays authoritative for its own data; the boundary ledger adds the cross-system limit that counts unknown outcomes.

What the tool-boundary ledger does not solve

  • Bypass. Calls that do not pass through the gateway are not counted. For a service’s own data, a transaction callers cannot bypass is strictly stronger.
  • Static ceilings. The limit is configured. It does not read captured or already-refunded amounts from the system of record, or count exposure that existed before deployment.
  • Hot resources serialize. Each resource has one ledger updated by compare-and-swap, so concurrent effects on one order contend as they would on a row lock. PostgreSQL contention has not been benchmarked yet.
  • One store is one enforcement domain. A replica on another store, or missing the guarantee, escapes the limit.
  • Finality is a trust assumption. Reconciliation trusts the provider’s answer or an authorized operator’s attributed assertion. The cited evidence is recorded, not independently fetched and checked.
  • Orphaned dispatches stay held. If a worker disappears after dispatch started and before it recorded an observation, the reservation cannot be released on an assertion of absence.

None of the techniques are new: O’Neil’s escrow transactional method (1986), sagas (Garcia-Molina and Salem, 1987), fencing tokens and card authorization holds. The work is packaging them at the tool boundary for agents that already exist.

Run it

With Python 3.9+ and Bash, from an unpacked v0.1.0-alpha.2 archive (macOS on Apple Silicon, or Linux with glibc 2.28+), or from a source checkout with the pinned Rust toolchain:

./examples/why-not-a-row-lock/run.sh

Then change one assumption at a time: lock the row across a slower provider call, add a third tool to the guarantee, or reconcile the lost response as not_applied and watch a later refund exceed what the provider actually paid. A boundary is only as good as the evidence used to release it.

Found this useful? Share it.

Share:XBSMRedditHNEmail

Continue through the same topic without returning to the index.

View the series