---
title: "Idempotency Stores & the Effect Guarantee"
description: "How the seen-set Store turns at-least-once delivery into an exactly-once effect (ADR-0022): the three-method seen / remember / forget contract every SDK's idempotency helper keys on meta.id, the in-memory reference store that ships in every core, and how to back it with a shared persistent store (Redis, a database table, a cache) for a fleet. The dedupe-key contract and the broker-free cross-SDK conformance fixtures that lock it. The wire envelope is untouched at schema_version 1."
source: https://babelqueue.com/docs/spec/1.x/idempotency-stores/
updated: 2026-06-21T00:00:00.000Z
---

# Idempotency Stores & the Effect Guarantee

The [Idempotency](/docs/spec/1.x/idempotency/) page covers the helper and the consumption
contract. This page is the companion deep-dive: the **`Store`** behind the helper, the
**exactly-once-effect** guarantee it buys you under at-least-once delivery, the
**dedup-key** contract, and the **store backends** — the in-memory reference every SDK
ships, and how to back it with a shared persistent store for a production fleet.

> **Status:** Authoritative · Helper layer · envelope frozen at `schema_version: 1`

Nothing here touches the wire. The envelope is frozen at `schema_version: 1`; `meta.id`
already carries the per-message identity the dedupe is keyed on. The store is a **consumer
choice** about how it treats that field, not a protocol change.

## At-least-once delivery → exactly-once effect

Every broker BabelQueue speaks is **at-least-once**: a message *can* arrive more than once —
a worker crashes after running the handler but before the ack, a broker redelivers on a
visibility-timeout lapse, a proxy double-sends, an operator replays a queue. The wire
contract makes this explicit and tells handlers to be idempotent (error-handling §1).

The idempotency helper mechanises that: it gives you **exactly-once *effect*** without
exactly-once *delivery*. Delivery stays at-least-once — the broker may still hand you the
same message many times — but the **side-effect** (the charge, the email, the row insert)
fires **once**. The store is what makes the difference: it remembers which `meta.id`s have
already produced their effect, so a redelivery is recognised and skipped.

This is **not** exactly-once delivery (no broker offers that without a transaction the queue
cannot see) and **not** an in-flight concurrency lock — two deliveries of the same id that
race *before the first completes* may both run. It is **seen-set, post-success dedupe**: the
honest, dependency-free guarantee that composes with retry and [dead-letter](/docs/spec/1.x/dead-letter-and-tracing/).

## The dedup-key contract

The dedupe key is **`meta.id`, verbatim** — the canonical, unique identity of *this specific
message*, minted by the first producer and carried unchanged across every hop. It is **not**
`trace_id`: `trace_id` spans a whole causal chain, so many distinct messages share one (see
[`id` vs `trace_id`](/docs/spec/1.x/envelope/#id-vs-trace_id--do-not-conflate)). Keying on
`trace_id` would wrongly collapse unrelated messages; keying on `meta.id` dedupes exactly the
redeliveries of one message.

Because `meta.id` is part of the frozen envelope, the key is **the same in every language**:
a payment a PHP or Python producer sent is de-duplicated by a Go, Java or Node consumer —
same envelope, same identity, one effect. A message with **no usable `meta.id`** (empty or
absent) cannot be deduped, so the helper **fails open**: it runs the handler unchanged rather
than silently dropping work it cannot identify.

## The `Store`

Every SDK exposes the same three-method record of processed ids, keyed on `meta.id`:

| Operation | Meaning |
| :--- | :--- |
| `seen(id)` | Has this id already been processed (remembered)? |
| `remember(id)` | Record this id as processed. |
| `forget(id)` | Drop an id — manual eviction / replay; a backend may also expire ids on its own TTL. |

The helper drives those three methods around your handler:

1. Read `meta.id`. No usable id → run the handler (fail-open) and return.
2. `seen(id)` is true → **skip**: do not invoke the handler; return so the runtime acks it
   and the broker stops redelivering.
3. Otherwise run the handler; **on success only**, `remember(id)`. A handler that throws
   leaves the id **unmarked**, so a redelivery runs it again — retry / dead-letter still
   apply. `forget(id)` later un-remembers an id so a deliberate replay runs once more.

> The store answers *"was this id processed?"* — never *"what did it return"*. Queue handlers
> have no response to replay (unlike an HTTP idempotency key), so a seen-set is all the
> contract needs.

## Store backends

### The in-memory reference (ships in every core)

Each SDK ships a reference **in-memory** store implementing the three methods over a
process-local set or map. It is the right choice for **tests** and a **single-process
consumer**, and it is what the [conformance fixtures](#cross-sdk-conformance) run against.

| SDK | Reference store |
| :--- | :--- |
| Go | `idempotency.NewInMemoryStore()` |
| PHP | `BabelQueue\Idempotency\InMemoryStore` |
| Python | `babelqueue.idempotency.InMemoryStore` |
| Node | `InMemoryStore` (`@babelqueue/core`) |
| Java | in-memory `Store` (`com.babelqueue.idempotency`) |

It is **process-local and not persistent**: two workers each hold their own set, neither
sees the other's "seen" ids, and a restart forgets everything. That is fine for one process;
it is **not** enough for a fleet.

### A shared persistent store

A production fleet needs a **shared** store so every worker dedupes against the same record —
and one that **survives a restart**. The interface is the extension point: implement the
same three methods (`seen` / `remember` / `forget`) over a shared backend and pass your store
to the helper unchanged — no other code changes, because the helper only ever calls those
three methods. You can still bring your own; **Go and PHP now also ship ready-made
persistent stores** so you usually don't have to:

| SDK | Persistent stores |
| :--- | :--- |
| Go | `idempotency-postgres` (`postgres.New(ctx, dsn)` / `postgres.NewWithDB(db)`) and `idempotency-redis` (`redis.New(url)` / `redis.NewWithClient(client)`) — **separate submodules**, so the core stays driver-free. Both take a `WithTable`/`WithPrefix` and a `WithTTL` option; Postgres ships a `Migrate(ctx)` DDL helper. |
| PHP | `BabelQueue\Idempotency\PdoStore` (`new PdoStore($pdo, $table)`, with `PdoStore::ddl()` for the portable `CREATE TABLE`) and `RedisStore` (`new RedisStore($client, $prefix)`, sharing the Predis client with `RedisTransport`). |
| Python · Node · Java · .NET | In-memory reference + **bring your own** behind the same interface (no persistent store ships yet). |

The backends follow the obvious shapes behind the same three methods:

- **Redis** — one key per id with a TTL. `seen` is `EXISTS`, `remember` is `SET` (a TTL caps
  unbounded growth), `forget` is `DEL`. Fast, shared across workers, naturally expiring.
- **A database table** — a `(message_id PRIMARY KEY, …)` row. `remember` is an `INSERT`; a
  unique-constraint conflict *is* the "already seen" signal. Durable, queryable.

> **Pick a TTL deliberately.** A seen-set grows forever without one. Size the TTL to your
> broker's maximum redelivery window — long enough that no legitimate redelivery outlives the
> remembered id, short enough to bound storage. `forget` is the manual escape hatch for a
> planned replay. The shipped stores expose a TTL option (`WithTTL` in Go; a TTL on the
> claim in PHP).

### Closing the in-flight race: the claim contract

The base seen-set is **post-success dedupe**: two deliveries of the same id that race *before
the first completes* may both run (it is not an in-flight lock — see the
[Idempotency](/docs/spec/1.x/idempotency/) page). The shipped persistent stores add an
**atomic claim** to close that window for callers who want it:

- **Go** — `postgres.Store` and `redis.Store` expose `Claim(ctx, id) (bool, error)`: an atomic
  `INSERT … ON CONFLICT DO NOTHING` (Postgres) or `SET … NX PX` (Redis). The winner gets
  `true`; a concurrent duplicate gets `false` and parks.
- **PHP** — the `BabelQueue\Idempotency\ClaimingStore` interface **extends** `IdempotencyStore`
  with `claim($id, $ttlSeconds): bool` and `release($id): void`; `PdoStore` and `RedisStore`
  implement it. `claim` wins the race atomically and marks the id in-flight (TTL'd); `release`
  drops an *uncommitted* claim if the handler errors, while a committed `seen` mark is
  permanent.

This is an opt-in upgrade over the three-method seen-set, not a replacement: a store that
implements only `seen` / `remember` / `forget` still works with the helper unchanged.

### The narrow at-least-once window

Whatever the backend, there is one documented gap: the handler succeeds, then `remember`
fails (or the worker dies) **before** the id is recorded. A later redelivery then reprocesses.
This is bounded and acceptable under "handlers are idempotent" — a redelivery simply re-runs
harmlessly. Closing it entirely requires writing the dedupe record **inside the handler's own
database transaction** — the **transactional** mode (ADR-0022), the consumer-side sibling of
the [transactional outbox](/docs/spec/1.x/outbox/) on the produce side: the PHP `PdoStore` /
Go `postgres.Store` above are the stepping stone, since a DB-backed `remember` can run on the
same connection as the handler's write. The seen-set store is the dependency-free default; the
transactional store is the opt-in upgrade for callers who run their writes through a DB the
store can join.

## Cross-SDK conformance

The dedupe contract is locked by the **broker-free** cross-SDK
[conformance suite](https://github.com/BabelQueue/conformance) — the `idempotency` block in
`manifest.json`, vendored into every core SDK. It is transport-agnostic: pure envelope +
expected-outcome assertions over a single in-process store, so every SDK's helper is proven
to make the same decision.

- **`dedup_key`** — the key is `meta.id` verbatim; two deliveries with the same `meta.id`
  collapse to one effect, distinct ids never collapse.
- **`sequences`** — ordered delivery scenarios, each asserting whether a delivery **runs** the
  handler or is **skipped**, and the total side-effects after the sequence:
  - a duplicate delivery runs **once**;
  - an at-least-once redelivery storm is a **no-op**;
  - distinct ids each run;
  - a **throwing** handler leaves the id unmarked, so a later redelivery runs it (retry
    survives the guard);
  - a missing `meta.id` **fails open** (runs every time);
  - `forget` allows a re-run.

Because the assertions are the *decision* (run vs. skip) over the in-memory reference store,
the block is **store-agnostic** — a persistent backend that honours the same three-method
contract passes it unchanged.

## Per-SDK entry points

The helper is optional and identical in spirit everywhere — same `Store` shape, same seen-set
semantics, same fail-open on a missing `meta.id`.

| SDK | Entry point | Reference store |
| :--- | :--- | :--- |
| Go | `idempotency.Wrap(store, handler)` | `idempotency.NewInMemoryStore()` |
| PHP | `Idempotent::wrap($store, $handler)` | `BabelQueue\Idempotency\InMemoryStore` |
| Python | `idempotency.wrap(store, handler)` | `babelqueue.idempotency.InMemoryStore` |
| Node | `Wrap(store, handler)` | `InMemoryStore` (`@babelqueue/core`) |
| Java | `Idempotent.wrap(store, handler)` | in-memory `Store` (`com.babelqueue.idempotency`) |
| .NET | `Idempotency.Wrap(store, handler)` | `InMemoryStore` (`IIdempotencyStore`, `BabelQueue`) |

For a runnable, broker-free walkthrough (no Redis, no Docker) see the
[in-memory idempotency example](https://github.com/BabelQueue/babelqueue-examples/tree/main/idempotency-inmemory);
for the cross-language Python → Go version over Redis, the
[idempotency-payments example](https://github.com/BabelQueue/babelqueue-examples/tree/main/idempotency-payments).

This guard stops an **accidental** duplicate from re-running an effect. To stop an **intended**
replay (off a DLQ) from re-firing effects that already ran, see
[DLQ redrive & replay-bypass](/docs/spec/1.x/redrive-and-replay/) — the complementary primitive.
The producer-side sibling that makes the message *commit* once is the
[Transactional Outbox](/docs/spec/1.x/outbox/). To keep one trace across the deduped hop, see
[Observability (OpenTelemetry)](/docs/spec/1.x/observability/).

Back to the [Idempotency contract](/docs/spec/1.x/idempotency/).