Dead-letter & tracing

Last updated:

Two cross-cutting behaviours every SDK shares, both designed so a failure or a trace crosses language boundaries unchanged: dead-lettering (where a message goes when it can’t be processed) and trace propagation (how one correlation id ties a whole chain of work together). Both are part of the wire contract and neither touches the frozen schema_version: 1.

Dead-letter queue: <queue>.dlq

When a message exhausts its retries, or arrives with a URN no consumer handles (and the unknown-URN strategy is dead_letter), it is republished to a dead-letter queue rather than dropped. The convention is the source queue name plus a .dlq suffix — a message that failed on orders lands on orders.dlq. Because the dead-lettered message is still the canonical envelope, any SDK can read the DLQ and triage it, regardless of which language produced or failed the message.

The default suffix is .dlq across the ecosystem (e.g. Laravel’s dead_letter.suffix, Go’s WithDeadLetterSuffix); Amazon SQS is the exception, because an SQS queue name cannot contain a dot: there the convention is -dlq (orders-dlq; FIFO: orders-dlq.fifo), and the Go, Python and Laravel SQS transports use it by default (an explicit suffix or queue name still wins; on a FIFO queue a suffix goes before .fifo, so orders.fifo + -dlq is orders-dlq.fifo). Dead-lettering is opt-in: enable it per SDK (dead_letter: true in Python, WithDeadLetter(true) in Go, maxTries + the DLQ path in the Node transports, the dead_letter block in Laravel config). With it disabled, a terminally failed message is dropped instead.

Retry, dead-letter queue and redrive A work queue, a consumer and a dead-letter queue joined by arrows, with a redrive arrow back to the work queue. The caption below gives the full description. work queue orders attempts: 0 consumer handler routes on the URN dead-letter queue orders.dlq + dead_letter block deliver retries exhausted on failure: retry, attempts + 1 Redrive drop dead_letter, reset attempts to 0, keep job · trace_id · data · meta back to original_queue, or to a sandbox queue
Failure handling end to end. A consumer takes a message from the work queue; on failure it is retried and attempts goes up. When retries are exhausted the envelope is published to <queue>.dlq (for example orders.dlq) with an additive dead_letter block. Redrive later drains the DLQ, drops the dead_letter block, resets attempts to 0, keeps job, trace_id, data and meta verbatim, and publishes the message back to its original queue or to a sandbox queue. Retries are not the only way in: a message with an unknown URN (when on_unknown_urn is dead_letter) or an undecodable poison body goes to the DLQ without being retried. A poison body is a body that does not decode into an envelope, including a JSON object with neither a valid job nor a valid urn (an identity-less object is poison, not an unknown URN). Where it is dead-lettered it is recorded with its original bytes kept, as a poison record with reason poison. The rule differs a little by transport. On Redis it holds in every configuration: the poison body is dead-lettered whatever on_unknown_urn says and whether or not dead-lettering is on (Go, Python, Java, .NET and Node always, even with Node's deadLetterQueue: null; Laravel when its DLQ is enabled, and into failed_jobs when it is not). On Kafka it holds in Java, .NET, Node and PHP when a DLQ topic is configured; with none configured, and in Python when it has no retry tiers, the record is committed, dropped and logged so the partition advances. On Artemis, SQS and Azure Service Bus the Java, .NET and Node consumers dead-letter it at its first delivery when a DLQ is configured, and with none configured Artemis drops it while SQS and Azure Service Bus leave it to the broker's own dead-letter mechanism. In the Go and Python runtimes on their other transports, in the Java, .NET and Node Pulsar consumers and in the PHP Consume\Dispatcher, poison is dead-lettered without retry only when dead-lettering is on and on_unknown_urn is not delete; with dead-lettering off it takes the bounded retry and is then dropped. The one exception is the Go and Python runtimes on SQS and Azure Service Bus: with dead-lettering off they do not retry or drop it but hand it to the broker's native mechanism on its first delivery, like the Java, .NET and Node consumers do there. The Node BullMQ adapter fails a poison job without retry and, if you give it a deadLetterQueue, also records it there. One pair still forwards a poison body raw: Node RabbitMQ, listed as a known gap in the error-handling contract. On Kafka, when retry topics are configured, retries run through SDK-owned retry topics rather than back onto the work queue.

The additive dead_letter block

A message sitting on a DLQ carries one extra, optional top-level field, dead_letter, describing why it failed. It appears only on DLQ messages — never on a normal queue — and is purely additive, so the envelope stays frozen at schema_version: 1. Normal consumers ignore it; a triage tool reads it.

{
  "job": "urn:babel:orders:created",
  "trace_id": "7b3f9c2a-e41d-4f88-9b2a-1c0d5e6f7a8b",
  "data": { "order_id": 1042 },
  "meta": {
    "id": "f1e2d3c4-b5a6-4789-90ab-cdef01234567",
    "queue": "orders",
    "lang": "php",
    "schema_version": 1,
    "created_at": 1749132727000
  },
  "attempts": 3,
  "dead_letter": {
    "reason": "failed",
    "error": "Payment gateway timeout",
    "exception": "App\\Exceptions\\GatewayTimeout",
    "failed_at": 1749132730000,
    "original_queue": "orders",
    "attempts": 3,
    "lang": "php"
  }
}

The original envelope is preserved unchanged inside the dead-lettered message — job, trace_id, data, meta and attempts are exactly what they were, so the DLQ message is a faithful record of what failed and where it came from (original_queue). Every SDK exposes a helper that wraps an envelope in this block and returns a copy, leaving the input untouched:

SDK Helper
PHP BabelQueue\DeadLetter\DeadLetter::annotate($envelope, $reason, $e, $originalQueue, $attempts) + DeadLetterPublisher (core)
Laravel the dead_letter policy in config/babelqueue.php (driver applies it)
Python dead_letter=True runtime option (runtime applies it)
Go babelqueue.Annotate(env, reason, queue, attempts, err)
Node.js annotate(env, reason, queue, { attempts, error }) from @babelqueue/core
Java DeadLetters.annotate(env, reason, queue, attempts, error, exception)
.NET DeadLetters.Annotate(env, reason, queue, attempts: …, error: …)

dead_letter keys are descriptive, not contract envelope fields. This block is a diagnostic side-channel that exists only on the DLQ. Do not confuse it with the frozen envelope — and note the forbidden top-level/meta fields (timestamp, meta.max_retries, meta.source, meta.ts) still never appear, on the DLQ or anywhere else. See forbidden fields.

Not every DLQ record is a canonical envelope

A reader of <queue>.dlq must not assume the dead_letter block is there. Three shapes can share a DLQ, and a triage tool tells them apart without guessing:

Record Written when Shape
Failure record retries exhausted, or an unknown URN under on_unknown_urn: dead_letter the canonical envelope plus the dead_letter block (reason failed or unknown_urn)
Poison record the body does not decode into an envelope, or has neither a valid job nor a valid urn a JSON object body with a dead_letter member added (reason: "poison"); anything else wrapped under raw, as base64 plus raw_encoding when it is not valid UTF-8
Raw version-gate record a delivery whose schema-version property (bq-schema-version, Artemis bq_schema_version) is not the supported version the incoming body exactly as received, byte for byte, with no dead_letter block — a reader recognises it by that absence

A version-gated message is never decoded, so there is nothing to wrap: the raw body is what a future, compatible consumer or a redrive tool needs. It is written on Pulsar, Artemis, SQS and Azure Service Bus, on the first delivery, and only when the BabelQueue DLQ is on (or the dead_letter strategy has a writable DLQ). The original is acknowledged after the DLQ write succeeds; a failed write leaves it unsettled, so it is redelivered and the write is retried.

One exception is documented, not accidental. With the BabelQueue DLQ disabled, Laravel’s babelqueue-* connections fail a poison body or an exhausted job into Laravel’s own failed_jobs table. The Laravel SQS and Artemis drivers also keep their own version-gate path: with the DLQ enabled they publish a version-gated delivery as a stub envelope with a dead_letter block (reason: "poison"), not as the raw body, so a reader must not expect the original bytes there; with it disabled they throw, and Laravel’s retry and then failed_jobs take it.

Broker-native dead-lettering

Some brokers have a native dead-letter mechanism, and the bindings use it alongside the cross-language <queue>.dlq convention rather than instead of it:

  • Redis / RabbitMQ / Go’s runtime — the SDK republishes the wrapped envelope to <queue>.dlq.
  • Apache Kafka — no native DLQ, so the SDK owns it: terminal failures go to <topic>.dlq with the dead_letter block (after the tiered <topic>.retry.<n> topics). The consumed record’s key is carried onto the DLQ record. See the Kafka binding.
  • Apache Pulsar — a native DeadLetterPolicy exists, but it is a client-side setting the SDKs do not apply: the bound is the SDK’s own max-attempts (default 3). When it is used up the SDK writes the cross-language <queue>.dlq (opt-in; with dead-lettering off the message is acknowledged, dropped and logged), so other SDKs can triage. That holds for the Go and Python runtimes, the Java, .NET and Node consumers (dead_letter.attempts is the counted value, 3 with the default) and the PHP Consume\Dispatcher given a DeadLetterPublisher; the Dispatcher writes <queue>.dlq with the queue taken from the consumer’s topic and never default.dlq. A poison body is recorded there without retry. A retry is a re-publish with bq-attempts + 1, so the cap holds on every subscription type. The bare PHP PulsarConsumer leaves it to the handler. A message with an unsupported bq-schema-version is settled on its first delivery, never retried: acknowledged and dropped without a DLQ, otherwise written to <queue>.dlq as the raw body. See the Pulsar binding.
  • ActiveMQ Artemis — a native dead-letter address exists, and the binding also writes the cross-language <queue>.dlq so other SDKs can triage. A poison body is recorded there as the poison record (Java, .NET and Node used to forward the raw bytes), and a message with an unsupported bq_schema_version is written as the raw body; with no DLQ configured both are acknowledged, dropped and logged.
  • Amazon SQS / Azure Service Bus — the SDK-driven dead-letter sends the annotated envelope to the cross-language DLQ (opt-in): <queue>.dlq on Azure Service Bus and <queue>-dlq on SQS (FIFO: <queue>-dlq.fifo), because an SQS queue name cannot contain a dot. The broker’s own facility (SQS RedrivePolicy, ASB $DeadLetterQueue) is a complementary safety net you configure on the broker: it moves the message verbatim, so it carries no dead_letter block. The envelope on them is still canonical. When the BabelQueue DLQ is configured, a poison body (as the poison record) and a version-gated message (as the raw body) go to that DLQ on their first delivery. When it is off, the broker’s facility is the settle for those two cases: the SDK does not delete the SQS message (the RedrivePolicy takes it after maxReceiveCount) and Azure Service Bus abandons it until MaxDeliveryCount or dead-letters it natively. Nothing is silently dropped.

See the broker bindings for the per-broker detail.

Trace propagation (trace_id)

trace_id is a required, immutable id that the first producer in a chain mints (a v4 UUID), and that every SDK then preserves and forwards unchanged across every hop — the fourth golden rule (GR-4). It is how you follow one unit of work as it fans out across services and languages.

Producers mint only UUIDs, but a trace_id that is not a UUID (from a non-BabelQueue producer or a hand-built message) is carried on exactly the same way: preserved byte for byte, never rejected, never regenerated. See a trace_id that is not a UUID.

The rule, straight from the consumer rules: when a consumer produces a downstream message while handling one, it copies the inbound trace_id onto the new envelope and mints a new meta.id. Do not conflate the two — see id vs trace_id:

meta.id trace_id
Scope one message one causal chain (may span many messages/services)
Minted by the producing SDK, per message the first producer; reused thereafter
Lifetime dies with the message propagates across every hop

Continuing a trace, per SDK

Every producer API takes an optional inbound trace id; pass it to continue the chain, omit it to start a fresh one:

# Python — reuse the inbound trace_id on a downstream message
app.publish("urn:babel:shipping:requested", {"order_id": 1042}, trace_id=inbound["trace_id"])
// Go — WithTraceID continues the trace; omit it to mint a new one
env, _ := babelqueue.Make(
    "urn:babel:shipping:requested",
    map[string]any{"order_id": 1042},
    babelqueue.WithTraceID(inbound.TraceID),
)
// Node — pass traceId in the options
const env = EnvelopeCodec.make(
  "urn:babel:shipping:requested",
  { order_id: 1042 },
  { traceId: inbound.trace_id },
);
// Java — pass the inbound trace id (or null to mint a fresh one)
Envelope env = EnvelopeCodec.make(
    "urn:babel:shipping:requested", Map.of("order_id", 1042L), "orders", inbound.traceId());
// .NET — pass traceId to continue the trace
var env = EnvelopeCodec.Make(
    "urn:babel:shipping:requested",
    new Dictionary<string, object?> { ["order_id"] = 1042L },
    queue: "orders", traceId: inbound.TraceId);

The framework adapters automate this so you never plumb it by hand: Laravel hands the handler the $traceId and reuses it; Symfony’s babelqueue.messenger.trace_middleware attaches a BabelTraceStamp so any message a handler dispatches inherits the inbound trace_id.

Tracing through the DLQ

Because the dead-lettered message preserves the original envelope unchanged, trace_id survives dead-lettering too. A message that failed three hops into a chain still carries the chain’s trace_id on orders.dlq, so you can correlate the failure back to the original producer and every service that touched it — in any language.

To count dead-letters rather than trace them, every SDK can feed the babelqueue.dlq.messages metric from the same write points; see What babelqueue.dlq.messages counts. To find the failing chain in your application logs, see Log correlation.

In one line

A failure becomes a canonical envelope on <queue>.dlq with an additive dead_letter block any SDK can read, and one trace_id — minted once, forwarded unchanged — ties the whole chain together, DLQ included. Neither mechanism changes the frozen schema_version: 1.

Back to Broker bindings · or the broker setup recipes.