Back

Outbox Pattern: Reliable Event Publishing in Go

Walk through the outbox pattern for event-driven systems and the problems it solves. Includes a practical implementation in Go using Postgres and RabbitMQ.

George Lopez
Outbox Go Architecture

In this article, we walk through the outbox pattern for event-driven systems and the problems it solves. We also provide a practical implementation written in Go, using Postgres and RabbitMQ.

GitHub repository: https://github.com/georgelopez7/outbox-go

The Dual-Write Problem

The outbox pattern addresses a fundamental challenge in event-driven architectures, the dual-write problem. This occurs when a system needs to perform two writes as part of a single operation, typically writing to a database and publishing a message to a queue.

In this implementation, we demonstrate how to create a user and reliably publish the corresponding event to RabbitMQ.

The Outbox Pattern Solution

The outbox pattern solves this by making both writes part of a single database transaction.

Instead of publishing directly to the message broker, the service writes the event to an outbox_events table in the same transaction as the business data:

BEGIN
  INSERT INTO users (id, name) VALUES ($1, $2);
  INSERT INTO outbox_events (id, event, status) VALUES ($3, $4, 'pending');
COMMIT

Because both writes are atomic, either both succeed or both fail. There is no window for inconsistency.

A separate background process called the relay then reads pending events from the outbox table and publishes them to RabbitMQ.

Architecture

This project consists of two independent microservices that share a RabbitMQ instance:

  • gateway-api: HTTP API producer, writes to users & outbox_events tables
  • event-consumer: RabbitMQ consumer, processes events with idempotency

Each service has its own database to reflect realistic microservice boundaries where each service owns its own data.

The Relay

The relay is a background goroutine that polls the outbox_events table for pending messages and publishes them to RabbitMQ and then marks them as processed. A snippet can be seen below:

func (r *Relay) Start() {
    go func() {
        ticker := time.NewTicker(1 * time.Second)
        defer ticker.Stop()
 
        for range ticker.C {
            events, err := r.Service.OutboxService.GetPendingEvents()
            if err != nil {
                slog.Error("failed to get pending events", "err", err)
                continue
            }
 
            for _, e := range events {
                if err := r.Service.OutboxService.PublishEvent(e); err != nil {
                    slog.Error("failed to publish event", "id", e.ID, "err", err)
                    continue
                }
 
                if err := r.Service.OutboxService.MarkAsProcessed(e.ID); err != nil {
                    slog.Error("failed to mark event as processed", "id", e.ID, "err", err)
                    continue
                }
            }
        }
    }()
}

The relay queries outbox events where status = pending, using a partial index for efficient lookups:

CREATE INDEX idx_outbox_status_created ON outbox_events (status, created_at)
WHERE status = 'pending';

Tradeoffs & At-Least-Once Delivery

Benefits

  • Eliminates dual-write inconsistency. The business write and event recording are atomic.
  • Simple to implement. No distributed transactions or two-phase commit.
  • Broker-agnostic. Works with any message broker since the outbox is just a database table.
  • Natural event sourcing. The outbox table serves as an append-only log of all domain events

Tradeoffs

  • Polling latency. Events are not published instantly. The 1-second polling interval introduces up to 1s of delay.
  • Duplicate publishes. If the relay crashes after sending the event but before marking the event as processed, the same event may be published again on the next poll cycle. This is not an issue as our consumers handle events idempotently.
  • Storage overhead. The outbox_events table grows with every event and needs periodic cleanup of processed records.

At-Least-Once Delivery

The outbox pattern provides at-least-once delivery. Events will be delivered one or more times, but never zero times.

This means consumers must handle events idempotently. They must be able to process the same event multiple times without producing duplicate side effects.

Consumer Idempotency

The consumer uses an inbox_events table to deduplicate incoming messages:

CREATE TABLE inbox_events (
    id         UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    event_id   UUID NOT NULL UNIQUE,   -- unique constraint is the key
    event_type TEXT NOT NULL,
    created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

When a message arrives, the consumer:

  1. Opens a database transaction
  2. Inserts a record into inbox_events using ON CONFLICT (event_id) DO NOTHING
  3. If rowsAffected == 0, the event was already processed and therefore we log and return success (ACK)
  4. If first time, continue with business logic
  5. Commit the transaction and ACK the message

This protects against:

  • RabbitMQ redelivering messages after a consumer crash
  • Manual requeues during debugging
  • Network partitions causing duplicate deliveries

RabbitMQ Quorum Queues & Poisoned Messages

In this project the main event queue is configured as a quorum queue for high availability and data safety:

ch.QueueDeclare("event.queue", true, false, false, false, amqp.Table{
    "x-queue-type":              "quorum",
    "x-delivery-limit":          int32(3),
    "x-dead-letter-exchange":    "",
    "x-dead-letter-routing-key": "event.queue.dlq",
})

Quorum Queues

Quorum queues use the Raft consensus algorithm to replicate data across RabbitMQ nodes, providing:

  • Data safety. Messages are not lost if a node fails.
  • High availability. The queue remains available during node failures.

Automatic Poisoned Message Handling

The x-delivery-limit = 3 setting is critical.

When a consumer Nacks a message, RabbitMQ requeues it for redelivery. Without a delivery limit, a poisoned message (one that always fails processing) would cause an infinite retry loop.

With x-delivery-limit set to 3, RabbitMQ tracks how many times each message has been delivered. After 3 failed attempts, the message is automatically dead-lettered. No infinite loops, no manual intervention needed.

Dead Letter Queue

Messages that exceed the delivery limit are routed to a dead letter queue:

ch.QueueDeclare("event.queue.dlq", true, false, false, false, nil)

The DLQ has no further dead-letter configuration. Messages are parked there permanently for manual inspection and debugging. This is useful for:

  • Investigating why messages consistently fail processing
  • Recovering messages after fixing a bug
  • Monitoring error rates and patterns

Conclusion

The outbox pattern is a simple yet powerful solution to one of the most common pitfalls in event-driven systems, the dual-write problem. By leveraging database transactions and a relay process, it ensures reliable event publishing without requiring complex distributed coordination.

While it introduces trade-offs such as polling latency and potential duplicate delivery, these are manageable with well-understood patterns like idempotent consumers and dead letter queues.

You can find the full implementation of the outbox pattern on Github: https://github.com/georgelopez7/outbox-go