Idempotency Keys in Go SaaS: Preventing Duplicate Operations at Scale

When a mobile client retries a payment request because the network dropped, or a webhook fires twice because your delivery system has at-least-once semantics, idempotency is the only thing standing between your database and a corrupted billing record. This is how we implement idempotency keys across Go SaaS backends handling payments, order creation, and any operation that must complete exactly once regardless of how many times the client tries.

Why do distributed systems need idempotency beyond unique constraints?

The standard answer to duplicate prevention is a database unique constraint on some natural key. Unique constraints are necessary but not sufficient for SaaS operations that span multiple systems.

Consider a payment flow that creates an order, charges a card via a payment gateway, and updates the subscription record. If the card charge succeeds but the network drops before the response reaches the client, the client retries the whole operation. A unique constraint on order ID prevents the duplicate order, but the payment gateway has no way to know the original charge succeeded. Without idempotency at the API level, the second retry charges the card again.

Idempotency keys solve this by caching the original response for a given operation and returning it verbatim on subsequent requests with the same key, without re-executing the operation. The key is generated client-side (typically a UUID v4), sent in a request header, and stored server-side paired with the response.

In SaaS platforms serving Lebanese and Gulf e-commerce clients, where intermittent connectivity is a reality, this pattern prevents double-charges, duplicate shipments, and duplicate invoice records. It is not optional for any billing-adjacent operation.

What does the database schema for idempotency tracking look like?

The idempotency store needs to hold enough information to reconstruct the original response without re-running the operation:

CREATE TABLE idempotency_keys (
    key           TEXT        NOT NULL PRIMARY KEY,
    tenant_id     UUID        NOT NULL,
    endpoint      TEXT        NOT NULL,
    response_code INT         NOT NULL,
    response_body JSONB       NOT NULL,
    created_at    TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    expires_at    TIMESTAMPTZ NOT NULL
);

CREATE INDEX idx_idempotency_tenant ON idempotency_keys (tenant_id, expires_at);

The primary key is the raw client-provided key, not a server-generated ID. The client-provided key is the lookup mechanism, so it must be the primary key for O(1) lookup.

The endpoint column scopes the key to a specific operation. A key generated for a payment endpoint should not match a key generated for an order endpoint, even if the client accidentally reuses the same UUID.

The expires_at column drives cleanup. Idempotency keys are not permanent records. Keeping them for 24 hours covers all realistic retry scenarios without wasting storage.

How do you implement this in Go middleware?

The idempotency check lives in HTTP middleware that runs before any handler logic. The middleware handles three cases: no key provided (pass through), key seen before (return cached response), and key not seen (record, execute, cache).

type IdempotencyStore interface {
    Get(ctx context.Context, key, tenantID, endpoint string) (*CachedResponse, error)
    Set(ctx context.Context, key, tenantID, endpoint string, resp *CachedResponse, ttl time.Duration) error
}

func IdempotencyMiddleware(store IdempotencyStore, ttl time.Duration) func(http.Handler) http.Handler {
    return func(next http.Handler) http.Handler {
        return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
            key := r.Header.Get("Idempotency-Key")
            if key == "" {
                next.ServeHTTP(w, r)
                return
            }

            tenantID := TenantFromContext(r.Context())
            endpoint := r.Method + ":" + r.URL.Path

            cached, err := store.Get(r.Context(), key, tenantID, endpoint)
            if err == nil && cached != nil {
                w.Header().Set("Idempotency-Replayed", "true")
                w.WriteHeader(cached.StatusCode)
                w.Write(cached.Body)
                return
            }

            rec := newResponseRecorder(w)
            next.ServeHTTP(rec, r)

            // Only cache non-5xx responses
            // Server errors mean the operation may not have completed
            if rec.code < 500 {
                store.Set(r.Context(), key, tenantID, endpoint, &CachedResponse{
                    StatusCode: rec.code,
                    Body:       rec.body.Bytes(),
                }, ttl)
            }
        })
    }
}

The Idempotency-Replayed: true header tells clients that the response is a cached replay. Useful for debugging when a client suspects a retry scenario.

Not caching 5xx responses is deliberate. A server error means the operation may not have completed, so the client should retry and allow fresh execution.

How do you handle concurrent requests with the same key?

The failure scenario: a poorly-written mobile client fires two identical requests with the same idempotency key 50ms apart. Both arrive before either finishes executing. The first request creates the order and charges the card. The second request, if it passes the idempotency check before the first has written its cached response, will also attempt execution.

The fix is a database-level lock using the idempotency key row. The first request inserts the key row and holds a row lock during execution. The second request tries to insert the same key and blocks. When the first request commits its cached response, the second request reads the cached value and returns it.

func (s *PGIdempotencyStore) Acquire(ctx context.Context, key, tenantID, endpoint string) (acquired bool, cached *CachedResponse, err error) {
    tx, err := s.db.BeginTx(ctx, nil)
    if err != nil {
        return false, nil, err
    }

    // Try to insert a placeholder row
    _, err = tx.ExecContext(ctx,
        `INSERT INTO idempotency_keys (key, tenant_id, endpoint, response_code, response_body, expires_at)
         VALUES ($1, $2, $3, 0, 'null', NOW() + $4::interval)
         ON CONFLICT (key) DO NOTHING`,
        key, tenantID, endpoint, s.ttl.String(),
    )
    if err != nil {
        tx.Rollback()
        return false, nil, err
    }

    // Check if we inserted (acquired) or another request got there first
    var code int
    var body []byte
    err = tx.QueryRowContext(ctx,
        `SELECT response_code, response_body FROM idempotency_keys
         WHERE key = $1 AND response_code != 0
         FOR UPDATE`,
        key,
    ).Scan(&code, &body)

    if err == sql.ErrNoRows {
        // We acquired the lock, proceed with execution
        return true, nil, nil
    }
    tx.Rollback()
    return false, &CachedResponse{StatusCode: code, Body: body}, nil
}

This pattern serializes concurrent duplicates using the database, which is already the source of truth for idempotency state.

What TTL and cleanup policy should you use?

Set the TTL based on your clients' realistic retry window. A mobile client retrying a payment up to three times over 30 seconds needs a 15-minute TTL. A webhook delivery system retrying over 24 hours needs a TTL that matches.

Cleanup is a background job running hourly:

func (s *PGIdempotencyStore) Cleanup(ctx context.Context) (int64, error) {
    result, err := s.db.ExecContext(ctx,
        `DELETE FROM idempotency_keys WHERE expires_at < NOW()`,
    )
    if err != nil {
        return 0, err
    }
    return result.RowsAffected()
}

Partition the table by expires_at range if your platform handles high enough volume that hourly cleanup deletes tens of thousands of rows. A non-partitioned table with an index on expires_at handles several million rows comfortably before the DELETE performance degrades.

Key lessons from production

Do not scope idempotency keys only to payments. Any operation that creates records, sends notifications, or modifies external systems benefits from idempotency. Subscription creation, team member invitations, and webhook registrations all cause real user pain when they execute twice.

Log every replayed response with tenant ID and the original request metadata. A payment that replayed three times in five minutes signals a broken client integration, and you want visibility on that before the customer calls your support line.

Test the concurrent duplicate scenario explicitly. It is the one developers skip in testing, and also the one that causes production incidents. Run two goroutines simultaneously with the same idempotency key and verify only one execution occurs.