Idempotency Patterns — Create Orders Exactly Once (With Retries)

Git Repo: https://github.com/mnafshin/idempotency

Why This Matters

Retries are a fact of life in distributed systems. A mobile client times out, a load balancer gives up, a service restarts — and the same request arrives twice. Without idempotency, that can mean duplicate orders and duplicate charges.

This repository walks you through three runnable, progressively richer implementations so you can teach the pattern, validate it in tests, and eventually ship it in production:

Module	Teaches	Audience
`idempotency-easy` (port 8081)	The core concept — dedup by key	New to idempotency
`idempotency-mature` (port 8082)	Multi-tenant + payload consistency check	Building multi-tenant APIs
`idempotency-production-redis` (port 8083)	Filter-based cross-cutting concern + pluggable store (in-memory / Redis)	Production services

The Idempotency Contract

A client picks a random, unique Idempotency-Key per logical operation and sends it as a header.

POST /api/orders
Idempotency-Key: 550e8400-e29b-41d4-a716-446655440000
X-Tenant-ID: 2b8de313-9c3c-4a15-a9b8-0cd1e34be3da

Rules:

First call → business logic runs, response is 201 Created.
Retry with the same key + same payload → original response is replayed, 200 OK, Idempotent-Replayed: true header set. No business logic runs again.
Retry with the same key + different payload → 409 Conflict. Clients can't silently change the intent of an in-flight operation.
Different key → treated as a new, independent operation.

Level 1 — Easy Mode (Teaching)

Module: idempotency-easy
Goal: explain idempotency in the fewest moving parts.

What it does

POST /api/orders
Idempotency-Key: order-abc

First request → new order, 201 Created.
Same key → same order returned, 200 OK.
No tenant scope, no payload check — just key dedup.

How it works

EasyOrderService holds a ConcurrentHashMap<String, Order>. On each request it calls putIfAbsent:

Order existing = orders.putIfAbsent(idempotencyKey, newOrder);
boolean replayed = existing != null;
return new OrderResult(replayed ? existing : newOrder, replayed);

The controller computes the HTTP status from the result:

return ResponseEntity
    .status(result.replayed() ? HttpStatus.OK : HttpStatus.CREATED)
    .body(result.order());

Limitations (and why they matter in the blog)

Stored in HashMap → lost on restart, not shared across instances.
No tenant isolation → two tenants using the same key would collide.
No payload check → a client that accidentally changes the request body gets a silent replay of the wrong data.

These are exactly the gaps Level 2 closes.

Level 2 — Mature Mode (Multi-Tenant + Payload Safety)

Module: idempotency-mature
Goal: production-oriented semantics without external infrastructure.

Required headers

X-Tenant-ID: 2b8de313-9c3c-4a15-a9b8-0cd1e34be3da   # UUID
Idempotency-Key: order-abc                             # max 255 chars

What changes

Tenant-scoped dedup key

String dedupKey = tenantId + ":" + idempotencyKey;

Two tenants using order-abc get completely separate records.

Payload fingerprint

String fingerprint = customerId + ":" + amount;

This is stored alongside the order. On retry, fingerprints are compared before replaying:

if (!stored.requestFingerprint().equals(incomingFingerprint)) {
    throw new IdempotencyConflictException(...);
}

409 Conflict is returned when payloads differ — protecting clients from subtle bugs where a retry accidentally sends different data.

Race-safe `putIfAbsent`

ConcurrentHashMap.compute() ensures only one thread wins the first write, regardless of how many concurrent retries arrive simultaneously.

Behavior table

Scenario	Response
First call	`201 Created`
Retry, same tenant + key + payload	`200 OK` (replay)
Retry, same tenant + key, different payload	`409 Conflict`
Same key, different tenant	Independent — both succeed

Level 3 — Production Mode (Filter-Based + Pluggable Store)

Module: idempotency-production-redis
Goal: cross-cutting idempotency at the HTTP layer, switchable between in-memory and Redis.

The key design insight

In mature mode, the service owns idempotency. In production mode, that responsibility moves to a servlet filter (IdempotencyFilter extends OncePerRequestFilter).

Why? Because idempotency is not business logic — it's a protocol guarantee. The service shouldn't care whether the response it created 24 hours ago is being replayed right now. With the filter in place:

IdempotencyFilter → [cache hit?] → replay stored response (service never called)
                  → [cache miss] → acquire lock → ProductionOrderController → ProductionOrderService
                                                 ↓
                                            store response in HttpReplayStore
                                            release lock

ProductionOrderService becomes a plain order factory:

// No idempotency logic here — the filter ensures this is called exactly once per key
public Order createOrder(String tenantId, String idempotencyKey, CreateOrderRequest request) {
    return new Order(UUID.randomUUID(), UUID.fromString(tenantId),
                     idempotencyKey, request.customerId(), request.amount(),
                     "pending", Instant.now());
}

IdempotencyFilter — step by step

┌─────────────────────────────────────────────────────┐
│  IdempotencyFilter.doFilterInternal()               │
│                                                     │
│  1. Skip non-mutating methods (GET, HEAD, OPTIONS)  │
│     or requests with no Idempotency-Key header      │
│  2. Validate X-Tenant-ID (UUID) + Idempotency-Key   │
│     → 400 Bad Request on violation                  │
│  3. SHA-256 fingerprint of raw request body         │
│  4. Look up dedupKey in HttpReplayStore             │
│     ├─ HIT, same fingerprint  → replay 200 OK       │
│     │                            Idempotent-Replayed: true
│     └─ HIT, diff fingerprint  → 409 Conflict        │
│  5. Cache miss → acquireLock (SETNX in Redis)       │
│     └─ lock held by concurrent thread → 409         │
│  6. Run filter chain (controller + service)         │
│  7. Store response in HttpReplayStore + TTL         │
│     ├─ success → release lock                       │
│     └─ store fails → EXTEND lock (don't release)   │
│        (client gets 5xx; lock prevents unsafe retry)│
└─────────────────────────────────────────────────────┘

Key implementation details:

CachedBodyRequestWrapper — the HTTP request body is a stream that can only be read once. This wrapper caches the raw bytes so the filter can compute the SHA-256 fingerprint and Spring can still deserialize the body downstream.

ContentCachingResponseWrapper — the response body is captured after the chain runs so the filter can store it in the replay store before releasing the lock.

SETNX lock (step 5) — Redis SET … NX EX (or in-memory equivalent). If two threads with the same new key arrive simultaneously, exactly one acquires the lock; the other gets a clear 409 rather than creating a duplicate order.

Lock retention on store failure (step 7) — if Redis write fails after the order was created, releasing the lock would allow a retry that creates a second order. Instead, the lock TTL is extended to match the idempotency TTL. The client gets a 5xx, and retrying is unsafe until the lock expires.

Idempotent-Replayed: true header — set on every replayed response so clients, load balancers, and observability tools can distinguish replays from fresh creation responses.

Response size guard — responses exceeding idempotency.store.max-response-bytes (default 10 MB) are replaced with a 409 marker record. The business action completed; it just can't be replayed verbatim to avoid OOM.

HttpReplayStore — the pluggable store

public interface HttpReplayStore {
    HttpReplayRecord get(String dedupKey);
    boolean putIfAbsent(String dedupKey, HttpReplayRecord record, Duration ttl);
    boolean acquireLock(String lockKey, Duration ttl);
    void releaseLock(String lockKey);
    void extendLockTtl(String lockKey, Duration ttl);

    record HttpReplayRecord(
        int    statusCode,
        String contentType,
        String body,               // UTF-8 string (JSON in practice)
        String requestFingerprint  // SHA-256 of raw request bytes
    ) {}
}

Two implementations are wired via @ConditionalOnProperty:

Implementation	`idempotency.store.type`	Notes
`InMemoryHttpReplayStore`	`in-memory` (default)	`ConcurrentHashMap` with TTL. No Redis needed.
`RedisHttpReplayStore`	`redis`	Jackson JSON in Redis. Human-readable via `redis-cli`.

The filter depends only on the interface — swapping stores requires no code changes.

What a record looks like in `redis-cli`

# Cache entry
GET order:idempotency:2b8de313-...:order-abc
{"status":201,"contentType":"application/json","body":"{\"id\":\"a1b2...\", ...}","requestFingerprint":"e3b0..."}

# In-flight lock (present only while first request is being processed)
EXISTS order:idempotency:lock:2b8de313-...:order-abc

# Scan all cache entries for a tenant
KEYS order:idempotency:2b8de313-...:*

# Scan all in-flight locks across all tenants
KEYS order:idempotency:lock:*

The :lock: segment sits before the tenant ID — so cache entries and locks are separately scannable namespaces.

Switching to Redis

Step 1 — start Redis:

# from idempotency-production-redis/
docker compose up -d

This starts redis:7-alpine on the default port 6379 with a persistent volume.

Step 2 — run with the redis Spring profile:

./gradlew :idempotency-production-redis:bootRun \
  --args='--spring.profiles.active=redis'

application-redis.properties is activated automatically, overriding:

idempotency.store.type=redis
spring.data.redis.host=localhost
spring.data.redis.port=6379

Inspect live keys:

docker compose exec redis redis-cli
> KEYS order:idempotency:*               # all cache entries
> KEYS order:idempotency:lock:*          # all in-flight locks
> GET  order:idempotency:2b8de313-...:order-abc
> TTL  order:idempotency:2b8de313-...:order-abc

Tests

Production module test coverage

Test class	Store	What it covers
`ProductionOrderServiceTest`	—	Service creates order with correct fields
`OrderIdempotencyIntegrationTest`	In-memory	12 HTTP-level scenarios
`OrderIdempotencyRedisIntegrationTest`	Real Redis (Testcontainers)	Same scenarios against live Redis

Integration test scenarios

First call → 201 Created
Retry same key + payload → 200 OK, Idempotent-Replayed: true, identical body
Retry different payload → 409 Conflict
Different key → independent 201 Created
Missing Idempotency-Key → filter skips (passes through)
Missing X-Tenant-ID → 400 Bad Request
Malformed X-Tenant-ID → 400 Bad Request
Empty Idempotency-Key → 400 Bad Request
Oversized Idempotency-Key (> 255 chars) → 400 Bad Request
Same key, different tenants → isolated; both get 201
GET request → filter skips; no idempotency logic applied
20 concurrent threads, same key → exactly one order created; all others get 409 (SETNX lock)

Testcontainers Redis wiring

@Container
static GenericContainer<?> redis =
    new GenericContainer<>(DockerImageName.parse("redis:7-alpine"))
        .withExposedPorts(6379);

@DynamicPropertySource
static void redisProperties(DynamicPropertyRegistry registry) {
    registry.add("spring.data.redis.host", redis::getHost);
    registry.add("spring.data.redis.port", () -> redis.getMappedPort(6379));
}

@DynamicPropertySource injects the container's ephemeral port into the Spring context before it starts — no hardcoded ports, no external infrastructure in CI.

Run only the Redis-backed tests:

./gradlew :idempotency-production-redis:test \
  --tests "*.OrderIdempotencyRedisIntegrationTest"

Running All Three Modules

./gradlew :idempotency-easy:bootRun
./gradlew :idempotency-mature:bootRun
./gradlew :idempotency-production-redis:bootRun

Easy — port 8081

# First call
curl -i -X POST http://localhost:8081/api/orders \
  -H 'Content-Type: application/json' \
  -H 'Idempotency-Key: order-123' \
  -d '{"customerId":"25dfc44e-3ed7-4eb4-b412-6a6df8c6d355","amount":99.99}'
# → 201 Created

# Retry — same key, same payload
curl -i -X POST http://localhost:8081/api/orders \
  -H 'Content-Type: application/json' \
  -H 'Idempotency-Key: order-123' \
  -d '{"customerId":"25dfc44e-3ed7-4eb4-b412-6a6df8c6d355","amount":99.99}'
# → 200 OK  (same id, same createdAt)

Mature — port 8082

# First call
curl -i -X POST http://localhost:8082/api/orders \
  -H 'Content-Type: application/json' \
  -H 'X-Tenant-ID: 2b8de313-9c3c-4a15-a9b8-0cd1e34be3da' \
  -H 'Idempotency-Key: order-123' \
  -d '{"customerId":"25dfc44e-3ed7-4eb4-b412-6a6df8c6d355","amount":99.99}'
# → 201 Created

# Payload mismatch
curl -i -X POST http://localhost:8082/api/orders \
  -H 'Content-Type: application/json' \
  -H 'X-Tenant-ID: 2b8de313-9c3c-4a15-a9b8-0cd1e34be3da' \
  -H 'Idempotency-Key: order-123' \
  -d '{"customerId":"25dfc44e-3ed7-4eb4-b412-6a6df8c6d355","amount":999.99}'
# → 409 Conflict

Production — port 8083

# First call
curl -i -X POST http://localhost:8083/api/orders \
  -H 'Content-Type: application/json' \
  -H 'X-Tenant-ID: 2b8de313-9c3c-4a15-a9b8-0cd1e34be3da' \
  -H 'Idempotency-Key: order-123' \
  -d '{"customerId":"25dfc44e-3ed7-4eb4-b412-6a6df8c6d355","amount":99.99}'
# → 201 Created

# Retry — filter replays, service never called:
curl -i -X POST http://localhost:8083/api/orders \
  -H 'Content-Type: application/json' \
  -H 'X-Tenant-ID: 2b8de313-9c3c-4a15-a9b8-0cd1e34be3da' \
  -H 'Idempotency-Key: order-123' \
  -d '{"customerId":"25dfc44e-3ed7-4eb4-b412-6a6df8c6d355","amount":99.99}'
# → 200 OK
# Idempotent-Replayed: true

Configuration Reference

All values in idempotency-production-redis/src/main/resources/application.properties:

Property	Default	Description
`idempotency.store.type`	`in-memory`	`in-memory` or `redis`
`idempotency.store.key-prefix`	`order:idempotency:`	Redis key namespace
`idempotency.store.ttl`	`PT24H`	How long completed responses are kept
`idempotency.store.lock-ttl`	`PT30S`	Max time an in-flight lock is held
`idempotency.store.max-response-bytes`	`10485760` (10 MB)	Oversized responses get a marker instead of being cached

Pattern Progression (Beyond This Repo)

Level	Approach	Trade-offs
1	`ConcurrentHashMap`	Simple; in-process only
2	Tenant-scoped map + fingerprint	Multi-tenant; still single-instance
3 (this repo)	Filter + pluggable store (in-memory / Redis)	Horizontally scalable; no DB dependency
4	Redis + DB unique constraint on `(tenant_id, idempotency_key)`	Durable across Redis restarts
5	Outbox / event store	Full audit trail; at-least-once delivery

The filter approach in Level 3 is the right production baseline for most REST APIs: transparent to the service, testable in isolation, and the store can be swapped without touching business logic.

Known Limits (Intentional for Teaching)

No SQL persistence — Redis data loss means idempotency history is lost. A unique constraint on the orders table is the next step.
No distributed lock — the Redis lock is best-effort; a Redis failover mid-lock could theoretically allow a duplicate. Redlock or a DB advisory lock eliminates this.
No metrics — adding Micrometer counters (idempotency.hit, idempotency.miss, idempotency.conflict) is the obvious next production addition.
In-memory store is per-instance — run two instances without Redis and deduplication breaks across them.

Summary

Idempotency is a protocol guarantee, not a business rule. Keeping it in a filter (rather than the service) means:

The service stays a plain factory — easy to test, easy to reason about.
You can swap the store (in-memory → Redis → DB) without touching business logic.
Replay, conflict detection, locking, and the Idempotent-Replayed header are all in one place.

The three-module structure lets you walk a team from "what is idempotency?" all the way to "here's how we'd ship it."

Idempotency Patterns — Create Orders Exactly Once (With Retries)

Why This Matters

The Idempotency Contract

Level 1 — Easy Mode (Teaching)

What it does

How it works

Limitations (and why they matter in the blog)

Level 2 — Mature Mode (Multi-Tenant + Payload Safety)

Required headers

What changes

Tenant-scoped dedup key

Payload fingerprint

Race-safe `putIfAbsent`

Behavior table

Level 3 — Production Mode (Filter-Based + Pluggable Store)

The key design insight

IdempotencyFilter — step by step

HttpReplayStore — the pluggable store

What a record looks like in `redis-cli`

Switching to Redis

Tests

Production module test coverage

Integration test scenarios

Testcontainers Redis wiring

Running All Three Modules

Easy — port 8081

Mature — port 8082

Production — port 8083

Configuration Reference

Pattern Progression (Beyond This Repo)

Known Limits (Intentional for Teaching)

Summary

Comments

More from this blog

Multi-Tenancy Architecture in Spring Boot

Java Synchronization Mechanisms: A Comprehensive Comparison

Debugging Java SSL/TLS Connections: A Guide to javax.net.debug

Command Palette

Why This Matters

The Idempotency Contract

Level 1 — Easy Mode (Teaching)

What it does

How it works

Limitations (and why they matter in the blog)

Level 2 — Mature Mode (Multi-Tenant + Payload Safety)

Required headers

What changes

Tenant-scoped dedup key

Payload fingerprint

Race-safe putIfAbsent

Behavior table

Level 3 — Production Mode (Filter-Based + Pluggable Store)

The key design insight

IdempotencyFilter — step by step

HttpReplayStore — the pluggable store

What a record looks like in redis-cli

Switching to Redis

Tests

Production module test coverage

Integration test scenarios

Testcontainers Redis wiring

Running All Three Modules

Easy — port 8081

Mature — port 8082

Production — port 8083

Configuration Reference

Pattern Progression (Beyond This Repo)

Known Limits (Intentional for Teaching)

Summary

Comments

More from this blog

Race-safe `putIfAbsent`

What a record looks like in `redis-cli`