How to Implement Retries and Idempotency

Every integration retries. The question is whether retrying is safe. A retry of a read is harmless; a retry of “create invoice” can bill a customer twice. This guide shows how to build retries and idempotency together so that failures cost time, not data integrity.

The core rule

Only retry operations that are idempotent — or that you have made idempotent.

An operation is idempotent if performing it several times has the same effect as performing it once. PUT /items/ABC {qty: 5} is idempotent: setting quantity to 5 twice leaves it at 5. POST /orders is not: two calls create two orders. RFC 9110 defines PUT, DELETE, and the safe methods as idempotent; POST and PATCH are not by definition.

Step 1: Classify errors

Retrying everything wastes time and can make things worse. Retrying nothing turns every network blip into a support ticket. Classify:

Category	Examples	Retry?
Transient network	Connection reset, DNS timeout, TLS handshake failure	Yes
Server overload or unavailability	HTTP 502, 503, 504	Yes
Rate limited	HTTP 429, Shopify GraphQL THROTTLED error	Yes, honoring Retry-After or throttle data
Ambiguous timeout	Request sent, no response	Yes, only if idempotent
Client error	HTTP 400, 404, 422, validation errors	No — fix data, then replay
Auth error	HTTP 401, 403	No — alert; credentials or scopes are wrong
Conflict	HTTP 409, unique violation	Usually no — inspect; often means “already done”

Encode the classification in one place:

type Outcome = 'retry' | 'fail' | 'done';

function classify(err: unknown): Outcome {
  if (err instanceof UniqueViolationError) return 'done';           // already created
  if (err instanceof HttpError) {
    if (err.status === 429 || err.status >= 500) return 'retry';
    if (err.status === 408) return 'retry';
    return 'fail';
  }
  if (err instanceof GraphQLThrottledError) return 'retry';
  if (isNetworkError(err)) return 'retry';
  return 'fail';                                                     // unknown: do not loop forever
}

Step 2: Back off exponentially, with jitter and a cap

Immediate retries hammer a struggling system. Use exponential backoff: wait longer after each failure. Add jitter so many jobs failing simultaneously do not retry in synchronized waves. Cap the maximum delay and the number of attempts.

const BASE_MS = 2_000;       // first retry after ~2 s
const MAX_DELAY_MS = 30 * 60_000;
const MAX_ATTEMPTS = 10;

// "Full jitter": random delay between 0 and the exponential ceiling.
function nextDelay(attempt: number): number {
  const ceiling = Math.min(MAX_DELAY_MS, BASE_MS * 2 ** attempt);
  return Math.floor(Math.random() * ceiling);
}

async function scheduleRetry(job: Job, err: unknown, retryAfterMs?: number) {
  if (job.attempts >= MAX_ATTEMPTS) return deadLetter(job, err);
  const delay = Math.max(retryAfterMs ?? 0, nextDelay(job.attempts));
  await jobs.update(job.id, { status: 'failed', nextRunAt: new Date(Date.now() + delay), lastError: String(err) });
}

Choose the parameters from your business tolerance. If an order must reach the ERP within two hours, ten attempts with a 30-minute cap fits; inventory updates may want faster, shorter retry sequences because stale stock data is worse than late stock data.

Honor rate-limit signals

• If a response includes a Retry-After header, wait at least that long.
• For Shopify’s GraphQL Admin API, read extensions.cost.throttleStatus (currently available points and restore rate) and compute how long until enough capacity is restored for your next query.
• For APIs with quota headers (BigCommerce, for example, returns rate-limit headers with remaining requests and reset time), pause the whole client — not just the one job — until the window resets.

Step 3: Make writes idempotent

There are three practical techniques. Use whichever the target system supports, in this order of preference.

Natural keys and upserts

If the target lets you “create or update by external key,” use it. A product upsert by SKU, or a stock set by SKU and warehouse, is naturally idempotent. This is the simplest and most robust option.

External reference with uniqueness

For records that are created once (orders, invoices, payments), store the source system’s ID on the target record and enforce uniqueness. Retrying a create then either succeeds once or fails with a conflict you can treat as success. See preventing duplicate orders.

Idempotency keys

The client sends a unique key with a non-idempotent request. The server stores the key with the result of the first execution; subsequent requests with the same key return the stored result without executing again. The IETF HTTPAPI working group has an Internet-Draft standardizing an Idempotency-Key header, and many payment and commerce APIs use the pattern. Shopify’s GraphQL Admin API uses an @idempotent directive on mutations that support or require keys.

The key must be deterministic per logical operation. Derive it from the business event, not from the attempt:

// Good: same key on every retry of the same operation
const key = `invoice:create:${order.shop}:${order.id}`;

// Bad: new key per attempt — retries are no longer deduplicated
const key = crypto.randomUUID();

If an API requires a UUID format, generate it once when the job is created, store it on the job, and reuse it on every attempt.

Implementing idempotency keys in your own ERP API

If you control the ERP’s API, adding idempotency is a contained change:

CREATE TABLE idempotency_keys (
  key           TEXT PRIMARY KEY,
  request_hash  TEXT NOT NULL,
  status        TEXT NOT NULL,             -- in_progress | completed
  response_code INT,
  response_body JSONB,
  created_at    TIMESTAMPTZ NOT NULL DEFAULT now()
);

On each request with a key:

1. Insert the key as in_progress. If it already exists and is completed, return the stored response. If it is in_progress, return 409 (a concurrent duplicate).
2. If the stored request_hash differs from the current request’s hash, reject with 422 — the client reused a key for a different request.
3. Execute the operation in the same database transaction that marks the key completed and stores the response.
4. Expire keys after a retention period that exceeds your longest retry and replay window.

Step 4: Bound the whole process

• Maximum attempts per job, after which it goes to a dead-letter queue. See dead-letter queues.
• Maximum age: some jobs become meaningless if too old (an inventory update from yesterday). Drop or supersede them instead of retrying.
• Circuit breaker: if most calls to the ERP are failing, stop sending for a period instead of letting every job burn attempts. Resume with a trickle of test requests.

Security considerations

• Idempotency keys should not contain secrets or personal data; they often appear in logs.
• Scope keys per tenant or shop so one client cannot replay another’s result.
• Validate that a reused key matches the original request to stop accidental or malicious reuse.

Testing

• Simulate a timeout after the server commits; confirm the retry returns the original result.
• Send two concurrent requests with the same key; confirm one executes and one gets a conflict or the stored result.
• Force 503 responses and confirm the delay sequence grows, is randomized, and stops at the cap.
• Return 400 and confirm the job fails immediately into review without retries.
• Return 429 with Retry-After: 30 and confirm the next attempt waits at least 30 seconds.

Common mistakes

• Retrying non-idempotent creates without a key or external reference.
• Random idempotency keys per attempt.
• Retrying validation errors until the attempt limit, delaying human attention.
• Backoff without jitter, causing retry storms after an outage.
• Infinite retries with no dead-letter destination.

Summary

Make operations idempotent first — through upserts, unique external references, or deterministic idempotency keys — and only then retry. Classify errors, back off exponentially with jitter and a cap, honor rate-limit signals, and stop after a bounded number of attempts by moving the job to a dead-letter queue. Retries then become a reliability tool instead of a source of duplicates.