Rate limits and retries

Rate limits#

Limits are per key, not per account. A separate key per environment gets you separate budgets.

Each ingest request carries up to 500 works, so 60 requests per minute is up to 30,000 works per minute.

A request over the limit returns 429 RATE_LIMITED with a Retry-After header. The value is a whole number of seconds. Wait that long, then retry the same request. Do not retry sooner.

HTTP/1.1 429 Too Many Requests
Retry-After: 12
x-request-id: req_01J8ZK...

Idempotency#

A commit accepts an Idempotency-Key header, so a network failure never causes a double write. Dry-runs ignore the header, because they store nothing.

Send a unique key per logical request. A UUID works well. When you page a large catalogue, use one key per chunk.

We remember keys for 24 hours.

The durable guarantee#

An Idempotency-Key protects a retry for 24 hours. A stable client_work_ref protects it forever.

client_work_ref is the upsert key. Re-sending a work with the same ref updates that work in the database rather than creating a second one. A retry outside the idempotency window still cannot create a duplicate.

Use both. The Idempotency-Key is the convenience layer. The client_work_ref is the durable one.

A retry loop#

key = uuid()
backoff = 1s

while true:
    res = POST /api/v1/catalogue/works
          Authorization: Bearer trk_live_...
          Idempotency-Key: key
          body: { client_artist_ref, catalog_name, works: chunk }

    if res.status in (200, 201):
        break                              # success; an idempotent replay counts as success

    if res.status == 429:
        sleep(res.headers["Retry-After"])  # honour the header, do not guess
        continue

    if res.status in (500, 502, 503, 504):
        sleep(backoff)
        backoff = min(backoff * 2, 60s)
        continue

    raise res                              # any other 4xx: correct the request, do not retry