Catalogue Ingest API

Load a full catalogue

Chunk, retry and idempotency rules for catalogues of thousands of works.

The endpoint takes up to 500 works per request. To load a catalogue of thousands, page it into chunks and make each chunk retry on its own.

The rules#

Split into chunks of 500 works.
Send one Idempotency-Key per chunk. Use a fresh UUID for each. A retried chunk then replays instead of writing twice.
Put a stable client_work_ref on every work, so a re-send merges rather than duplicates.
To get one catalogue per artist, group each chunk by artist and send the same client_artist_ref.
On 429 or any 5xx, retry the same chunk with the same body and the same key.
Optionally dry-run each chunk first, to see the gaps before you commit.

At 500 works per request and 60 requests per minute, a 5,000 work catalogue is 10 requests. The limit is not the constraint. Your own error handling is.

The loop#

code

for chunk in chunks(works, size=500):
    key = uuid()
    backoff = 1s

    while true:
        res = POST /api/v1/catalogue/works
              Authorization: Bearer trk_live_...
              Idempotency-Key: key
              body: { client_artist_ref, catalog_name, works: chunk }

        if res.status in (200, 201):
            break                              # success; an idempotent replay counts as success

        if res.status == 429:
            sleep(res.headers["Retry-After"])
            continue

        if res.status in (500, 502, 503, 504):
            sleep(backoff)
            backoff = min(backoff * 2, 60s)
            continue

        raise res                              # any other 4xx: correct the request, do not retry

Where the chunks land#

Omit catalog_id and send the same client_artist_ref on every chunk. We route them all to that artist's catalogue. See Catalogue routing.

If you hold no artist id, create the catalogue on the first chunk by omitting catalog_id, then pass the returned catalog_id on every later chunk.

A first load#

1

Dry-run every chunk

Send each chunk with ?dry_run=true and collect the gaps and validation reports. Nothing is stored, so you can do this as often as you like.

2

Read the reports together

Aggregate across chunks. A field missing on 4,000 of 5,000 works is usually one wrong column in your mapping, not 4,000 separate problems.

3

Correct the mapping, then commit

Fix the mapping, then send the chunks again without dry_run and with an Idempotency-Key per chunk.

4

Store the id pairs

Keep every results[].client_work_ref beside its work_id. That is your reconciliation key from now on.

Correcting a mapping before the first commit is much cheaper than correcting stored works afterwards.