Syncing 500K diamonds without a single double-sell

A double-sell is the worst failure mode in commerce. Not because of the refund — because of what it tells the customer. They chose a stone, entered their card details, and afterwards were told the thing they bought does not exist. No amount of design quality recovers that.

Calavera sells diamonds sourced through Nivoda: a catalogue of 500,000+ stones from global suppliers, where each one is unique and any of them can be sold by someone else at any moment. Inventory is not a number that decrements. It is 500,000 individual items whose existence is controlled by a third party.

Zero double-sells since launch. This is the architecture that produced that.

Why the obvious approach does not work

The obvious approach is to query the supplier API at page load. It fails on latency: a search across 28 filter parameters becomes a live call to someone else's infrastructure, and a luxury storefront cannot ask a customer to wait two seconds for a filter to apply.

The second obvious approach is a nightly full sync into a local database. It fails on freshness: a stone sold at 09:00 stays listed until the next night's run. That is not a rare edge case at this catalogue size — it is the normal state of the system.

The working answer is neither, and it is not a compromise between them. It is a pipeline that continuously reconciles a local canonical copy, plus a reservation layer that makes the remaining staleness harmless.

Adaptive scanning instead of brute force

Iterating 500,000 records page by page on a fixed schedule is both too slow where it matters and wasteful where it does not. Diamond inventory is not uniformly distributed: the price bands with the most stones also see the most turnover, while the rare high-carat end changes slowly.

The scheduler splits the catalogue into price bands and scans by density — bands with more stones and higher churn are visited more often. Work is emitted as messages, one page per message, and processed by roughly 200 concurrent workers.

Two properties make this robust rather than merely fast:

• →One page per message. A worker crash loses one page of work, retried independently. There is no long-running job to lose.
• →Concurrency is a dial, not a constant. It is tuned against the supplier's rate limits, and lowering it degrades freshness rather than breaking correctness.

The three-phase pipeline

Raw supplier data is never written straight to the storefront's source of truth. It passes through three phases, each with a single responsibility:

1. 01Map — normalise supplier fields into the canonical schema. Suppliers disagree about certification bodies, shape names and measurement formats, and that disagreement is contained here.
2. 02Price — apply markup rules, currency conversion and rounding. Pricing logic lives in exactly one place, which is what makes it auditable.
3. 03Rate — score and rank each stone for search relevance, so the catalogue can be ordered by something more useful than price alone.

Separating these is what keeps the system debuggable. When a stone shows the wrong price, the phase is known before the investigation starts. A single monolithic transform turns every bug into a full-pipeline read.

Search: sub-50ms across 28 facets

The canonical dataset is indexed into Typesense, which serves search across 28 parameters — carat, cut, colour, clarity, fluorescence, certification, measurements and so on — with responses under 50ms.

Search runs entirely against the local index. It never touches the supplier API. That is the decision that makes the storefront feel instant, and it is only defensible because the pipeline behind it keeps the index honest.

In front of that sits a versioned cache with ETag support: a 95% hit rate, and 85% of conditional requests answered with a 304. The version is part of the cache key, so a data update invalidates atomically rather than by expiry.

// Version participates in the key, so a new dataset version
// invalidates every dependent entry at once — no staggered expiry window.
const cacheKey = `search:v${datasetVersion}:${hash(filters)}`

The reservation system

Everything above reduces the staleness window. None of it closes it. Between the last sync of a given stone and the moment a customer clicks buy, that stone can be sold elsewhere.

So the last step does not try to prevent staleness — it makes it harmless. When a stone enters a cart, it is reserved, and availability is re-verified against the supplier at that moment. Reservations refresh hourly. The check happens once, on the one item the customer actually cares about, at the one moment correctness matters.

This is the whole trick, and it generalises well beyond diamonds:

Browsing can be eventually consistent. Checkout cannot. Spend your consistency budget at the point of commitment, not on every page view.

A live availability check on every search result would be unaffordable and would slow the catalogue to a crawl. On a single stone at cart time it is one call, and it is the call that eliminates the double-sell.

Operating it

A pipeline that runs unattended still needs to be observable. A 15-page admin dashboard tracks 487K diamonds across three feeds at a 99.7% pipeline success rate — per-phase throughput, failure counts, dataset version and age.

The number that matters operationally is not success rate but data age: how long since each price band was last reconciled. Success rate tells you the pipeline is running. Age tells you whether the storefront is telling the truth.

What generalises

Most stores will never sync half a million third-party products. The structural decisions still transfer:

• →Never let a customer-facing request depend on a third-party API. Sync into something you control and serve from there.
• →Split ingestion into phases with single responsibilities, so failures localise.
• →Make cache invalidation a function of data version, not elapsed time.
• →Verify at the point of commitment. Accept eventual consistency everywhere else.
• →Monitor data age, not just job success.

The result on this project was 500K+ stones searchable in under 50ms, 100% inventory accuracy at checkout, and zero double-sells since launch — from eight microservices that run without a human watching them.