Rebuilding a 1.2 million row catalogue, and keeping the evidence

A catalogue migration isn't a copy. The old database had rows that had stopped meaning anything, rows that meant two different things, and relationships pointing at records nobody had deleted. Copying it faster just moves the mess.

The numbers, and what they actually mean

• 1,202,321 rows reconciled in total.
• 379,232 canonical rows — the records the new catalogue treats as real.
• 823,089 raw rows kept as they were, because the only honest way to prove a transformation is to keep its input.

That split is the point of the exercise. Canonical rows are what the product shows; the raw rows are what I can re-run the reconciliation against when somebody finds a discrepancy.

How I kept it checkable

• Imports from a frozen dump, off an allowlist, so the same input produces the same output and a second run is a check rather than a gamble.
• Checksums on the transformed set, so two runs that differ say so immediately.
• Zero foreign-key orphan groups after reconciliation, and 28 sequences checked and repaired where the old system had left them behind its own data.
• Separate checks for rows, relationships, rendering and cutover, because passing one says nothing about the other three.
• Rust and PostgreSQL, reading a MariaDB source, with a React surface for looking at what came out.

Where it stands

Reconciling the rows is not a finished production migration. The cutover, the final interface pass and a rollback rehearsal were all still open when I recorded this. What I can claim is that the data path is controlled and the evidence is kept.