A Real-World Debugging Story: Tracing a Silent Data-Corruption Bug in Production

The bug had no stack trace, no failed request, and no error log — just an inventory count that was occasionally, quietly wrong by a small amount. This is the actual sequence of steps that found it: the dead ends included, because the dead ends are usually the most useful part of a debugging story.

A retail client noticed their warehouse inventory counts drifted from reality by a handful of units, a few times a week, always small, never in a pattern anyone could immediately see. Nothing crashed. Nothing logged an error. The system looked, by every conventional monitoring signal, completely healthy — which made it one of the more uncomfortable categories of bug: the kind that erodes trust in a system precisely because it never announces itself.

The Dead Ends, In Order

First suspicion: a bad migration silently corrupting historical rows. We diffed a week of database snapshots — the historical data was untouched; only fresh writes were affected. Second suspicion: a third-party integration double-counting on retries. We audited every webhook handler for idempotency — all correctly deduplicated by an idempotency key. Third suspicion: a rounding error in unit conversion logic. We unit-tested every conversion path with adversarial inputs — all correct. Three plausible theories, three dead ends, and three days gone.

The Signal That Actually Mattered

The breakthrough came from a question we should have asked on day one: does the discrepancy correlate with anything other than time? It did — every affected item had been updated by two different warehouse terminals within the same second. That's not a data problem. That's a concurrency problem wearing a data problem's clothes.

Silent corruption bugs are rarely about bad data. They're almost always about two correct operations that were never supposed to run at the same time.

The Actual Bug

The inventory update endpoint read the current count, computed a new count in application code, and wrote it back — a classic read-modify-write with no locking and no atomic increment. Two terminals updating the same SKU within the same request window would both read the same starting count, both compute their own new value, and whichever write landed second would silently overwrite the first — losing one of the two updates without a single error anywhere in the stack.

-- Before: read, compute in app code, write back — a race condition
-- waiting for two concurrent requests on the same row.
-- SELECT quantity FROM inventory WHERE sku = $1;
-- (app computes new_quantity = quantity - sold)
-- UPDATE inventory SET quantity = $2 WHERE sku = $1;

-- After: the database performs the arithmetic atomically —
-- no window where two concurrent writes can clobber each other.
UPDATE inventory
SET quantity = quantity - $2
WHERE sku = $1
RETURNING quantity;

The fix took ten minutes once we understood it. Finding it took three days, because we spent the first three looking at the data instead of the timing. The lesson we now apply to every 'the numbers are occasionally wrong' report: check for concurrent writes to the same row before checking anything about the data itself.

Correlate With Timing, Not Just Content: When corrupted data has no pattern in its content, look for a pattern in when it was written — and by how many things at once.

Read-Modify-Write Is a Race Condition by Default: Any endpoint that reads a value, computes a new one in application code, and writes it back needs an explicit lock or an atomic database operation — every time, with no exceptions.