A parity check refused to publish new dashboard snapshots for three days. One of the metrics it guards is a ratio, and the guard compared that ratio with an absolute difference against a threshold constant named NUMERIC_TOLERANCE_PERCENT. Every liveness and health check stayed green the whole time, because they were watching whether the job ran, not whether its output was fresh. The fix was two lines. The part that took longer was a dead-man's-switch.

What broke

I run a small pipeline that recomputes a daily snapshot behind a dashboard. One morning the numbers had not moved in three days. Nothing had paged. The job was running on schedule and exiting cleanly enough that nothing screamed, and every health indicator was green.

Before it publishes a fresh snapshot, the pipeline runs a parity check against the last known-good one. If a key metric swings wildly between runs, that usually means an upstream feed broke, so the safe move is to hold the old snapshot instead of publishing garbage. The threshold was one constant, and the comparison was an absolute difference:

NUMERIC_TOLERANCE_PERCENT = 1

# for every guarded metric:
if abs(new_value - old_value) > NUMERIC_TOLERANCE_PERCENT:
    reject()   # "too big a jump, hold the old snapshot"

Read those two lines together. The constant is named _PERCENT. The comparison is absolute. For most of the guarded metrics it never mattered, because they were small counts where an absolute delta of 1 and a 1% delta are about the same size. The name had been lying about what the code did for a long time, and nobody had a reason to notice. Then one of the guarded metrics was a ratio.

Why a ratio detonated it

One metric was a return-on-ad-spend figure, revenue / spend * 100. Spend had recently been wound down to almost nothing, and when the denominator approaches zero the ratio explodes. The value was sitting around 11,000. So every day the pipeline computed a fresh number that differed from yesterday's by far more than 1 in absolute terms, every day the guard called that a broken feed, and every day it held the old snapshot. For three days. A guard written to catch a broken feed spent three days protecting me from my own real data.

A ratio compared against an absolute threshold of 1The guard's threshold sat at 1. The actual difference on the ratio metric was about 11,000, four orders of magnitude past the guard, so it rejected every snapshot.guard threshold_PERCENT = 1≈ 11,000actual abs. diffrejected every snapshot · 3 days
The guard compared a ratio (~11,000) against an absolute limit of 1 named _PERCENT. It could never pass.

Why nothing alerted

The pipeline was not crashing. It ran on schedule, computed everything, made a decision, and exited. The only outward symptom was a generic scheduler exit=1 in a log nobody reads, and even that was ambiguous, because holding a snapshot is sometimes the correct call.

The health indicators were all green because they were checking a different thing than the one that failed. They checked liveness: is the job running, is the endpoint up, is the payload well-formed. All true. Nothing checked freshness: is the promoted data any newer than last time. A pipeline that runs perfectly and republishes stale data forever looks identical to a healthy one when your only signal is "did it run." A liveness check is blind to a freeze by construction.

Health checks fired while the snapshot stayed frozenThe liveness check reported OK at every interval for three days, while the promoted snapshot did not advance past day zero until the fix landed.health checkfires every interval · always OKsnapshot promotedfix3 days · no new snapshotDay 0Day 3
Liveness and freshness are orthogonal. The health check fired every interval; the snapshot hadn't moved in three days, and nothing was watching for that.

Liveness tells you the job ran. Only freshness tells you it did any work.

The two fixes

The first fix was to make ratio metrics use a relative tolerance, which is what the variable claimed all along:

# ratio metrics: a RELATIVE %, not an absolute delta
if abs(new - old) / max(abs(old), EPS) * 100 > TOLERANCE_PERCENT:
    reject()

That unfroze the snapshot the same morning. The tolerance math was only the symptom. What actually let it hide was that a silently frozen pipeline could sit behind green health checks for three days. So about forty minutes later I shipped a dead-man's-switch, a separate check that ignores whether the job ran and asks one question: is the promoted snapshot older than it should be?

import time

def check_freshness(last_updated_epoch, max_age_hours=24):
    """Page if the thing that should be moving has stopped moving.
    Independent of whether the producing job 'ran' or 'exited 0'."""
    age_h = (time.time() - last_updated_epoch) / 3600
    if age_h > max_age_hours:
        raise SystemExit(
            f"STALE: freshest record is {age_h:.1f}h old (limit {max_age_hours}h)"
        )

A variable name is a claim the code has to keep, and _PERCENT doing an absolute comparison was harmless right up until a ratio metric showed up. No test fails when a name drifts out of sync with what it does. Ratios are tolerance-check poison for the same reason: a denominator that can hit zero produces huge, real, volatile numbers that any absolute threshold reads as broken. I hit a cousin of this later, where Apple renamed an analytics report and a fallback fabricated numbers off a false zero. The bug was four days old before I noticed, the fix was two lines, and the thing worth keeping is that my monitoring could not tell working from frozen.