Concepts › Monitoring and hysteresis

Monitoring and hysteresis

Registering a domain does not just verify it once. A sweeper keeps re-checking it on a schedule that adapts to what state it is in, and reconciles what it sees across more than one resolver before it believes any of it. That sweeper is one long-running process, never a per-invocation function.

The sweeper

Every SWEEP_TICK_SECONDS (60 by default) the sweeper looks for domains whose next_check_at has passed, claims up to SWEEP_BATCH_SIZE of them (100) with select … for update skip locked, and hands each one to the queue. Postgres decides what is due; Redis only carries the work. Losing Redis costs in-flight attempts and never an obligation, because the next tick re-derives the same list from next_check_at.

A claim is held for SWEEP_LEASE_SECONDS (300) before the row becomes claimable again. That has to comfortably exceed the ten-second check budget, or a slow-but-healthy check gets claimed a second time while the first is still running. Five minutes is a wide margin over ten seconds. CHECK_CONCURRENCY (4) bounds how many of those claimed jobs run at once, because each check is up to twenty upstream queries and that multiplies straight into load on other people's authoritative servers.

Adaptive scheduling, not uniform sweeping

docs/DESIGN.md calls this out as one of three decisions that is the infrastructure bill: sweeping every domain on one interval costs roughly ten times as much and gives a worse experience at both ends. The actual intervals, from apps/api/src/sweep/schedule.ts:

The TTL floor only applies to verified. For a domain that already passes, nothing can change faster than the TTL, so polling inside it just re-reads a cache, and raising the interval to match is free accuracy. Applying the same floor to pending would be actively wrong: a provider serving a one-hour negative TTL while a customer is mid-onboarding would push the first re-check an hour out and make onboarding feel broken.

These intervals come from the cost model in docs/DESIGN.md. They are injectable rather than environment-tunable, because changing the scheduling policy is a code change, not a runtime knob.

Consensus across vantage points

Every authenticated check runs against a pool of resolvers concurrently. It is the same pool for POST /v1/domains/:id/checks and for the sweeper, because a verify that consulted one resolver and a sweep that consulted another could disagree about the same domain a minute apart. A strict majority wins:

• Unanimous — the answer passes through untouched.
• Majority, not unanimous — the majority's verdict is kept but raised to at least warn, with an ANSWER_DIVERGES_BY_VANTAGE_POINT finding attached. This is usually mid-propagation and usually resolves on its own; the customer should see it, not have it hidden by outvoting.
• No majority — a two-way split, or a three-way split across three points — there is nothing to believe, so the verdict is indeterminate. The lookups are kept either way: they are real queries and worth showing, even when the answer they produced is not.

Hysteresis: how many failures it takes to believe one

A single failing check never fires a domain.failed webhook on its own. Each domain carries a running count of consecutive failures, reset to zero by any pass or warn, and left completely untouched by indeterminate. Resetting it on uncertainty would let a domain behind a flaky resolver fail forever without ever accumulating enough evidence to be reported, and incrementing it would let our own resolver's bad minute page somebody.

Two thresholds decide where that count lands the domain, both in apps/api/src/domains/hysteresis.ts and both overridable by environment variable:

• DEGRADED_AFTER_FAILURES — default 1. The conservative choice for a warning: it costs a possibly-noisy event and buys the shortest time to notice.
• FAILED_AFTER_FAILURES — default 3. The number that reaches a customer's pager. At the degraded cadence of five minutes, three consecutive failures is roughly ten minutes of sustained failure: long enough to outlast a resolver restart or a zone reload, short enough to still be useful.

state_transitions records every transition with the evidence behind it, so either threshold can be revisited later against real data. Changing one is a restart, not a deploy.

Next

Diagnosis codes covers what a failing check actually tells you, beyond the fact that it failed.