Experiment 005, infrastructure

The test that would have caught two dead domains eight weeks earlier.

Two of our sending domains were dead. Everyone found out eight weeks later. We went back through the data to see when we could have known.

What happened

In one campaign, several domains sent the same emails to the same kind of people over the same weeks. Two of them stopped getting replies. Our provider's health grade marked them burned on 4 August. By then we had kept sending on them for most of the summer.

The backtest

We took every campaign we have ever run, tens of thousands of sends, and asked one question: inside a campaign, when did the dead domains first look different from their siblings?

Pooled across the campaign: healthy domains replied at about 0.4%, burned domains at about 0.1%. The gap is not subtle once you look at siblings side by side. Nobody was looking, because every dashboard graded each domain on its own.

Why siblings and not a threshold

A fixed rule such as reply rate under 1% after 200 sends sounds sensible. We tested that one too: at our volume it flagged 97% of domain days, healthy domains included. Our reply rates are low enough that any absolute threshold fires on everything. A sibling has the same copy, the same list and the same weeks, so the comparison cancels all of that out and leaves only the domain.

What we do now

A daily check compares every sending domain against its siblings in the same campaign, using that campaign's own reply rate as the baseline. It has run since 11 September. A domain that falls behind gets read by hand the same day, not graded by a vendor eight weeks later.

Try this

Export replies by sending domain for one campaign. Put the domains next to each other by week. If one has zero while its siblings have some, over hundreds of sends, you have found what the dashboard is not telling you.