Experiment 004, copy
11 of our 13 replies were robots.
A campaign dashboard showed 13 replies. We opened them. Two were people.
What the dashboard counted
Our HVAC campaign, a few hundred people in, read on 26 September:
| Line | Count | Of contacted |
|---|---|---|
| Replies, as the platform counts them | 13 | 3.3% |
| Automatic: out of office, auto responders | 11 | 2.8% |
| Written by a person | 2 | 0.5% |
| Positive | 0 | 0% |
| Bounced | a handful | 1.3% |
A 3.3% reply rate sounds like a campaign that is working. A 0.5% human reply rate is a campaign asking a question nobody wants to answer. Same emails, same inbox, same day. The difference is who did the counting.
It was not a one off
Our roofing campaign before it: 9 reply messages, 3 from people, 6 automatic. And a number we once quoted in sales conversations, a reply rate of 1.74% on Google recipients against 0.43% on Microsoft, came from exactly this mistake. We withdrew it when we reread the replies.
Why the field does not exist
No sending platform we have used has a reliable flag that separates a person from an auto responder. Out of office messages arrive as normal replies, in the same thread, counted in the same column. The only way to know is to read the body of every reply, and we do, with a classifier that is checked by hand.
What we do now
Every readout we give, in a weekly report or on a call, has one shape:
| Line | Always shown as |
|---|---|
| Replies | total, human, automatic, positive |
| Bounces | count and share of contacted |
| Replies by the recipient's mail provider | count and share of contacted |
If a number cannot be shown in that shape, we do not quote it.
To be clear: a few hundred contacts is a snapshot of a campaign that is still running, not a verdict on the offer. Two human replies is what an early read looks like at this volume. The point is the counting, not the campaign.
Try this
Open the last 20 replies your dashboard counted. Mark each one person or robot. If more than half are robots, your reply rate is a different number than the one you have been reporting.
Want this for your outbound?