The short answer
Fix the variables you control before you start: one dial cadence, one script, tagged sources, a defined success threshold written down in advance, and a sample big enough that a single week does not swing it. Most vendor tests fail as experiments because the leads were worked differently from the incumbent’s, which measures the floor rather than the leads.
A test is only worth running if its result would change what you do. That means deciding the threshold before the leads land, because a number decided afterwards is a number chosen to justify whatever you already believed.
Set these before the first lead arrives
- Sample size. Fifty is where averages start to settle; below that one week swings the whole read.
- The cadence, written down, identical to what your incumbent leads get.
- Source tagging in the CRM, so the comparison exists at all.
- The success threshold: "this works if we fund two" beats arguing about it later.
- Who is working them. One rep on new leads and a different one on old leads is not a test.
What to measure, in order
| Metric | What it tells you | Confounded by |
|---|---|---|
| Contact rate | Whether the numbers are real | Time to first dial |
| Qualification rate | Whether the screening held | Rep consistency |
| Statements received | Whether interest was genuine | How the ask was made |
| Submissions | Whether files were fundable | Packaging quality |
| Funded | The only number that pays | Everything above it |
| Cost per funded deal | The comparison figure | Sample size |
Read them top down. A poor funded rate with a strong contact rate is a floor problem; a poor contact rate is a lead problem. Most arguments between brokers and lead vendors are two people looking at different rows of that table.
Two vendors beats one
If the budget allows, split it across two vendors and tag both. A single vendor test tells you how one vendor performed on one fortnight of your floor; two tell you which vendor performed better on the same fortnight of the same floor, which is a far stronger signal for the same money.