Answering All 31 Methodology Questions for AI Visibility Platforms

Where he is most right

• •Q3 — representativeness. Nobody in this industry has a sampling frame. We all write the exam we then sit.
• •Q4 and Q11 — independence. I have been quoting capture counts as though they were observations. They are not, and my own data on synchronised model changes proves it.
• •Q9 and Q10 — uncertainty. Publishing a rate from five runs with no interval is the kind of thing I would criticise in someone else's chart.
• •Q19 — cross-vendor agreement. The absence of this test is the loudest silence in the whole category.
• •Q29 — outcome prediction. "The score went up" predicting "the score will go up" is the trap, and I have not yet run the regression that would get me out of it.

My one pushback

• •The standard he sets — validated against ground truth, externally replicated — is one the alternatives also fail. Bookings do not tell you which channel produced them. Analytics files AI-created demand under branded search, because roughly 94% of assistant users return to Google before booking. Server logs prove a model read your page and say nothing about what it then recommended.
• •Every instrument in this channel errs in the same direction: low. That asymmetry matters. Reading the channel as bigger than it is costs you some wasted effort; reading it as smaller — which every default dashboard nudges you toward — costs you the channel.
• •So "directional, synthetic, and honest about both" is a real category, not a dodge. He concedes this himself near the end, which makes the actual disagreement much smaller than the tone of 31 questions suggests.
• •He is not wrong about any of the 31. He would only be wrong if the conclusion were "therefore do not measure". The conclusion I take is: measure, publish the method, name the uncertainty, and stop calling a panel rate a benchmark.

What I am changing because of this

An answer that costs nothing is not an answer. Five things I can do with data I already have:

1. 1
   Confidence intervals on every rate chart

   Wilson interval computed on the actual run count. Removes the worst offence in Q9.

2. 2
   Report seeds, captures and mentions separately

   So the independent unit is visible instead of buried inside the largest number. Q4.

3. 3
   Publish the run-count stabilisation curve

   I already have 37–77 repeated runs per cell. Bootstrapping how many runs a prompt needs before its rate settles is a re-analysis, not a new study. Q8.

4. 4
   A capture-health line on every study

   Refusals, timeouts and malformed captures counted and shown, instead of silently dropped. Q24.

5. 5
   The cross-vendor test — open invitation

   Publish a frozen panel plus a sampled raw corpus, and co-run the comparison with any vendor prepared to publish the disagreement rate whichever way it falls. Q19.

The underlying work