That’s the bold claim made by Avinash Kaushik.
In his latest blog post, he didn’t just question the practice of hypothesis testing through controlled experiments — he outright dismissed it, saying he gave up on A/B testing back in 2012. The title of his post? Just as dramatic: “Stop Experimenting!”
Kaushik highlighted two key strategic flaws with A/B testing:
1. In 2025, A/B tests are still being calculated using last-click attribution data.
2. Most tests focus on trivial, insignificant ideas whose impact is either negligible or skewed by Simpson’s paradox.
To put it mildly, these claims raise some eyebrows. Unfortunately, most of his argument is locked behind a paywalled newsletter, which I don’t subscribe to. Luckily, not everyone is as tight-lipped.
Take Ron Kohavi, for example. He went all in and published a detailed rebuttal to Kaushik’s post.
Kohavi’s main criticism of Kaushik (and I tend to agree) is that Kaushik’s arguments don’t really attack A/B testing as a methodology. Instead, they criticize common mistakes people make when running experiments. For example:
• “Most A/B tests are used for visual tweaks: button placement, colors, images, etc.”
• “A/B testing takes too much time.”
• “Interactions between tests create hard-to-control effects.”
Honestly, these aren’t exactly deal-breakers.
It seems like Kaushik could’ve benefited from chatting with peers (the ones who didn’t abandon experiments back in 2012) before posting such a provocative take. They could’ve told him about hypothesis prioritization, managing variance, layered approaches, sequential testing, CUPED and other wonders of modern experimentation science LOL.
If you’re curious, Kohavi’s full document and comments are available here.
If you work with GA4 to BigQuery exports, be sure to check out my SQL cheat sheet.
