Recommended Resources for Starting A/B Testing

Starter kits of resources for product teams and data teams get started with A/B testing.
A/B Testing

Emily Robinson


April 20, 2023

This post was co-written with Eddie Wharton, a data science consultant for consumer and web3 start-ups.

A/B tests are a powerful tool used by thousands of companies, from small start-ups to tech giants to companies founded in the 1850s. While they sound simple in theory, remember that “getting numbers is easy, getting numbers you can trust is hard!” Mistakes can be costly - we’ve caught and fixed multi-million dollar issues. To make A/B testing useful, you need to design the experiment well, check data trustworthiness, run some statistics, oversee an experimentation platform, and effectively communicate results. With so many moving parts, it’s best to learn from others. After over a decade of combined experience building and running A/B testing systems at various companies and teaching experimentation best practices, we wanted to share our favorite resources for getting started with A/B testing.

“Only a fool learns from their own mistakes. The wise person learns from the mistakes of others.” -Otto Von Bismark

There is a wealth of public material on A/B Testing, from blog posts to academic papers to talks to courses. But that same abundance can be overwhelming and make it hard for someone new to experimentation to know where to start. We’ve put together these two starter kits – one for product teams and one for data teams – to help. We also created an accompanying GitHub page with even more resources, including advanced topics. We hope these will get you up-and-running quickly and avoid many of the common A/B testing pitfalls. If there’s a resource you love that we haven’t included in our GitHub page, please file an issue or submit a pull request to add it!

Product Team Starter Kit

This starter kit is an action-oriented guide of short, to-the-point resources meant for members of a product team–Analysts, Product Managers, Designers, Engineers, etc. If you’re going to be the primary person responsible for running and analyzing experiments, or you’re curious and want to dive deeper, check out the Data Team Starter Kit.

Experimentation Introduction and Best Practices

Statistics for Experiments

Data Team Starter Kit

If you’re a data team members (Analysts, Data Scientists, Data Engineers and Data Execs), we recommend reading the Product Team Starter Kit first and then coming here. For an in-depth guide to A/B Testing, we highly recommend the book, Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing by Ron Kohavi, Diane Tang, and Ya Xu. The resources below are mostly papers and can be dense, so it’s okay if you don’t understand everything or end up skimming. Their main purpose is to give you an overview so that you can uncover blind spots and know key terms if you decide to go deeper.

Designing Experiments

Experimentation Platform

Despite sounding simple, experimentation platforms are complex, large engineering investments. If you didn’t previously work somewhere that built one or if this is your first time reading about them, seriously consider buying instead of building. We consider a full “how to” to be advanced topics. Here are a few resources to give you a feel though:

  • Anatomy of a Large-Scale Experimentation Platform by Pavel Dmitriev et al. An overview of the architecture behind Microsoft’s Experimentation Platform, including its four core components and the reasons behind their design.
  • The Modern Experimentation Stack by Che Sharma. Video explainer of the components and challenges in an exp platform. If you don’t have an hour to watch this, you don’t have time to build it. “Experimentation is a big pile of medium-sized problems that all need to work in concert.”
  • Supporting Rapid Product Iteration with an Experimentation Analysis Platform by Arun Balasubramani. DoorDash’s analysis engine for their exp platform, including why they automated and how they implemented it. Links to other useful posts.
  • Experimentation Platform in a Day by Dan Frank. You’re an adult who can decide to build instead of buy. We both worked at companies where building made sense. It could make sense for you too (though, if you have to ask…). This post promises a quick start with code snippets for all pieces. Do this at your own risk or only as long as sensible.


We hope these starter kits have provided you a good jumping off point in your A/B testing journey. If you’re looking for more, check out our GitHub repo, which includes resources on specific advanced topics like switchback tests and estimating network effects. If you found this post useful, we’re considering making more resources for common A/B testing pain points - if you want to register your interest (and have a say in which topics we tackle), take our survey!