Read more
This practical guide for students, researchers and practitioners offers real world guidance for data-driven decision making and innovation.
List of contents
Preface - how to read this book; 1. Introduction and motivation; 2. Running and analyzing experiments: an end-to-end example; 3. Twyman's law and experimentation trustworthiness; 4. Experimentation platform and culture; Part II: 5. Speed matters: an end-to-end case study; 6. Organizational metrics; 7. Metrics for experimentation and the Overall Evaluation Criterion (OEC); 8. Institutional memory and aeta-analysis; 9. Ethics in controlled experiments; Part III: 10. Complementary techniques; 11. Observational causal studies; Part IV: 12. Client-side experiments; 13. Instrumentation; 14. Choosing a randomization unit; 15. Ramping experiment exposure: trading off speed, quality, and risk; 16. Scaling experiment analyses; Part V: 17. The statistics behind online controlled experiments; 18. Variance estimation and improved sensitivity: pitfalls and solutions; 19. The A/A test; 20. Triggering for improved sensitivity; 21. Guardrail metrics; 22. Leakage and interference between variants; 23. Measuring long-term treatment effects.
About the author
Ron Kohavi is a Technical Fellow and corporate VP of Microsoft's Analysis and Experimentation, and was previously director of data mining and personalization at Amazon. He received his Ph.D. in Computer Science from Stanford University. His papers have over 40,000 citations and three of them are in the top 1,000 most-cited papers in Computer Science.Diane Tang is a Google Fellow, with expertise in large-scale data analysis and infrastructure, online controlled experiments, and ads systems. She has an A.B. from Harvard and an M.S./Ph.D. from Stanford University, with patents and publications in mobile networking, information visualization, experiment methodology, data infrastructure, data mining, and large data.Ya Xu heads Data Science and Experimentation at LinkedIn. She has published several papers on experimentation and is a frequent speaker at top-tier conferences and universities. She previously worked at Microsoft and received her Ph.D. in Statistics from Stanford University.
Summary
Getting numbers is easy; getting trustworthy numbers is hard. From experimentation leaders at Amazon, Google, LinkedIn, and Microsoft, this guide to accelerating innovation using A/B tests includes practical examples, pitfalls, and advice for students and industry professionals, plus deeper dives into advanced topics for experienced practitioners.
Additional text
'Experimentation is the future of digital strategy and 'Trustworthy Experiments' will be its Bible. Kohavi, Tang and Xu are three of the most noteworthy experts on experimentation working today and their book delivers a truly practical roadmap for digital experimentation that is useful right out of the box. The revealing case studies they conducted over many decades at Microsoft, Amazon, Google and LinkedIn are organized into easy to understand practical lessens with tremendous depth and clarity. It should be required reading for any manager of a digital business.' Sinan Aral, David Austin Professor of Management, Massachusetts Institute of Technology, and author of The Hype Machine
