Название: Experimentation for Engineers: From A/B testing to Bayesian optimization (Final Release)
Автор: David Sweet
Издательство: Manning Publications
Год: 2023
Страниц: 250
Язык: английский
Формат: pdf (true)
Размер: 10.2 MB

Experimentation for Engineers teaches readers how to improve engineered systems using experimental methods. Experiments are run on live production systems, so they need to be done efficiently and with care. This book shows how.

In Experimentation for Engineers: From A/B testing to Bayesian optimization you will learn how to:

Design, run, and analyze an A/B test
Break the "feedback loops" cause by periodic retraining of ML models
Increase experimentation rate with multi-armed bandits
Tune multiple parameters experimentally with Bayesian optimization
Clearly define business metrics used for decision making
Identify and avoid the common pitfalls of experimentation

Experimentation for Engineers: From A/B testing to Bayesian optimization is a toolbox of techniques for evaluating new features and fine-tuning parameters. You’ll start with a deep dive into methods like A/B testing, and then graduate to advanced techniques used to measure performance in industries such as finance and social media. Learn how to evaluate the changes you make to your system and ensure that your testing doesn’t undermine revenue or other business metrics. By the time you’re done, you’ll be able to seamlessly deploy experiments in production while avoiding common pitfalls.

About the technology
Does my software really work? Did my changes make things better or worse? Should I trade features for performance? Experimentation is the only way to answer questions like these. This unique book reveals sophisticated experimentation practices developed and proven in the world’s most competitive industries that will help you enhance machine learning systems, software applications, and quantitative trading solutions. The code is written to Python 3.6.3, NumPy 1.21.2, and Jupyter 5.4.0.

About the book
Experimentation for Engineers: From A/B testing to Bayesian optimization delivers a toolbox of processes for optimizing software systems. You’ll start by learning the limits of A/B testing, and then graduate to advanced experimentation strategies that take advantage of Machine Learning and probabilistic methods. The skills you’ll master in this practical guide will help you minimize the costs of experimentation and quickly reveal which approaches and features deliver the best business results.

What's inside:

Design, run, and analyze an A/B test
Break the “feedback loops” caused by periodic retraining of ML models
Increase experimentation rate with multi-armed bandits
Tune multiple parameters experimentally with Bayesian optimization

Who should read this book:
If you want to build things, you should also know how to evaluate them. This book is for Machine Learning engineers, quantitative traders, and software engineers looking to measure and improve the performance of whatever they’re building. Performance of the systems they build may be gauged by user behavior, revenue, speed, or similar metrics. You might already be working with an experimentation system at a tech or finance company and want to understand it more deeply. You might be planning or aspiring to work with or build such a system. Students entering industry might find that this book is an ideal introduction to industry practices. A reader should be comfortable with Python, NumPy, and undergraduate math (including basic linear algebra).

Table of Contents
1 Optimizing systems by experiment
2 A/B testing: Evaluating a modification to your system
3 Multi-armed bandits: Maximizing business metrics while experimenting
4 Response surface methodology: Optimizing continuous parameters
5 Contextual bandits: Making targeted decisions
6 Bayesian optimization: Automating experimental optimization
7 Managing business metrics
8 Practical considerations