A/B testing

Example of A/B testing on a website. By randomly serving visitors two versions of a website that differ only in the design of a single button element, the relative efficacy of the two designs can be measured.

A/B testing (also known as bucket testing, split-run testing, or split testing) is a user experience research method.[1] A/B tests consist of a randomized experiment that usually involves two variants (A and B),[2][3][4] although the concept can be also extended to multiple variants of the same variable. It includes application of statistical hypothesis testing or "two-sample hypothesis testing" as used in the field of statistics. A/B testing is a way to compare multiple versions of a single variable, for example by testing a subject's response to variant A against variant B, and determining which of the variants is more effective.[5]

Multivariate testing or multinomial testing is similar to A/B testing, but may test more than two versions at the same time or use more controls. Simple A/B tests are not valid for observational, quasi-experimental or other non-experimental situations—commonplace with survey data, offline data, and other, more complex phenomena.

  1. ^ Young, Scott W. H. (August 2014). "Improving Library User Experience with A/B Testing: Principles and Process". Weave: Journal of Library User Experience. 1 (1). doi:10.3998/weave.12535642.0001.101. hdl:2027/spo.12535642.0001.101.
  2. ^ Kohavi, Ron; Xu, Ya; Tang, Diane (2000). Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing. Cambridge University Press. Archived from the original on 22 October 2021. Retrieved 22 October 2021.
  3. ^ Kohavi, Ron; Longbotham, Roger (2023). "Online Controlled Experiments and A/B Tests". In Phung, Dinh; Webb, Geoff; Sammut, Claude (eds.). Encyclopedia of Machine Learning and Data Science. Springer. pp. 891–892. doi:10.1007/978-1-4899-7502-7_891-2. ISBN 978-1-4899-7502-7. Archived from the original on 21 April 2023. Retrieved 21 April 2023.
  4. ^ Kohavi, Ron; Thomke, Stefan (September–October 2017). "The Surprising Power of Online Experiments". Harvard Business Review. pp. 74–82. Archived from the original on 14 August 2021. Retrieved 27 January 2020.
  5. ^ Hanington, Jenna (12 July 2012). "The ABCs of A/B Testing". Pardot. Archived from the original on 24 December 2015. Retrieved 21 February 2016.