One of the benefits of web technology is that it is relatively easy to make design changes to a web site or intranet both at the development stage and even when in production. The same is true of course of open source enterprise applications, such as e-commerce and enterprise search. In principle it seems so easy. Measure the performance of Version A, make some changes and then measure the performance of Version B. All you then have to do is compare and implement. Easy!<\/p>\n
Not according to this recently published book on A\/B testing<\/a> by Ron Kohavi, Diane Tang and Ya Xu. The very fact that the book runs to almost 300 pages is an initial indication that A\/B testing is not as easy as many might think. The authors have extensive experience from working at Microsoft, Google and LinkedIn and this experience is very visible throughout the book but is coupled with references to around 300 research papers. The blend between authors, and between practice and research, is exemplary in all regards.<\/p>\n <\/p>\n Part 1 of the book is a general introduction to testing, illustrated with some case studies from Google and Microsoft. Part 2 then goes more deeply into organizational metrics, metrics for experimentation and the overall evaluation criteria, institutional memory and meta-analysis and finally a thoughtful chapter on ethics in controlled experiments.<\/p>\n In Part 3 the authors consider complementary techniques and observational causal studies. Part 4 goes into very considerable detail on building an experimentation platform and the book concludes with a 60 page section on advanced topics for analysing experiments. \u00a0One of the features that intrigued me in the book were the number of named laws, such as Simpson\u2019s Paradox, Goodhart\u2019s Law, Campbell\u2019s Law, the Lucas Critique and Twyman\u2019s Law.<\/p>\n The depth and clarity of the exposition on many quite complex issues is exceptional, and this is clearly a direct result of many years of experience and experimentation. However the authors are not prescriptive in setting out a \u2018best practice’ testing regime, instead guiding the reader through the decisions they need to make in developing a robust A\/B testing programme. It is difficult to see how any other author is going to match this and I would guess that this book is going to be the benchmark title for some years to come.<\/p>\n My only criticisms of this book are first the way that four pages of \u2018recommendations\u2019 from the good and the great of the web design world are presented in the front of the book. They are unnecessary and look like a triumph of PR over good editorial judgement. The second is that I was also unimpressed with the index, with just a long alphabetical list of topics under the headings of \u2018experiments\u2019 and \u2018metrics\u2019. They, and some others, are crying out for some clustering of the terms. I would expect better from Cambridge University Press on both counts.<\/p>\n","protected":false},"excerpt":{"rendered":" One of the benefits of web technology is that it is relatively easy to make design changes to a web site or intranet both at the development stage and even when in production. The same is true of course of open source enterprise applications, such as e-commerce and enterprise search. In principle it seems so… Continue reading Book Review – Trustworthy Online Controlled Experiments: A Practical Guide to A\/B Testing<\/span><\/a><\/p>\n","protected":false},"author":3,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[194,226],"tags":[],"class_list":["post-6599","post","type-post","status-publish","format-standard","hentry","category-book-review","category-summer-2020","entry"],"_links":{"self":[{"href":"https:\/\/archive-irsg.bcs.org\/informer\/index.php?rest_route=\/wp\/v2\/posts\/6599","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/archive-irsg.bcs.org\/informer\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/archive-irsg.bcs.org\/informer\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/archive-irsg.bcs.org\/informer\/index.php?rest_route=\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/archive-irsg.bcs.org\/informer\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=6599"}],"version-history":[{"count":0,"href":"https:\/\/archive-irsg.bcs.org\/informer\/index.php?rest_route=\/wp\/v2\/posts\/6599\/revisions"}],"wp:attachment":[{"href":"https:\/\/archive-irsg.bcs.org\/informer\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=6599"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/archive-irsg.bcs.org\/informer\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=6599"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/archive-irsg.bcs.org\/informer\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=6599"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}