Blog for Web Analytics, Statistics and Data-Driven Internet Marketing | Analytics-Toolkit.com

The Mann Whitney U Test (MWU), also known as the Wilcoxon Rank Sum Test and the Mann-Whitney-Wilcoxon Test, continues to be advertised as the go-to test for analyzing non-normally distributed dat...

https://blog.analytics-toolkit.com/2024/stop-abusing-the-mann-whitney-u-test-mwu/

Sequential statistics are gathering interest and there are more and more questions posed by CROs looking into the matter. For this article I teamed up with Lucia van den Brink, a distinguished CR...

https://blog.analytics-toolkit.com/2023/qa-on-sequential-statistics-in-a-b-testing/

A central feature of sequential testing is the idea of stopping “early”, as in “earlier compared to an equivalent fixed-sample size test”. This allows running A/B tests with fewer users a...

https://blog.analytics-toolkit.com/2023/sequential-testing-is-about-improving-business-returns/

Have you heard how there is a much greater probability than generally expected that a statistically significant test outcome is in fact a false positive? In industry jargon: that a variant has be...

https://blog.analytics-toolkit.com/2023/false-positive-risk-in-a-b-testing/

Google Analytics 4 has been a let down in many aspects based on every discussion I’ve seen and had with professionals of all stripes – marketers, advertising specialists, CROs, GA professiona...

https://blog.analytics-toolkit.com/2023/a-lightweight-google-analytics-4-integration/

Analytics Toolkit was conceived in 2012 as a set of tools that automate essential Google Analytics-related tasks and augment the GA functionalities in various ways. This goal was achieved in the ...

https://blog.analytics-toolkit.com/2023/analytics-toolkit-to-discontinue-google-analytics-related-functionalities/

Running shorter tests is key to improving the efficiency of experimentation as it translates to smaller direct losses from testing inferior experiences and also less unrealized revenue due to lat...

https://blog.analytics-toolkit.com/2022/how-to-run-shorter-a-b-tests/

In A/B testing sequential tests are gradually becoming the norm due to the increased efficiency and flexibility that they grant practitioners. In most practical scenarios sequential tests offer a...

https://blog.analytics-toolkit.com/2022/comparison-of-the-statistical-power-of-sequential-tests/

One topic has surfaced in my ten years of developing statistical tools, consulting, and participating in discussions and conversations with CRO & A/B testing practitioners as causing the most con...

https://blog.analytics-toolkit.com/2022/statistical-power-mde-and-designing-statistical-tests/

How long does a typical A/B test run for? What percentage of A/B tests result in a ‘winner’? What is the average lift achieved in online controlled experiments? How good are top conversion ra...

https://blog.analytics-toolkit.com/2022/what-can-be-learned-from-1001-a-b-tests/

What is the best design for a statistical test with sequential evaluation of the data at multiple points in time? This is a question anyone who has realized that unaccounted for peeking with inte...

https://blog.analytics-toolkit.com/2022/fully-sequential-vs-group-sequential-tests/

In online A/B testing it is not uncommon to see session-based metrics being used as the primary performance indicator. Session-based conversion rates and session-based averages (like average reve...

https://blog.analytics-toolkit.com/2022/when-session-based-metrics-mislead/

Navigating the maze of A/B testing statistics can be challenging. This is especially true for those new to statistics and probability. One reason is the obscure terminology popping up in every ot...

https://blog.analytics-toolkit.com/2022/a-b-testing-statistics-a-concise-guide/

Hundreds if not thousands of books have been written about both p-values and confidence intervals (CIs) – two of the most widely used statistics in online controlled experiments. Yet, these con...

https://blog.analytics-toolkit.com/2022/p-values-and-confidence-intervals-explained/

Have you ever thought that statistically rigorous A/B tests are impractical? Or do you have trouble selling the need for rigor in testing to your clients, coworkers, or boss? This article debunks...

https://blog.analytics-toolkit.com/2022/top-misconceptions-about-scientific-rigor-in-a-b-testing/