An online casual gaming company was testing a new piece of functionality. It did a standard A/B test where half of the players saw variant A and half saw variant B. The test concluded that variant A improved a critical element of the site’s conversion rate with a high degree of confidence, and the business moved all of their players to the winning variant. Unfortunately, revenue immediately started to decline. Only when the business looked further did it make a critical discovery. What is typical in casual gaming apps is that often only 1% of the visitor base spends any money. So, while variant A performed better overall, variant B performed much better for visitors who actually spent money.