ΞAskAurelius.ai answers
How do I avoid confounds and peeking in my A/B tests?
You can design a test; now you are wrestling with the ways a clean design still gets fooled. That is a sign you are maturing as an analyst, because confounds and peeking are exactly what trip up people who think the hard part is over. The discipline is decided in advance. Set your sample size and your stopping point before you start, and do not check for a winner every hour, because early noise looks like signal. For confounds, randomize well and ask what else changed during the test window besides the thing you are testing. One small step: before your next experiment, write down the metric, the duration, and the stopping rule, then hold to them. Deciding the rules while you still cannot see the results is what keeps you honest.