How do I design an A/B test that actually teaches me something?

You can change the button, but changing something is not the same as learning something, and you have put your finger on exactly the right gap. A test only teaches when it is built to answer a question you actually have. Most tests fail to teach not because the change was wrong, but because there was no clear question behind it in the first place. So start before the button. Ask what you believe and want to check, something like, I think a clearer label will make more people click because they understand the offer. Now you have a real hypothesis, and the test has a job: to prove or disprove that specific belief. Whichever way it lands, you learn something, because you were testing an idea, not just poking at a page. For your next test, write one sentence first: I believe that changing this will do that because of this reason. Then test only that one change, so the result has a clear meaning. A test built on a real question teaches you whether the result is a win or a loss. That is the difference between guessing and learning.