How do I know what to test first when prototyping a game?
You can build a prototype now, and the harder question has arrived: what is this prototype actually for? A prototype is not meant to be a small finished game. It exists to answer a question, and the trouble you feel usually means the question has not been named yet. Once you know what you are trying to learn, what to build and how fast gets much clearer. Before you make anything, finish this sentence: I am building this to find out whether. Whether the core mechanic is fun, whether players understand it, whether the idea holds up past a minute. Then build only enough to answer that one thing, and no more. Everything beyond the question is polish you cannot yet afford. Move fast on purpose, because speed is the point of a prototype. The goal is a cheap, quick answer, not a beautiful artifact. Ugly and fast beats pretty and slow here. Learn the one thing, throw away what you do not need, and let each rough prototype earn its keep by teaching you something real.