How do I know when my model is actually failing?

Looking at accuracy is a fine place to start, but you have sensed the limit of it, and that instinct is good. Accuracy alone can hide real failure. A model can be right most of the time and still be badly wrong in the ways that matter most. Feeling unsure about the other metrics is not a gap in your ability. It is the next thing to learn, and you are already asking the right question. The reason one number is not enough is that not all mistakes are equal. Missing the rare important case can matter far more than getting the common one right. Different metrics exist to show you different kinds of failure: how often the model cries wolf, how often it misses what matters, whether it fails one group worse than another. Accuracy blends all of that into one figure that can look fine while something serious hides underneath. Take your model and look at where it is wrong, not just how often. Pull up the actual mistakes and ask what kind they are and who they affect. That habit teaches you more than any single metric. Once you see the shape of the failures, the other measures will make sense because you will know what each one is trying to catch. Start by studying your model's wrong answers this week.