Why does my robot's perception break in messy real-world settings?

You can build perception that works, which is already hard. It breaks in messy, unpredictable environments for a reason worth accepting rather than fighting: the real world is far stranger and more varied than any clean setup, and it will always throw something your system has never seen. That is not a failure of your skill. It is the nature of the problem at its edge. The clean environment is the exception; the mess is the rule. So the goal shifts from building something that never fails to building something that fails gracefully and recovers. Feed your system the ugly cases on purpose, the odd lighting, the clutter, the things that do not belong. Every strange failure you can reproduce and study is a gift, because the surprises out there are what you have not yet let it see in here. Keep a collection of the cases that broke it and work through them one by one. The messy world does not become predictable, but your system becomes harder to surprise. You are not chasing perfection, which does not exist in perception; you are widening the range of chaos your work can handle. That widening, case by ugly case, is the whole craft.