How do I handle people who don't belong without feeling scared?

Checking a badge is easy. Turning to someone who should not be there is a different thing, and the fear you feel is honest. You are stepping between a person and where they want to go, and part of you braces for how they might react. Anyone would. The fear shrinks when you stop trying to be forceful and start being clear. You do not need to be the biggest presence in the room. You need a simple, steady script and the willingness to use it. Most people who do not belong leave the moment they meet calm certainty, not aggression. Write your few lines now, before you need them. Something like: this area is restricted, I will need to see your pass, let me help you find where you are meant to be. Practice them out loud until they feel automatic. Prepared words carry you when nerves would not.