The short answer
You are keeping your team small because failure would land on you. That is self-protection dressed up as care. The fix is to separate your reputation from their growth — then give one person a real challenge this week, with your visible backing behind them.
You are managing your own fear of looking bad, and your team is paying the cost. Research on talent retention is clear on this: people leave when they stop growing. Stretch assignments are one of the main ways good people decide to stay. When you take those away, you do not keep them safe — you just give them a reason to find a manager who will back them.
Marcus Aurelius wrote about keeping your own purpose free from compulsion — acting from what is right, not from what feels safe. Right now, a quiet compulsion is running the show: the fear that their failure becomes your failure. That fear is real, but it is not the same thing as leadership. A leader's job is not to eliminate the risk of failure. It is to make the attempt worth taking.
This week, pick one person on your team who is ready for more. Give them one assignment that is a real stretch — not impossible, but genuinely hard. Tell them openly that you know it is a stretch, that you expect them to struggle some, and that you are in their corner when it gets hard. Put your name behind their attempt, not just their outcome. That is what changes the dynamic.