SSG does not gain access to your phone number, Discord does.
It's a discord verification option for servers to make things slightly harder for people/malicious actors to join. Yes, a focused malicious actor can bypass it, but that takes effort, and most people are lazy and would rather target easier to access servers. Hence it is treated as a security measure.
I'm not saying you should add your number if you don't want to. You are free to not do so, but at least understand what is actually happening. SSG is not actually gaining knowledge of or access to your phone number by having phone verification on their discord server.