Your Odds are the same as the other three guys for hitting your Set. SO actually they have to hit a card(s) to beat you. If someone goes all in with Suited Connectors this early then he deserves to be knocked out. If he hits his cards then you're out but that is why you play to take a shot at winning the tourn. by playing scared you will never win the whole thing. You may make it to a late round and get some money back but you will never win.

Example: The other day in a live tourn I pushed all in on the 2nd hand with QQ, a player two to my right raised from 25 to 200. He had A Q suited I had QQ. I assumed (correctly) that he had the ace and had to ask did I want to take the chance of being knocked out this early. Well my Qs held up and I went on to the final table with the help of AA a few hands later and KK after that both hands I put someone all in and they called with A in the hole.

My point, sometimes you win, sometimes you lose.