The S&P 500 beats buying a house over 20 years

Everyone around me is buying houses right now. They tell me it’s the smartest thing to do because "real estate always goes up." But when I mention investing in the S&P 500, they say it's too risky because I could lose everything. It’s not a perfect comparison, but if you put $100K down on a $500K house, or invest the same amount in the S&P 500, after 20 years—accounting for inflation, taxes, and fees—the stock market historically gives you a much higher return. I don’t know. Maybe I missed something.