Lot of unhelpful answers here. I’ll try my best.

In a nutshell, Joe was a UFC commentator who also avidly practices various martial arts. He’s also a successful stand-up comedian. Being personable and well connected, he started his podcast interviewing friends, comedians, and celebrities.

It should be fairly obvious how he appeals to people interested in masculine personalities. In particular teenagers, young men, and people who aren’t very secure in their masculinity tend to like him.

He had/has a rule of letting nearly anyone on the show so long as he felt he could have an interesting conversation with them. To massively simplify, this ultimately led to him having some questionable political provocateurs on the show, many right wingers. Combine this with Joe’s non-combative interview style, and his show ended up being a platform for some pretty out-there political theory. The way he talked about COVID struck many people as pretty irresponsible, for example.

Eventually, many who are left-of-center were scared of even associating with him. That’s a problem for your public image if you claim to be a centrist, as Joe does. Or at least he did, I haven’t kept up with him in a while.

TL;DR: if you listen to Joe Rogan’s podcast people might think you’re overly concerned with your masculinity or that you’re being indoctrinated into extreme right-wing politics.