I think that’s a problem with jury trials in general, and even authority in general. Sometimes people who you disagree with strongly end up with the power to affect the lives of others. Jury powers, just as any others, can be abused. But they can also check abuses, like if a prosecutor is overzealous, a judge is biased, or lawmakers corrupt.

I think the best we can do is spread power out as much as possible and hope that people will ultimately prove to be good more often than bad, so that all of the maneuvering and countering people use their power for will trend more towards good. Any solution that concentrates power to keep it out if the hands of bad people has the footnote “assuming those who get that power are good and use it in good faith and don’t change while they have power”, which is not a safe assumption.