It’d have to pass constitutional muster. Usually, it’s hard for one branch of government to impose restrictions on another branch of government without a constitutional amendment. It’d end up in front of the Supreme Court to decide if they would accept that restriction or not.

One of the reasons increasing the size of the Supreme Court comes up as a solution is because Congress is explicitly allowed to just do that.