That's like saying someone on trial should get to agree with the state on who the judge is
I'd say it's more akin to jury selection (voir dire) in which the defense does have a say.
Edit: And in any case, if a judge has conflict of interest the defense can ask for them to recuse themselves anyway.