Let's not minimize this.
What he allegedly did was pretty horrible. He didn't go to The Bunny Ranch. He went to an establishment that treated these women horribly, held them against their will, and forced them to have sex with strange men for money. The fact that there was a clientele that included the likes of Bob Kraft allowed this to happen.
It's not just embarrassing. It's wrong.
That said, one mistake - even a horrible one - shouldn't permanently ruin the reputation of a man who's done so much good in the world. He doesn't get a free pass, but I think it's something that people should eventually get over.