No, it isn't.
The undeniable fact is that there is evidence of an extra-marital affair. I won't say there is proof, but there certainly is evidence, enough evidence to make belief that something happened a very reasonable conclusion.
We also know for a fact that the Missus is a devout Catholic. That is not an assumption; that is a widely reported fact. There are countless articles describing how she brought him back into the fold after he had lapsed during his college days. There are also countless articles about them supporting their local diocese when he was in Tenneessee.
It takes a special type of moron to look at what we know and conclude "hey gee, maybe the devout Catholic wife of 25 years and mother of 2 is perfectly comfortable with an open marriage and polyamorous relationship!!"