I'm aware that coaches and players change, so that someone like Brady, who has a long career, is essentially playing the uniforms. However, if you ever have played a sport, you just "know" that you will beat some teams or some individuals. On the other hand, you just "know" that some teams (think Miami in Miami) or individuals (Federer versus Nadal on clay) will give you fits.
What you're talking about is being psyched-out, mainly. Not that it it's irrelevant, but still. The point being that the only constant related to the Pats difficulties in Miami is the location (whereas Federer v. Nadal on clay is pretty much all constants, so it doesn't seem to apply, really).
I get it, though. I used to play a lot of golf in a regular foursome back in Michigan. There was one guy who'd typically shoot about 10 strokes better than I did on every course but one. I'd often shoot my usual 95, but he'd often blow up to 100 there.
I also saw that with my HS basketball team when I was a freshman 3rd-stringer. Our team typically finished ~.500 every season and there was another annual ~.500 team that our team had lost to every season, both Home and Away, for over a decade, through three different coaches and all those graduating classes. When the first game against them came up my freshman season, we were 3-1 and they were 1-3, and the upper-classmen/starters had a palpable sense of foreboding that I simply couldn't understand. But, I wasn't the only one. There was a sophomore, "too short to start", 2nd-stringer who didn't get the pervasive negativity, either.
Sure enough, though, we went into their building "nervous" and our starters played like crap. They beat themselves. With just a couple minutes left, down 20-40 (!), and four of our starters fouled out, we freshmen got a chance to play (against some of the other team's #2s). I did the only things I was good at - stealing the ball off the dribble, blocking passes, and passing the ball off. I passed the ball off to this other kid who did what
he was good at - driving straight to the basket for the layup (screw setting up a weave). I created five turnovers, and this other kid scored on four layups and two foul shots for a 10-0 run in about a minute. The other team put a couple of their starters back in and put a lid on it, so we lost anyway. For me, it was kind of a "Chauncey Gardener" thing - I didn't know that I wasn't supposed to be able to do that. For this other kid, it was more like a Brady thing.
We also lost our home game against them later in the season, but it was a much closer game.
The next season, I only made the JV. This other kid, though, made the varsity again, and ended up hanging a 20-burger on our "nemesis" in the Away game and another 30 on them in our own gym. Would've been 32 if dunking had been legal (he seriously didn't give a sh*t). Anyway, he went on to become the leading scorer in school history and our school never lost to those other clowns until well after we both graduated.