The reasoning of the OP is correct insofar as it shows that if every team thinks the referees are biased against them, then most of them are inaccurate.
But the general conclusion that the referees are not biased against the Patriots is inaccurate for two reasons:
(1)
The OP's argument is that psychological factors (P) can cause fans to believe that referees are biased (B) against their team. But that does not imply that referees are not in fact biased against their team.
(2)
Second of all, there is clear, objective evidence that the NFL is in fact highly biased against the Patriots. This evidence most simply is in the sequence of multiple actions by multiple levels of bureaucracy up to Goodell in the NFL that caused the severe punishment of the Patriots due to deflategate, which any reasonable observer who had read the evidence and had some background in statistics would conclude was unwarranted. Indeed, there are multiple papers to that effect.
The OP is correct that bias can cause inaccurate perception. But the remedy for this inaccuracy is simply a careful perusal of all the evidence and the rules. For example, one simple way to know that deflategate was nonsense is that nobody not paid by the NFL who has actually read the full Wells report believes it. Obviously, if one person has read the Wells report carefully and has background in the appropriate fields, then that person's viewpoint is more credible than a person who has only skimmed the summary. Again, if you look at the arguments that try and claim deflategate happened, it is immediately clear that the people making the claims did not read the report, because they don't cite the specific scientific data in the report.
Let us look at a contemporary dispute, such as the Steelers' claim that the touchdown was inappropriately called back in the last game. If you look at their argument, they never actually quote the relevant rules. They only use unattributed claims and feelings. The people who believe the pass was incomplete quote the specific relevant rule. That makes the claims of the people who quote the specific rule involved inherently more credible.