"get physical with Smith"--what does this mean, since he mostly runs deep patterns and you can't touch a receiver after 5 yards??? Also, if you miss a jam on him, it is good night. So, this strategy seems dubious, unless you are also rolling a safety towards him, which proves my point that he is dangerous and as you acknowledge, must be accounted for.
Here is an excerpt from a Clark Judge article about the AFCCG that explains why Smith was quiet in that game after burning the Pats twice previously:
"A coach I trust told me the Patriots will be wise to Smith this time, trying to smother him with double coverage, which could make guys like Anquan Boldin and tight end Dennis Pitta more attractive as short or intermediate options."
So, for whatever reason, you are discounting Smith's ability to dictate defenses--which has immense value beyond stats.
Your argument that the Ravens skill players are average is patently absurd. Their defense was certainly average, at best. Their QB's best play was throwing it up for grabs and having the skill players go get it. I remind you that this team won the Super Bowl. If they did it without a very good defense or QB, I suggest to you that it had a lot to do with their skill players.