Yeah, but see, it wasn't -directly- in the head. That's what the angle they keep not showing demonstrates. He lowered his shoulder and hit Brown in the shoulder. The impact made Brown's head whip around like it been hit, and it may have been brushed, but it wasn't direct.
Also, according to my timing, your one second is the time he had on the /slow motion/ replays, not the real time replays. It was less than that. Not a lot of time when you're going full speed at someone.
As for Browner, I think there were actually a couple of penalties. One was ....Saunders I think, along the sideline. I think that's the one you're thinking of. Although that might have been McCourty actually.
The other was someone coming across the middle and it was pretty much a clean shoulder to shoulder hit.