Thanks to everyone for your thoughts, especially those who saw him play. It's been great to learn more about the legend.
Again, not trying to tarnish the guy in any way, but I really know so little about the era and I'm interested in learning more so this thread has helped motivate me to dig around some more.
A contemporary of Brown's was Jim Taylor of the Green Bay Packers. Brown started his career in 1957, and played 118 games. Taylor started his career in 1958, and played in 124 games. Brown was an instant impact player, winning AP MVP his first two seasons, and leading the league in rushing every year but one. Taylor wasn't as impactful, but by the 1960s, had finished just behind Brown several times, and led the league in rushing the only year Brown didn't. Both are HOFers.
Brown was obviously the better player in pretty much every way, but it was useful to see another era HOFer playing the same position to get a relative understanding of just how good Brown was. For example, I could see that Brown averaged 5.2 YPC in the regular season and only 3.65 YPC in the playoffs. Taylor averaged 4.4 YPC in the regular season, and only 3.48 YPC in the playoffs. So the argument that defenses geared up on the run makes a lot of sense now, and digging back through the archives showed a lot of big yardage games on the ground, but it wasn't uncommon at all to see the winning team averaging 3 or 3.5 yards per carry.
At the same time, Taylor's teams went 6-1 in the playoffs while Brown's team went 1-3. Taylor played for the great Vince Lombardi, so that's got to be part of it. But Brown and the Browns were very good too. Back in the day, only the team(s) with the top record got in the playoffs. So Brown played on some very good teams during this time.
In 1957, the Browns had the best regular season record in football. In the other division, the Lions tied the 49ers, and beat San Fran in San Fran 31-27 despite the 49ers running well (33 carries for 127 yards). The Lions would then blow Cleveland out 59-14 in the championship game. Brown ran 20 times for 69 yards. He wasn't even the leading Cleveland rusher that day. That honour belonged to teammate Lew Carpenter, who ran 14 times for 82 yards.
In 1958, the Browns were 9-3, tied for the top record in the league. Unfortunately they were also tied with the Giants in their own division, and lost the divisional playoff game 10-0 in a game where Brown ran only 7 times for 8 yards. I'm wondering what the hell happened in this game. Was Brown injured? It wasn't the QBs lighting it up since there were two who combined to go 10 for 27, 114 yards, 0 TDs and 3 INTs. I'm not sure how to adjust for era, but I think that's pretty bad in every era. That same Giants D which held Brown and the Browns to 24 rushing yards would give up 138 in the championship game to the Colts.
He had his best playoff game in 1964, running for 127 yards over the Colts as they finally won the championship. Oddly, it was the only year of the 4 where his team won the division but had a worse regular season record than the other division winner.
His next year would be his last, and again the Browns would finish the regular season with the best record, only to lose in the finals behind 12 carries for 50 yards from Brown.
A big part of that is probably the QB. Otto Graham and Paul Brown were finishing up their run by the time the Browns drafted Jim Brown. Paul Brown would only be around for the first 6 seasons of Jim Brown's career, and with what looks like 2 JAG QBs. And so maybe he can't win that many games. But some of the performances themselves feel so underwhelming.
It sounds like if we were going to find the QB equivalent of Jim Brown, he'd be like a combo of Peyton Manning and Aaron Rodgers, phenomenal regular season stats and ridiculously unmatched athleticism, but a spotty playoff record. And Brady's claim to GOAT has nothing to do with freak athleticism. Ditto GOAT WR Jerry Rice, who was too slow for many teams who spent his entire career trying to catch him. Though Rice has ridiculous production in the regular season, he was able to match that in the playoffs too, which is where Brady also forges his name in the discussion.
It is impossible to really evaluate across positions as several have noted. But it's been an interesting exercise and a chance to look back at an era I never really knew existed. So I appreciate everyone's thoughts and insights, and if you have more, please keep posting them.