Nah, I'm ok with it. When 12-4 Jacksonville came up to Foxboro to play the 10-6 Pats in 2005, we benefited from it. Divisions create rivalries, and give teams a common "goal" to start the year with. You always know that you can get in by winning your division, and winning division games.
The end result in doing away with the current system is to do away with divisions AND conferences, and lump all 32 teams together, which would be asinine. And even then, you'd have people *****ing about the schedule (Why did Team A play the Lions, while Team B played the Pats, and so on)... there's never going to be a completely equitable solution.
I still think the NFL has the best playoff setup of any major sport.