Net neutrality is the idea that traffic over a network be treated agnostically, rather than handled by traffic type, content type, hardware type, or content provider. That's essentially how the internet works now with some exceptions (e.g., attempts at "traffic shaping" meant to undermine P2P traffic by some ISPs), and as we all pretty much know it works very well. The internet is open, and all we pay is a connection fee and sometimes overage fees for all the freely available content and connections it provides. It is highly democratic.
If it works, why get rid of it? Well, first you have to follow the money (as usual). ISPs have a prime opportunity to take advantage of their position as the middle man and overcharge us and popular web sites. For example, AT&T will want to charge google extra money to serve youtube videos at a decent download rate. It's a classic case of double-dipping. In the end, prices go up for everybody while telecom gets richer.
What's worse is that tools like deep packet inspection essentially enable network level DRM, making companies like Disney and the record and film insustries very happy. What this means is that they will have an unprecedented amount of control over the content that travels over these networks. Downloading an MP3 or video that you have fair use rights to from your own server at a remote location? Perfectly legal things like this could easily be blocked in the content industy's zeal to prevent piracy (which is really just an excuse - studies have shown that in spite of industry claims the impact of piracy to their bottom line is minimal). They will use musicians and artists to make their argument, but of course the people who gain the most wealth in the content industry are the rights-holders, not necessarily the artists. And this is just one example. Lacking a principle of net neutrality it's not hard to imagine the controls that can be put in place that will limit our access to information.
Tim Berners-Lee, the founding father of the internet, is steadfastly opposed to incursions to net neutrality. Here's one blog entry.
Net Neutrality: This is serious | Decentralized Information Group (DIG) Breadcrumbs
Finally, in an insult to every person of average or greater intelligence, McCain has the audacity to call this bill the Internet Freedom Act. Opposing net neutrality is anything
but freedom on the internet. It's freedom of corporate abuse.
McCain introduces bill to block FCC's net neutrality rules