Maybe that is the way teams think, but it's clearly a business mistake.
If you draft a player, you have them for four years (up to five if they are a first-rounder). After that, they are free agents and the drafting team has to compete with all of the rest. It's a big difference to their future earnings if they are 25 or 28, but shouldn't make any significant difference to their value to the drafting team whether they are with them from 21-25 or from 24-28. That's what you are getting if you draft someone.