Sure, him at $20M is no slam dunk value proposition. But you have to look at the full picture. They're going to go into next offseason with over $60M in cap space and every meaningful player on their roster is signed besides Chaisson, Hawkins, Tonga, Hooper, Gibbens. To level, set, they've used about $75M in cap space since the start of last offseason. So they're hardly pressing for cap space at all. You take a look at the FA market, and a guy like Alex Pierce is projected at $15-20M/year. A guy like Romeo Doubs is projected at $11.8-$12.8M/year.
If they traded for a #1 WR type (because there isn't one in FA unless you ant to gamble on headcase Pickens) then you could make an argument to let Diggs go to balance spending. But if they don't, "overpaying" Diggs at $20M as a stop gap #1 is exactly what you should be doing vs. investing long-term money in subpar options. And even if you do add a #1 WR type, part of the benefit of having a QB on a rookie deal with a pretty full depth chart and still tons of money to spend is that you can "overpay" to keep a great #2 target that the QB is familiar with.
I just fail to see any realistic scenario where letting Diggs go is going to make sense, even if I do concede that $20M is a high salary for him.