The people moving into California are wealthier than those moving out. It's not close. The poor and the lower middle class simply can't afford the areas near the coast. I know that a certain world-view paints CA as some giant craphole where everything is falling apart, but contrary to the demagogues (who lie willingly for money and power, as they admit to each other in private texts), it just isn't true.
I bought a house in California 7 years ago - it's almost doubled in value. Whenever an older house goes on the market, it's grabbed immediately, bulldozed and replaced with a beautiful duplex/townhouse that carries double the value of the old house for each unit.
CA has issues, like any other state. It's also the 5th richest state in the country and has the world's 4th or 5th largest economy (climbing from 6th).
The schools are top-notch, the state-college system wonderful, the hiking parks and playgrounds and fields as nice as anywhere, and the money is everywhere. Can't beat the beaches (South Bay is heaven - saw Edelman and Amendola playing catch on Manhattan Beach a few years ago) and the weather is perfect.
Again, all states have problems. CA attracts homeless people because of the climate and hasn't appropriately dealt with the issue, imo. Housing is super expensive and land is tight in the desirable areas.
There's also the issue of water in some areas, which is crippling entire towns inland to the point of turning the tap and getting dust (hopefully, this winter helps with that).
But no, rich people aren't "fleeing" California. I visited a friend in a new Beverly Hills development (you won't touch an outhouse there for under $10m) and they're selling as fast as they go on the market.