They are very expensive
Young pilots are told never to go into many controlled areas because of the cost of dealing with ATC so they don't learn proper procedures early. That came about when the government decided that a "user pays" system was more fair. I figure I use ATC every day when two planes don't crash on my house.
Pilot training in Australia is in such bad shape that most of the people I know who tried it are now heavy in debt (often for prepaid flight hours that disappeared in bankruptcy) and won't ever get enough hours for the job they wanted. That issue will have far more consequences for long term aviation safety than some of ATC going away.