You just, kind of, made my argument for me: They ARE a crutch that can be over-used!!!! We see many that say "I learned to fly only because of SAFE", but did they really? Many learned how to point their plane in a direction and watch it fly in that direction until they decide to go another. The question, however, is did they really learn how to control roll rates or angles of climb or dive? Usually, the answer is NO. Because SAFE only allows a certain amount of control, many times they go full throw on the sticks and think they are doing everything right. As you know, being an experienced pilot, that is not the case.
That said, I do agree with what you are saying as I've seen planes go down for no apparent reason. It happens and, unfortunately, is usually due to pilot error BUT NOT ALWAYS.