Certainly the main problem is the "chicken factor". It may block your mind.
I feel comfortable torque rolling at 10 feet high, no less, specially if there is wind.
You may think that being at ground level the plane should not suffer much damage in a crash.
Anyway it scares me to think you can make a mistake and pull out in a wrong direction at full power at that ground level, the plane can go inverted or missing control and hit you or hit other people nearby. No margin for error.