All you will find is white paper on the AS3X system . You won't find any hard technical paper on it...Horizon would be foolish to leak that.
Here is a video of a hover by a newb pilot. I can keep mine in a rather stable hover until a breeze comes along. You are not going to fly a heli without making slight stick corrections even with a 3 axis gyro. The gyro is an assist ,it's not an autopilot !
Sounds to me like you need to program a more tame throttle and pitch curve in.
http://www.youtube.com/watch?v=i-zRhMmc5lY