Sounds like there is just too much stiffness in the whole system. The control surfaces should flop down with gravity when they are disconnected. If they do not then you must do such remedial work as needed to make them do so. A good 5kg servo will help as will a digital type to assure that the servo is centering properly. Yes it will fly with only the bottom two ailerons working, but you need a bit more movement to assure good roll control. Use the inner holes on the servo disc, 100% movement (ATV) and adjust the aileron horn height to achieve the required movement. This will assure that the maximum torque is being applied to the surfaces.
Evan.