It's my guess that it's a combination of a charger with too-strict rules and batteries with too-high internal resistance.

Your approach of using a wall wart is good, but they are 50ma, not 500ma, and charging yor 2700mah pack from zero would take at least 4 days.

I suggest you look for a different pack with low internal resistance, the kind designed for electric motors. I'll bet that the plane will fly better also, I doubt you are getting full power to the servos as it is now.

TF