I think you figured it out. Maybe you're having to hold down-elevator (or down trim) when rightside up? If so, perhaps it's tail heavy, and when inverted that would require enough up elevator to to cancel the down trim plus that much more to compensate for CG. It sounds like a CG problem to me. Straddle the heli with the blades in the fore/aft direction and hook a finger under the flybar on both sides, as near the hub as you can get, and lift it off the floor and see if it's tail heavy... the skids should be near parallel to the floor if it's balanced. The only thing I can think of to do about it if it's tail heavy is to replace the battery with a heavier (more mAh) battery.... or add a brick up front.

Needless to say, if it's tail heavy rightside up it's gonna be tail heavy upside down; it's just that the elevator control usage is reversed when inverted.