Here's a wild thought - though I liked the comment about you're flying slower with the gear down - with the gear out you've widened your moment arm for lateral forces; like a tight rope walker carrying a pole or extending his arms. You're also adding a lot of drag. In free flight if we had a plane that "rooted" or "hunted" (nose worked side-to-side) we checked the tail surfaces for warpage and alignment. But they had a lot of dihedral. If you're swimming in straight flight I would suspect vertical stab out of alignment or twist.