Sounds to me like you forgot to use the rudder. I've done the same thing with the same results. The first instinct sometimes is to correct with aileron, but rudder is what it needs.