Here's the real answer: ADD DIHEDRAL

No wait, I know you are not flying an airplane.

However, it sounds like you need to add lateral stability - like someone earlier said. If you look at the fan forces, if they are all thrusting straight up, there is nothing to restore the UFO to a level hover when disturbed. WHat you need to do, is tip each fan unit so the exhaust is pointed slightly outward from center. Either that or modify the exhaust slightly outward using a vane of some sort. In this case lateral and longituinal stability are the same mechanism. THIS (angled jets) is what the Harrier uses for stability - the puffer jets are for control.

It works on the same principle as dihedral, to a certain extent. The "down" side automatically experiences more lift and the "up" side gets less -> return to level flight.