Flying better with flaps down maybe hiding a problem. Either you're flying too slow and are on the verge of a stall when flying flaps up. Or there is a problem with the aircraft. Flaps are designed to give additional lift, this allows you to fly slower. It also increases drag.
What kind of airplane are we talking about?