I think it's because down flaps essentialy change the incidence of the wing to a more positive angle, thus creating more lift, which lifts the nose, which needs to be counteracted by down elevator. Put an incidence meter on a plane with flaps fully extended, and compare the incidence with the flaps retracted- big increase in incidence with flaps down. Either that, or it's something else.