I flew ultralights and was a BFI - basic flight instructor and I will admit that it was a definite disadvantage. As already noted, flying from outside the cockpit is more difficult. Sure, I know W&B, CG/CL, and the aerodynamics of flight, but controlling the plane from the ground was a new experience for me. Many traits do carry over though. I am very meticulous about pre/post flights, I exercise caution at every trip to the field, and while flying, etc.
The hardest for me was getting used to flying with the plane coming towards me. I used to wear my Excalibur ultralight and flying was fun because I could sense and feel every maneuver, but not I have to see the maneuver to feel it.
I don't think it is a permanent disability , it will just take me more stick time than most. The simulators do help.

Good luck!