Some planes fly just great in the wind. One plane that I could recommend is the Sig Kougar. Its a nice little sport ship that can easily be flown in winds up to about 30 mph. Seasoned flyers can fly it in even higher winds.
Other planes are nearly impossible to control in 15 mph, and that is not fun alt all.
So, its not just you. The plane is important. Why fly a plane in the wind if it isn't fun ? Thats what kites are for.