Yep you missed something.

Your plane is pretty much always flying aligned with the airflow until you poke the rudder in. Wind or no wind when you poke the rudder in you yaw the plane to one side, the fuse generates a sideways force which changes the flightpath and when you let go of the rudder the plane points back into the airflow again and your plane continues along the new flightpath. If the airmass is traveling across the ground at an angle to your flightpath (ground track), you won't be going to where your nose is pointing.

If in a "crosswind" you start out flying towards something with your nose pointing at it, you will drift to one side and need to apply rudder to keep the nose pointing where you want to go, but you're flying in an arcing groundtrack (your compass will be moving), NOT straight (compass not moving) which is why you need to hold the rudder in. Guys that fly long distances please ignore the compass moving/not moving references ;-)