I'll say flying is like riding a bike. if all you do is ride on a nice smooth path, normally you don't have any problems. The problems arise when you try to ride a wheelie across a fallen log spanning a raging river, 200' bellow.
Once you learn, and flying becomes reaction instead of concious thought, then you surely can get back to it with little or no problems.
I hadn't really flow but a handfull of times in the past 4 years, this summer I finally finished a Sig Cap231ex. Went out for the maiden flight, no one was around so I fired it up, took off, landed, remover 1/2" of elev travel, checked my nuts (and bolts). Went back up 3 or 4 more times and was flying well enough. Yes i was a bit rusty yet, but nothing a couple gallons of fuel wouldn't cure.
If your plane crashes on you first instruction flight, then you have a very bad instructor. I've had guys trying my Zen 50 who had never touched a control. A few of them could handle it in a race trac pattern, but it was a bit hot for most of them.