I think it is a little extreme to say that you aren't controlling the plane if you aren't using rudder. I have a park flyer that is just ailerons and elevator, does that mean that this plane is uncontrollable? What about delta wings? Are they uncontrollable?

I am not trying to start an argument or anything, but I think that using rudder while just flying around and having fun is not all that necessary most of the time. The OP said that he uses the rudder in aerobatics that demand it, so he is using rudder, just not while flying the pattern, which is alright as far as I'm concerned.

Now, with that being said, some planes do need rudder all the time to make coordinated turns, other planes are very forgiving with just using ailerons and elevators. The T-28 is one that can just use its ailerons and elevator for nice turns.