You are a victim of torque especially if you fly a taildragger on take off. To begin toe in the main gear like this / \ (top is forward) just slightly. Secondly, you said it all when you said the rudder was straight. You have to allow for torque both on take off and hand lauching. The prop wants to turn one way and the airplane in the opposite direction
"torque". The part is true on ALL aircraft except jets and aircraft with contra-rotating propellers. Right rudder, just the amount to counteract the torque, will be necessary no matter how you launch. If you do not have a steerable tailwheel for takeoffs you have added to the problem. Hand launch still requires right rudder.