Imagine having to hold trim while flying something like a WWII Bf109. I think at least the first few models had no pilot adjusted rudder trim. I know of one memoir that mentions how tiring it was to fly because of the lack of trim.

BTW, there are a number of modern aircraft that use spoilers instead of ailerons. Want to turn left? The spoiler on the left wing comes up and the airplane rolls in that direction and yaws in that direction. Most efficient way to turn left.