Are there any drawbacks to using a castoring tailwheel as opposed to a steerable other than windy taxi conditions ??

I am building my first tail dragger (4* 60) and the tailwheel mounts directly to the rudder which I am a little leary of.
Maybe I shouldn't be concerned. Is there much stress on the tailwheel ??

Thanks !