Sure, you can go smaller. Just watch that the openings for the prop blades in the spinner cone DO NOT contact the prop blades... bad juju there. If you're really wanting to just drop the spinner but continue using a starter, Du-Bro makes a cone shaped hub nut that works perfectly with starters.

Andy