Have you thought about using helicopter gear parts? Just a quick impromptu idea, put the prop on a short axle with a smaller toothed wheel, put a wheel of the same size on the engine. Connect these using a longer axle with two bigger wheels at the ends. All of these parts would have their own vibration, but maybe it woulsd sum up to less than the vibration a long high-rpm axle. Maybe the play between the toothed wheels can swallow some of the vibration. Or maybe you could use belts instead of toothed wheels?

[edit] Talking about engine cooling, maybe you could include a heli cooling fan also.