What you call a slight wobble at the top shaft is the reason that I went to the CNC bearing cups. I couldn't get the plasitc bearing cups to seat squarely. If you have wobble at the top, that wobble messes up the fly bar as it spins. The flybar may look fine at rest, but at speed you aren't spinning it centered on the shaft. You are spinning it in a wobbling circle and that will bend the flybar at speed.
If you have a digital camera, spin up just the upper rotor in a dark room and take a photo of it from straight above using the flash. The flash should freeze the image with a little blur showing the actual track of the upper hub. Or don't bother and just take my word for it. At any rate, the wobble is very likely bending your flybar at higher RPM. Is that the entire cause of your problem? I don't know. But I do know that a flybar that's not running true FOR ANY REASON is an issue - and it's not making your life any better at all.