It looks like you've made excellent choices of hardware. The aileron horns look especially good. They'd be even better if they were longer, but what the heck.
You ask about using metal servo arms. No real advantage since that application really doesn't need that much strength.
and flaps were still down a few millimeters neutral. Could this contribute to the flutter I experienced?
Actually, yes, but maybe in a surprising way. Every aerodynamic structure will have a specific flutter speed, even a bowling ball. The "airfoil" that resulted from the deflection could have been in "flutter speed range" at the speed you saw the flutter, whereas a "straight" orientation could have been outside that speed.
BTW. Every surface has a flutter speed. But sloppy linkages introduce new and wider flutter speed ranges for those surfaces.