But if the plane is flying a perfect vertical downline, and the incidence is 0, etc. adverse yaw would not be a factor since the wing is not generating a net lift (the lift - and therefore drag - associated with each aileron would be equal and opposite).

And, if the vertical CG was not aligned with the aerodynamic center, as you put it, there would by a pitch or yaw present when flying the vertical downline without rolling, right? But it only appears when the ailerons are deflected. Otherwise, it tracks a virtually perfect downline.