What I ended up doing is sanding as much as I could without ruining the surface, priming and sanding with two coats of primer. That evened the surface out enough for the final coats.
Thanks