If you don't sand the first couple of primer coats down until you see "spots and lines", you're not actually filling the weave and valleys, you're just thickening the paintjob.

You really should sand until most of the primer is gone. You want to see primer in the weave as spots. You really should see mostly weave with no primer hiding it. It's important to have the first couple of coats of primer being more gone than present after sanding.