When idling, the pressures and velocities involved with the scavenging period of a fourstroke are pretty low. At the time the intake opens (begins to open, to be more precise), at idle the exhaust period is almost ending, and because of low combustion pressure and low RPM (longer duration or the same upward piston movement) the pressure in the cylinder at the moment of opening of the intake valve is basicallly equal on both sides. So not much going on there.
Because of the low combustion pressure and the slow piston movement, there is not much velocity in the exhaust gas either. So very little inertia.
So the intake opens, and basically "nothing" happens.
Now the piston passes TDC, and begins its downward movement, Both valves are open at this moment. The piston begins to draw in gas, through both valves. BUT.... the throttle is closed to idle position so the intake offers great resistance, and the gas that can flow in easiest is that from the exhaust. So there, the flow reverses. Then the exhaust valve closes, and the inrushing gas "bounces" against the closed exhaust valve.
The "poyt" sound is the reversal of the gas in the muffler followed by the bounce of the pressure wave when the valve closes.

When the engine picks up speed because the throttle is opened, there will be more velocity in the exhaust port and because the restriction in the intake tract is gone, the gasflow will be unidirectional as long as the exhaustvalve is open, There is no more reversal of gasflow after TDC, no more bouncing of the pressure wave against the closing exhaust valve, and the "poyt" sound dissapears.

That is how I can figure things out if I play the movie of the process in extreme slo-mo in my head.