That is usually a sign of being too rich on the low end.

At WOT the mixture is good but the needle valve is set slightly lean to compensate for the extra amount of fuel coming in from the low end adjustment, so when you throttle up, the transition is good.

Now you go back to idle, and the mixture slowly richens.

Try adjusting the low end by 1/8-turn increments. You'll have to richen the high end slightly each time too