You can get too small
The smallest single outlet for a 5" Dynamax fan that I have seen used is 3 3/8". The JMP Starfire used this size. I've built a couple of Regal Eagles with a single 3 1/2" outlet. That seemed to be a good compromise between speed and static thrust (for that particular plane used off of grass).
2 5/8" seems to be the magic number for a bifurcated pipe. I built one that was 2 5/16" for an F-4 one time and it had just enough power to get off the ground. But hard as I tried, I couldn't keep it off the ground very long.