The swash-plate should not move up and down if you remove the rubber tube. The swash-plate is suspended at a constant height from the flybar-link. the only thing that might cause the swash-plate to move slightly up and down is the binding in the ball-links that's common with these helis.

The only advantage of having the rubber tube in my opinion is the extra crash-protection. The rubber will help absorb the vertical force of the mast in an impact. I changed it with a flanged wheel-collar so that it only touches the inside edge of the ball bearing.