What make/model boat are we talking about? It would be helpful to have you post a few pictures of your boat including the servo installation. Is this original servo or did you install it? If so was the original sail servo a drum or standard arm style servo? There are some difference in the geometry involved when a change is made from one style to another.
Let's take a look at the performance specifications for this particular "winch" style servo. Here is a link to a data page from Servo City that provides those details.
This winch should produce 3.5 rotations according to the specifications. Suggest you confirm you are getting the full 3.5 rotations the specs call for as the first step. If you have a table fan you can simulate the wind and sail load using a fan. Make sure you have the boat secured so the fan's force doesn't knock the boat over on it's stand.
You can also check to see if your sheets (lines that control the sails) are the correct length to permit full extension and retraction. Do you have any tensioning devise to keep the sheets from becoming slack - which can result in the lines fouling inside the drum -
I will wait to here back once you determine if the winch actually rotating per specs.