Elevator vs rudder
The reason the outside rollers are easier for you is because your doing it as a really bad slow roll...using the rudder mostly to make the turn .......This makes for a good start and you can heavy hand the rudder to carry you around.....
Rollers to the inside are not as rudder heavy the reason the proper rudder input to keep the planes altitude is also pushing your circle the wrong way....So heavy handed rudder inputs when going to the inside mean your not keeping it a circle..and rather your trying to go the other direction in the turn.....
USE MORE ELEVATOR.
When rolling to the inside of the circle all you have to make the turn is Elevator...because the rudder inputs go the wrong way for the turn.....Just use enough rudder to keep the nose up and no more and then lean on the elevator a little more.
Good Luck
Troy Newman