The timing is slightly different when you do them inside and I find most people including myself are late on the rudder input which causes you to loose altitude quickly. Then you panic and bail out instead of trying to fight it.
It all comes down to practice and I think if you remember how much time you spent learning outside rollers and put in the same focused time on the inside rollers then it would start to be a lot easier. I think most just try opposite once in awhile and still tend to focus on the one they are better at.
For what it's worth I can tell you I have chatted with many who started inside and find them a lot easier than outside rollers