Technically, yes, it is possible for an engine to start without the glow driver attached, but it is very unlikely. You can find whole threads in many different forums dedicated to just this questions and get 1000 different viewpoints of whether or not it can happen and there are stories of it happening. I know somebody who it happened to so I know its possible. A good way to view your engines and never have to worry about it is... always treat it as if it could start. When you prime it, hold one blade firmly with your WHOLE hand and turn it over like that, you dont have to "flip" the prop with your fingers to get it primed, simply turning it through the compression stroke a few times with your hand is fine, you can see it suck fuel into the line and carb as you do it. If you are holding the prop firmly and turn it over even after its primed, with the ignitor on you can feel the engine "kick" with a light bump as you go past the compression and the fuel in the cylinder ignites. Also, you mention the fact that you value your fingers... get a chicken stick or electric starter or thick leather glove. If you do choose to prime it by "flipping" the prop as if trying to start it, do it with a chicken stick or an electric starter anyways as many of the nylon/composite props available these days are razor sharp and even turning it over slowly with one finger can and will slice you... even the little ones.

Hope this helps,
Brian