It absolutely matters. Some of my larger engines (usually the ones that I flip-start) will start backward. Although they run backward, they have no power and no real accelleration capability.

If you need to setup an engine in a pusher configuration, you need to buy a pusher prop to get reverse thrust.