First decision would be what frequencies you want to use.
If you fly 2.4 then not much point getting 2.4 video. If you want to buzz around fairly close (definitely line of sight) then teh 5.8 GHz would be OK but if you are after range then maybe 900MHz might be better. Make this decision early, and you won't have to change much gear as you progress.
1- for the plane : Camera and TX AND Pan/Tilt
The ability to easily fly FPV really happened with pan/tilt. You can try it with a fixed camera, but don't get discouraged, try it again with pan/tilt.
You can add OSD at any time, so just a camera, TX and pan/tilt will get you going.
2- for the ground : RX, display and head tracker.
FPV comes alive with the goggles. It places you in the cockpit to give that true virtual flight feeling.
The head tracker would not be needed if you don't use a pan/tilt, but you will if you want the experience to work for you.
So a good recipe for dissatisfaction is a fixed camera and trying to fly it on a dull computer screen. You could possibly do it, but I doubt it.
I have sourced all my gear from Range Video
http://www.rangevideo.com/
Good gear and good service.