The FAA is like almost every other government agency - they lack funding and don't have enough staff. Look at the food industry - very important and watched by homeland security - a person "might" check on a food manufacturing plant every 5 years if they are lucky.
If the FAA declared all remote control planes illegal nothing would happen unless you are flying next to a police station and even then I doubt they would even know the federal law.
Keep flying and having fun...................
Keith