I fly until the second yellow (the one next to the red) comes on. This corresponds to a battery voltage of about 4.77 volts on my voltwatch, which should leave plenty of reserve.

Make sure you check the voltwatch after a flight (before turning off the switch). Once the switch is turned off the battery voltage will slowly start to creep back up and the next time you turn it on you will get a higher, misleading, voltage reading.