By saying what he said, obviously hinting at changing the original agreement, he placed you in an awkward position...his error, not yours. Put it all behind you. You did the right thing....end the discomfort.
"don't worry...be happy"