Who knows what great art you might have created. Color isn't everything in art. It's not even the main thing.

By the way, one of the weirdest color perception phenomena I've heard of is where people "see" different letters of the alphabet in different colors. This condition is called "synaesthesia." Some with this condition have claimed it has actually helped them remember words and spellings.

http://otherthings.com/uw/syn/

So how would this work if these people spoke (or learned) a language with a non-alphabetic script (such as Chinese with its thousand of characters) or scripts where each letter has three different shapes (initial, medial, and final) as in Arabic?