Turns out I forgot I stripped out my image of all its color to only produce white when the selected colour was present! So instead of...