Why is it considered offensive for some races to be refereed to by the colour of their skin?

but it's practically mandatory to identify some races by the colour of their skin, or rather whatever colour is closest to their skin.