It isn't racist to acknowledge that the white man kept the black man enslaved for two hundred years or more in the Americas, kept him down, poor, and disenfranchised until well into the 20th century, and that we're still battling generations of inbred hatred, distrust, and denial as a result of that sordid past.
It's called living in American reality.