A lot of people seem to implicitly think that racism, misogyny, and homophobia are fairly rare things in our society. The corollary to this is that racist beliefs are only held by racists, and racists are the sort of extreme aberrations from mainstream society who burn crosses on people’s lawns. So when you say that what person X said is offensive, they think “No way! Person X wouldn’t burn a cross.” I think this is where “it wasn’t intended to be offensive, so it must not be offensive” comes from.
This worldview presents a double whammy, because even if you can convince them something is technically problematic, it’s not related to any systematic problems in our society, so it’s still no big deal.