Hate speech is hateful: that is not open for discussion.
Yet to criminalize hate speech as vandalism would stifle discussion altogether. When pundits want to start drawing lines about speech, there is no end to the vilification that can stifle true and meaningful dissent into meandering name calling and race-baiting.
Every person who finds a comment on race or culture offensive can simply throw up a red flag and cry out: "Hate speech!"
This approach to protecting the feelings of others impinges on the well-being of the public square. Other than indicating which specific terms are inappropriate, to denounce terms and phrases based exclusively on the discretion of a community or a committee only opens the door to censorship by elites.
Racist epithets cannot hurt other people as long as they choose to identify themselves by who they are, not what other people think of them. Part of the reason why the "slow-acting" poison of voiced prejudice causes so much harm is that political figures insist on giving those terms power. We have inadvertently created a culture of victims who feel that they must be pampered before they step out to make a statement in the world.
Booker T. Washington did not attempt to overturn centuries of prejudice by demanding that they white man change. He instructed and prospered other people to make use of their skills and will that they possessed in order to make their way in the world, irrespective of who was in power, or what anyone else had to say.
We cannot base our identity and our esteem essentially on the empty rhetoric of the crowd or the press. Vandalism is a crime against property. Words may hurt, but they cannot break a man if he choose to repudiate them.