The thing is, no one living has grand parents who were slaves. There may be a few who had great-grandparents, but those people are pretty old. Another fact is not all blacks are descended from slaves. It's actually closer to 1 in 4.
Still, both the hecklers and Kramer were in the wrong. It was a group of idiots acting like idiots, nothing more.