My theory is that those surviving the Middle Passage had to be very intimidating in intellect and physical prowess. So the white reaction was fear, which is still being passed down, but it was twisted around such that white men (in particular) made it sound like slaves were brutish and didn't do things like invent the cultivation techniques of rice along the Carolina coast.

I did a lot of research for my alternative history novel on a much earlier emancipation date, and you're right about Caribbean slaves.

Part of the Barbados Slave Code (and therefore official policy):

“If any Negro or slave whatsoever shall offer any violence to any Christian by striking or any other form of violence, such Negro or slave shall for his or her first offense be severely whipped by the Constable.

“For his second offense of that nature he shall be severely whipped, his nose slit, and be burned in some part of his face with a hot iron. And being brutish slaves, they deserve not, for the baseness of their condition, to be tried by the legal trial of twelve men of their peers, as the subjects of England are.

“And it is further enacted and ordained that if any Negro or other slave under punishment by his master unfortunately shall suffer in life or member, which seldom happens, no person whatsoever shall be liable to any fine therefore.”