David Goldenberg explains how Ham was misread.
David Goldenberg, The Curse of Ham: Race and Slavery in Early Judaism, Christianity, and Islam (Princeton: Princeton University Press, 2003), 156
To summarize the results of this chapter, the biblical name Ham does not mean “dark, black” or “heat.” The name is not related to the Egyptian km(t) ‘black’ nor to the Semitic words for “dark, black” or “heat.” Ham, which begins with ḫ, derives from a different root than these words, which begin with k or ḥ respectively. When Hebrew was put into written form, however, the two different phonemes ḫ and ḥ were represented with one graphic sign (the letter ח) and, as a consequence, the distinction be- tween the name ḫam (Ham) and words based on the roots ḥwm ‘dark, black’ and ḥmm ‘heat’ was lost, and it was eventually assumed that these words were related since they were all written with ח. We don’t know when this assumption first occurred, but we begin to see the confusion with the word for “heat” in the first century and with the word for “dark, black” somewhere between the second and the fourth centuries.