Ka'b al-Ahbār explains Ham knowing his children by their black skin color.
Ka'b al-Ahbār, in Qisas al-anbiya, ed. Isaac Eisenberg (Brill: Leiden: 1923), 101
When Noah died, Ham lay with his wife; and God opened his gaul-vesicle and that of his wife also so that they mingled and she conceived a black boy and girl. Ham despised them and said to his wife, “They are not mine!” They are yours!” said his wife, “for the curse of your father is upon us.” After that he did not approach her until the children had grown, when he again lay with her, and she bore two more black children, male and female. Ham knew that they were his, therefore he left his wife and fled.