Rumors of Mormon sexual license were circulating by 1835, when an "Article of Marriage" published in the Doctrine and Covenants said that Church members had been "reproached with the crime of fornication and polygamy." Coming from faithful Mormons, this evidence of marital irregularities cannot be ignored, but neither can it be taken at face value. From the Munster Anabaptists of the sixteenth century to the camp meetings of the nineteenth, critics expect sexual improprieties form religious enthusiasts. Marital experiments by contemporary radical sects increased the suspicions. John Humphrey Noyes, founder of the Oneida community, concluded that "there is no more reason why sexual intercourse should be restricted by law, than why eating and drinking should be." With old barriers coming down, people were on the lookout for sexual aberrations. What, if anything, lay behind the accusations of the Mormons is uncertain. They were apparently on edge themselves; the seventies resolved to expel any of their members guilty of polygamy.

No one intimated in 1835 that Joseph's actions caused the rumors. The sources written before 1839 indicate that most Church leaders knew nothing of a possible marriage. What they did know is suggested by the minutes of Oliver Cowdery's excommunication trial before the Far West High Council in April 1838, one of the few contemporaneous sources. Cowdery, long Joseph's friend and associate in visions, was a casualty of the bad times. In 1838, he was charged with "seeking to destroy the character of President Joseph Smith jr by falsly insinuating that he was guilty of adultry &c." Fanny Alger's name was never mentioned, but doubtless she was the woman in question.