Daniel H. Wells mentions Albert Smith's reaction to Joseph's marriage to Esther Dutcher.
Daniel H. Wells, Letter to Joseph F. Smith, June 25, 1888, [2]-[3], MS 1325, Church History Library
I write you now however in regard to a matter of business, a brother Albert Smith, who with his son Azariah Smith were in the Mormon Battallion, and are now living here in Manti, desires to be adopted into the family of Father Joseph Smith the Patriarch. As there was no one here representing your family, I told him that I would write to you in relation to this matter. Brother Smith is quite aged and feeble, but an old and staunch member, and was acquainted with the Patriarch, and has a large genealogy. . . [H]e is also much afflicted with the loss of his first wife. It seems that she was sealed to Joseph the Prophet in the days of Nauvoo, though she still remained his wife, and afterward nearly broke his heart by telling him of it, and expressing her intention of adhering to that relationship. He however got to feeling better over it, and acting for Joseph, had her sealed to him, and to himself for time.