First Presidency offers advice urging for members to wear garments more often than not.
Ezra Taft Benson, Gordon B. Hinckley, and Thomas S. Monson, circular, October 10, 1988, in Devery Anderson, The Development of LDS Temple Worship (Salt Lake City: Signature Books, 2011), 454-455
Church leaders and others including temple workers should refrain from giving unauthorized instructions and opinions concerning when the garment might be removed. . . The fundamental principle ought to be to wear the garment and and not to find occasions to remove it. Thus, members should not remove all or part of the garment to work in the yard or to lounge around the home in swimwear or immodest clothing. Nor should they remove it to participate in recreational activities that can be reasonably be done with the garment wore properly beneath regular clothing. When the garment must be removed, such as for swimming, it should be removed as soon as possible.