In 1757, Martin Howard, Jr., a lawyer and ardent Loyalist, bought the house. In 1765, Howard and other members of a Tory group known as the Newport Junto wrote a widely-read pamphlet in which the opponents of the crown were criticized for their disregard of Royal and Parliamentary authority. As a result, during the Stamp Act riots in Newport later that year, Howard and two other Junto members were hanged in effigy on Washington Square and a crowd attacked and vandalized their houses. Howard fled Newport under the protection of the British.

That same year, John G. Wanton, a prosperous Quaker merchant, purchased the house at auction. In 1782, Wanton's daughter, Polly, married Major Daniel Lyman, a lawyer and Revolutionary War veteran. In 1807, the Lyman's second daughter, Harriet, married another Newport lawyer, Benjamin Hazard. The house remained in the Hazard family until 1911, when Benjamin's daughter Mary Hazard died.

The house grew with each generation of residents. Originally it was built with a single room on each side of a central chimney. It was a full two stories with a garret. Sometime before 1725, a lean-to kitchen was built at the northeast end of the house. When Martin Howard bought the house he installed fine paneling and handsome moldings. These changes were in keeping with the high-style, genteel taste that was so important to refined Newporters during this period and identified Howard as person of social standing. The Lyman family, which needed more room as it grew to include thirteen children, added a large two-story addition to the rear of the house about 1785.

Throughout its history, the Wanton-Lyman-Hazard House reflected the tastes and aspirations of a thriving seaport town. It is a truly vernacular house, with each modification based upon the background of the owner, the needs of his family, the simplicity of a Quaker dominated town, and the formal architectural influences of a commercial and cosmopolitan entrepot like Newport.

In 1927, the Newport Historical Society purchased the run-down property and retained Norman Isham, a noted restoration architect, to stabilize and restore the building. Isham, who also worked on the Newport Colony House, Trinity Church, and the Brick Market, was a pioneer in the field of historic preservation, and his mark is still clear within the house. Rather than restoring the property to a single period, Isham chose to reveal many styles and periods, providing a unique look at the evolution of one of Rhode Island’s earliest homes. The early steeply pitched roof and plaster cornice still remain, as does the 1720s kitchen. The addition made by the Lymans in 1785 was removed, and the bedroom on the second floor was restored to its 17th century appearance.