Solitude House and its associated outbuildings are historically associated with the Taylor Wharton Company, the successor firm to the Allen & Turner ironworks. The ironworks and the subsequent steelworks were significant in the industrial development of New Jersey. The house was most likely built between 1740 and 1776 and significantly remodeled in the Italian Villa style during the middle of the nineteenth century. Other features in the vicinity of the house include a lake and dam, the remains of a small hydroelectric plant, iron furnace ruins, other outbuildings, and site features.
In the decades before the war, Union Iron Works had produced such items as horseshoes, farm tools, and fireplace backs. With the coming of the war, the iron works became very important to the military effort because of the production of such items as cannon balls and rifle barrels.
When the Revolutionary War began, Turner and Allen had to flee their property because they were Loyalists. Bookkeeper Robert Taylor became the superintendent of Union Iron Works. Two Loyalists were imprisoned at Solitude House, John Penn, who had served as the last royal governor of Pennsylvania, and his Chief Justice Benjamin Chew. They remained here for seven months beginning in 1777.
Among the Revolutionary War figures who visited Solitude House are George and Martha Washington, General Lafayette, Colonel Charles Stewart, and Aaron Burr.