The Chesterfield Monthly Meeting’s first minutes date to 1684, although it is assumed that informal meetings took place prior to this date.  The year 1684 also marked the sale of six acres to Frances Davenport, a Crosswicks Quaker and New Jersey Assemblyman, who gave the property for use by the Quakers.  This six-acre tract is still owned by the Quakers, and is where the meeting house and First Day School are situated.

By 1692, a small wooden meeting house had been built on the Quaker property in Crosswicks, and in 1706 it was replaced by a larger one located at the north of the property, approximately where the Crosswicks Community House is located today.

From the very beginning, West Jersey Quakers had entered into a peaceful coexistence with the local Lenape Indians, and the Chesterfield Friends were no exception. In 1756 and 1758, the colonial court in Trenton convened two councils at the meeting house in Crosswicks to decide the fate of the Lenape, and it was at the final council that the first reservations were created, in what is now Shamong Township. The Lenape insisted that the meetings be held in Crosswicks, since it was a place where they had enjoyed mutual trust and fellowship with the local Quakers.

In 1773, using the newly built Buckingham Meeting House in Bucks County, PA, as a model, the Chesterfield Friends built the current brick meeting house at a cost of £3,750.  The structure took nearly three years to complete and was finally finished on the eve of the Revolutionary War in 1776.

Throughout the war, the meeting house was used to house troops from both sides, with the understanding that the soldiers were to keep the meeting free for Sunday worship by the Quakers.

On December 29, 1776, about 2,100 American Troops under General John Cadwalader stayed in and around this Meeting House. About 1 a.m. on January 2, 1777, Cadwalader received a letter from General George Washington, delivered by Dr. Benjamin Rush. The letter contained orders to march to Trenton to join other American forces there.  They arrived at about 7 a.m.. Later that day, the Second Battle of Trenton was fought.

In June 23, 1778, five days before the Battle of Monmouth, there was a skirmish here. During the fighting, the house was hit by three cannonballs. One of these hit the brick wall, leaving an imprint. This cannonball was saved and later mortared into the wall at the site of the imprint.