The Lancaster Historical Society was originally constructed in 1895 on Broadway near the intersection with Central Avenue. The house was built by Dr. Samuel Potter and was used as his home and office. Dr. Potter's daughter Fanny, was married in the house on October 28, 1896. Fanny lived in the house until the 1930's. Fanny Potter Eaton died while on a visit to California on June 13, 1938. Fanny left her home to the Town of Lancaster.

The house was moved to Clark St. in 1940 and remodeled for use as the town's library. It remained the Library until 1973. The building was then used as the Senior Citizen Center.

The building has many notable architectural features including a round porch and a widow's walk. The museum contains many artifacts which chronicle the history of Lancaster, including a permanent display of items from the Lancaster Glass Factory.