The earliest known inhabitants of Bay Terrace were the Matinecock Native Americans, a tribe of the Algonquin nation. The tribal name Matinecock, meaning "hilly country", described the surrounding landscape, an area that may have been given to the tribe by the neighboring Lenapes.
In 1639, Dutch Governor Willem Kieft purchased the land from the Matinecock that today encompasses Queens County. William Lawrence, who served as a magistrate under the Dutch and English administrations, was granted a parcel of land by King Charles II in 1645 that included a large portion of what is today the areas of Bayside, Whitestone, Fort Totten and College Point. Bayside developed from an agricultural community to a suburb when the North Shore Railroad was extended in 1866. During the following several decades, the Bayside Land Association purchased farms for development. Bay Terrace, originally included within the bounds of Bayside, remained composed of farms and large estates until the 1950s, when Cord and Charles Meyer sold their 225-acre farm for development.
Bay Terrace was the name given in 1953 to the cooperative housing development built by the Norman K. Winston-Holzer Associates, because of its proximity to Little Neck Bay.
On February 20, 1981, the Bay Terrace library opened in the modern city-owned building at the present site.