Charlotte Amalie was founded in 1781, when four artisans built homes next to the new Fort Christian and were granted licenses to operate inns. The settlement was named after the Danish queen, but for the first century of its existence, Charlotte Amalie was better known as Taphuus, or pub. That probably says a lot. From the early days, Charlotte Amalie was a lively place. Early visitors described it variously as a freewheeling and exciting place, or as a den of pirates and scoundrels.

Charlotte Amalie grew steadily after its establishment. Warehouses were built along the waterfront on long, narrow lots, which allowed for the most efficient use of the valuable waterfront. Homes were built on the hillward side of Main Street, and the residential portion of town steadily expanded up Denmark and Government Hills, and in the valley between them. Authorities did not manage this growth, and many homes were built very close together along narrow, precarious lanes. In the mid-1700s it became clear that additional residential space would be required in the town, and Danish authorities subdivided the valleys west of Denmark Hill and east of Government Hill.

Many of the early residents of Charlotte Amalie were free black people. These former slaves often bought their freedom, although some were manumitted by their owners. Many were skilled artisans, and others started shops that catered to the demands of a growing town. Savanne, the area west of Denmark Hill, near the market square, was the heart of the free black population. Today the area is called Savan (sa-VON).

The absence of a building code and general unplanned nature of Charlotte Amalie made it susceptible to fire. No less than six fires destroyed more than half the town between 1804 and 1832. Eventually, following the last of these devastating fires, authorities put in place a restrictive building code that prohibited wood buildings on Main Street and required that all new roofs be covered with fire resistant materials. Many of the buildings still standing today date from the period of rebuilding after 1832.