One of the finest buildings in Sag Harbor  is the elegant Sag Harbor Whaling Museum (200 Main St., at Garden St., 631/725-0770, www.sagharborwhalingmuseum.org , 10 a.m.–5 p.m. Mon.–Sat. and 1–5 p.m. Sun. May–Oct., adults $5, seniors $4, children 6–13 $1), its entrance marked with tall Corinthian columns and the gleaming jawbones of a right whale (a type of whale). The mansion was built in 1845 by architect Minard Lafever for whaling-ship owner Benjamin Huntting.
Inside is a hodgepodge of jumbled exhibits—some absolutely fascinating, others looking suspiciously like junk—which makes exploring the museum a lot of fun. On display you’ll find everything from wooden boats and tools used by whalers to ostrich eggs and the needlework of a Miss Fannie Tunison, an 1800s Sag Harbor resident who was paralyzed except for her lips and tongue.
Around the corner from the whaling museum is the former home of 18th-century customs inspector Henry Packer Dering (Garden St., 631/725-0250 or 631/692-4664, 1–5 p.m. Sat.–Sun. May–Oct., adults $3, seniors and children under 14 $2). Dering’s job was to record all the goods entering the harbor and collect entry taxes. He used his front room as an office; the room is equipped with wooden window shields that Dering shut whenever he wanted to count money.
All along Main Street near the Custom House stand impressive sea captains’ houses, while nearby is the Old Whaler’s Church (Union and Church Sts., 631/725-0894, www.oldwhalerschurch.org , 9 a.m.–3 p.m. daily). Designed in 1844 by Minard Lafever, the church is built in an unusual Egyptian Revival style, and has a soaring interior where concerts are frequently presented.
About four miles west of Sag Harbor  is 187-acre Elizabeth Morton National Wildlife Refuge (784 Noyack Rd., 631/286-0485, sunrise–sunset daily, $4 parking). The refuge overlooks Peconic and Noyack Bays, and features sandy and rocky beaches, wooded bluffs, ponds, and nature trails. It’s also a nesting stop for the endangered piping plover. While the refuge is open year-round, much of the beach is closed April–mid-August.