From cnbc.com:

Location: Old Town, San Diego, CA
Notable ghost: “Yankee Jim” Robinson

In 1849, as news of the Gold Rush broke, young Thomas Whaley moved from New York to California and opened a hardware store in San Francisco. Arson destroyed his business in 1851, so he moved to San Diego — the present day Old Town San Diego — where he set up general store businesses. Always the entrepreneur, he started a brick-making business and used those kiln-fired bricks to build a granary. Then, in 1857, he built an adjacent two-story Greek Revival brick building where he and his wife, Rachel Pye, lived. It was considered the “finest new brick block in Southern California” by the San Diego Herald, and cost $10,000. The walls were finished with plaster made from ground seashells.

The site of the house is also where gallows once stood and where “Yankee Jim” Robinson was hanged for attempted grand larceny. Whaley reportedly witnessed the hanging, but was not fazed by it, since he bought the property a few years later, removed the gallows, and built the Whaley family home on the site. Shortly after moving in, heavy footsteps “from the boots of a large man” could he heard throughout the house. Whaley concluded it was Yankee Jim, whose spirit is “alive and well ” two centuries later.

Two later tragedies occurred in the house: the Whaleys’ second child, Thomas, Jr, died at 18 months of scarlet fever and their fifth child, Violet, committed suicide in 1885.

After this tragic event, Thomas Whaley built a single-story frame home for his family at 933 State Street in downtown San Diego. Attempting to capitalize on the boom in that area, he maintained a real estate office at 5th and G in the First National Bank Building, with various partners. After retirement from business in 1888 due to ill health, he died at the State Street address on December 14, 1890.

Thomas Whaley also had some prominent family history: His grandfather, Alexander Whaley, supplied George Washington with badly needed muskets during the American Revolution’s Battle of White Plains and his mother, Rachel, made some shrewd real estate deals including buying “Sheeps Meadows,” which was used as grazing land in New York City. It is now known as Central Park.

The Whaley House is one of two homes in California sanctioned by the U.S. Commerce Department as being haunted, with the Winchester House in San Jose being the other.

Located at 2476 San Diego Avenue in historic Old Town San Diego, the Whaley House stands today as a classic example of mid-nineteenth century Greek Revival architecture. Formally dedicated as a historic house museum on May 25, 1960 and open to the public ever since, it is one of southern California’s most popular visitor destinations. Over 100,000 people visit the Whaley House annually.