In 1830 Edmund Wakefield Meade, connected to the Waldos by marriage, had taken the additional name of Waldo. He inherited the Castle in 1841 but had no love for Hever. He lived in the north of England until buying and restoring Stonewall Park, three miles from the Castle. Hever became a working farm, being leased out to a series of tenant farmers.
The last tenants of the Castle were Captain Guy Thomas Saunders Sebright and his wife Olive who leased it in 1895 from Edmund Waldo, the son of Edmund Wakefield Meade Waldo. He was succeeded by his son, Edmund Gustavus Bloomfield Meade Waldo, the famous ornithologist. His wife, Ada Coralie, believed the Castle was haunted and complained that Anne Boleyn’s ghost made it unpleasant for her. At this time the Castle was uninhabitable: the roof leaked and damp seeped up from the moat. However, Captain Sebright saw beyond the problems and lovingly set about its restoration with a view to buying it from Meade Waldo. Sebright never completed his work for, to his fury, Meade Waldo chose to sell the Castle in 1903 to the wealthy American, William Waldorf Astor.