Edward Waldegrave was succeeded as 2nd Baronet of Hever Castle in 1644 by his eldest son Henry. Henry Waldegrave died in 1658 and was succeeded as 3rd Baronet by his eldest son Charles. Charles was to be the last of the Waldegraves to use Hever as a refuge from Catholic persecution. He died in 1684 and his son Henry inherited the Hever estate. Henry married Henrietta Fitzjames, the natural daughter of James, Duke of York, and Arabella Churchill.
When the Protestant William and Mary came to the throne the future for the Catholic Waldegraves looked bleak. Henry Waldegrave’s son, James, had learned a valuable lesson. He took the Oath of Supremacy, conformed to the Church of England and took his seat in the House of Lords. It proved to be an advantageous move but the little castle at Hever proved too small to match his growing status and it was sold in 1715 to Sir William Humphreys.