Walpole left the house to his niece, Lady Waldegrave, who was living there in 1800, as is evidenced by the Hon. Grantley Berkeley, who said that she was the occupant of the residence around the year of his birth (circa 1800).
This Lady Waldegrave was interested, as a relative, in the so—called Seymour case, which arose about the disputed guardianship of Mrs. Fitzherbert over Mary Seymour, the youngest child of Lord Hugh and Lady Horatia Seymour.
The executors of Lord Hugh did not wish Miss Seymour to remain under the care of Mrs. Fitzherbert, simply because there were relatives of her own able and willing to take charge of her, and they named Lady Waldegrave as being one of these.
As Mrs. Fitzherbert had been specially asked by the mother and father to extend her protection to their child, and as both she and the Prince Regent had become very devoted to Miss Seymour, a great deal of controversy took place, which resulted finally in Mrs. Fitzherbert winning the day.
In a letter from Mrs. Fitzherbert to Mrs. Browne dated 17th June 1806, telling the happy news, she mentioned the fact that "Lady Waldegrave came to town on purpose, wrote to all the press to support her, turned out the people she had let her house to in Berkeley Square and fixed herself in it, telling everybody my poor child was to go to reside with her on Saturday evening."