According to reports, Prince Andrew is once again causing problems for the royal family. In addition to Prince Harry and Meghan Markle’s eagerly watched attendance selections, King Charles III. is apparently concerned about dealing with a full-fledged royal squatter in the form of his younger brother.

Despite Charles’ persistent pleading, Andrew has reportedly refused to leave his longtime royal mansion, the $37 million Royal Lodge. Since 2004, the Duke of York has been at the Windsor home. He moved in after a round of renovations that were finished in the wake of the Queen Mother, the previous resident. Nearly 20 years ago, he and his two daughters as well as his ex-wife Sarah moved in together.

Now that Andrew is no longer a senior royal, following the legal scandal surrounding his relationship with Jeffrey Epstein and a rape allegation against him, Charles has now reportedly offered his brother Frogmore Cottage, the home the king allegedly evicted Harry and Meghan from in the past January. However, Andrew is reportedly refusing to move to the relatively smaller royal estate.

A source tells page six that “relationships between the king and his brother have never been so bad” and that the whole discussion “turned into a real fight that [Charles] never expected and it made him tired and angry.”

The Royal Lodge was built in 1662 as a Queen Anne style brick cottage on a 98 acre estate. The original cottage was later demolished and replaced in the 18th century by a three storey stucco fronted house. The main building, a 30-room, seven-bedroom house, has remained relatively unchanged since the 1930s, although Andrew ordered renovations to bring the property up to date in the early 2000s. The property also includes a royal chapel (where Princess Beatrice recently hosted her wedding), a gardener’s cottage, six lodge cottages and security quarters.

A general view of the Royal Lodge at Windsor Castle, England, on April 11, 1942 Studio Lisa/Hulton Archive/Getty Images

The Royal Lodge is three miles south of Windsor Castle, just a little further away than Adelaide Cottage, Prince William’s current home, Frogmore Cottage, the home offered to Andrew and Frogmore House. With Adelaide Cottage being a much smaller estate than Royal Lodge, Prince William has reportedly viewed Andrew’s residence as the one he wants to take next in line to the throne – but with Andrew refusing to vacate the premises, those plans have been indefinitely time put on hold.