Don't forget that Friar Park belonged to a group of nuns for decades before George moved in. It was a school or something similar and during WWII, a haven for the children who were sent out of the big cities.
Nuns were even living there when George moved in. They freaked out when he put up a picture of Krishna in the great hall. He told them, "God has many forms," but I know my Catholic mom doesn't believe that.