Prince Harry stayed at Frogmore Cottage during short visit to UK — and didn’t see family