A petition is calling for people from varied backgrounds to sit alongside Sir Martin Moore-Bick, who is heading the inquiry.
The campaign has been backed by families of around 50 victims, along with Grenfell United, an association of survivors from Grenfell Tower and nearby Grenfell Walk.
The inquiry into the west London disaster, which killed 71 people in June, is being supported by a legal team, civil servants and three assessors who have community work backgrounds to provide advice.
Local people have continued to express their misgivings over the appointment of Sir Martin, who they believe lacks first-hand experience of the complex cultural factors surrounding the tragedy.
The council-owned block where the fire broke out was home to various social classes, religions and ethnicities.
:: Grenfell: Insulation industry awards cancelled following Sky News investigation
Adel Chaoui lost four relatives in the blaze - his cousin Farah Hamdan, her husband Omar Belkadi and two of their daughters, eight-year-old Malak and six-month-old Leena.
He said: "It's not about ethnicity. It's nothing to do with whether you're black, white, Arab, whatever - it is to do with experiences.
"Sir Martin is very, very good at what he does, but he does not necessarily understand us.
"At the same time, we are up against these industry bodies that are spending millions of pounds on legal resources that we are never going to get anywhere near.
:: Christmas for Grenfell: Fundraiser to buy presents for children affected by fire
"The only thing we have got is the reliance on a judge, supported by a panel that have a diversity of experience that will treat us impartially and fairly."
Sir Martin's decision to appoint three assessors with backgrounds in community work was widely seen as a concession to those concerned about his suitability.
The petition is calling on Mrs May to use powers under the Inquiries Act 2005 to make the inquiry panel-led to avoid the risk of losing the support of families and residents.
Mr Chaoui warned that unless the format is changed he and others were unlikely to attend the inquiry, which is due to begin hearing evidence in the new year.
He added: "I'm really hoping the Prime Minister sees all we're asking for is a fair crack at justice."
But Karim Mussilhy, who lost his uncle Hesham Rahman in the fire, believes the appointment of assessors does not go far enough.
He said: "No matter what the assessors recommend, it is up to him what he reports back to the Prime Minister - the fact that that gets left up to one person doesn't seem right, it doesn't give us any confidence."
A government spokesman said: "The Prime Minister has given a commitment to consider the inquiry panel after the chair determined what further expertise he required, and this process is ongoing.
"We would like to assure all those affected by the tragedy that legal representatives of core participants will receive all relevant evidence, be able to offer opening and closing statements at hearings, and will be able to suggest lines of questioning for witnesses."
The petition can be found here.