A doctor would have been interviewed by police for rape and child cruelty over allegations a "truth drug" was used to carry out abuse at a hospital.
The late Dr Kenneth Milner ran Aston Hall psychiatric hospital in Derbyshire from 1947 to the 1970s, which former patients described as "pure hell".
A report by police found allegations children were drugged to a zombie-like state, stripped and abused.
Police said there would have been "sufficient evidence" to pursue a case.
The Derbyshire force, under the direction of the area's safeguarding children's board, started an investigation in February 2016 after concerns were raised about the hospital which closed in the 1990s.
Allegations first came to light in 2011 then 2014 and 2015. No allegations of sexual abuse were identified in 2011 by police.
Then in July 2014 an individual made an allegation of rape followed by another person alleging abuse at the hospital in September of that year.
In July 2015 another person came forward and said she was injected, gassed and woke up feeling sore between her legs while at the hospital.
Aston Hall was opened in the 1920s for those deemed to have mental health problems. It could cater for about 100 children of both sexes, but took adults as well.
Allegations centred on the use of sodium amytal as part of "narcoanalysis" - the uncovering of repressed traumatic memories.
But stories emerged, in part via social media, of children being isolated, stripped and abused, both sexually, physically and mentally.
Police collated 115 witness statements, recorded 77 crimes and heard from 65 alleged victims.
Dr Milner died in 1975 and police said no inference should be drawn from the report other than the doctor would be questioned if he were here today.
After trying to take his own life Jason Stubbings, from Cornwall, was sent to Aston Hall in the 1960s, aged 14, where he said he was injected with drugs and abused.
He said he remembers being injected with what he now knows is sodium amytal, also known as the truth drug, as well as another drug which is still unknown to him.
"Then (the doctor) placed a pad over my face and dropped ether underneath my nose," he said.
Mr Stubbings added that he was then taken back to a memory where his brother was beating him with a garden cane and it felt like he was being hit at the time.
When he woke up, he found his hands had been tied and he had marks on his body.
Mr Stubbings also remembers looking out the window once and seeing "a guy in the exercise yard in a straightjacket screaming" and thinking he did not know if he was ever going to get out of there.
"It said on the administration form 'a slight timid boy', and they did that to me," he added.
The report said there were consistent allegations of drugs being given to restrained children, with limited or no records being taken.
Current medical opinion was these methods were not acceptable, even by the standards of the day, it added.
But officers said the length of time since the alleged crimes had proven a major issue - to the extent they had been unable to find out whether some members of staff were still alive.
The report said: "Having regard to these difficulties and the scale of the alleged physical and sexual abuse that has emerged in the evidence; we do not consider that it would be either practicable or appropriate for us to attempt to reach firm conclusions on each specific allegation.
"There would have been sufficient evidence to justify interviewing Dr Milner under caution in relation to a number of potential offences."
The report said the offences would have namely been "rape, indecent assault contrary to the Sexual Offences Act 1956, Child Cruelty and Assault contrary to Children and Young Persons Act 1933 and Offences Against the Person act 1861 respectively".
While it noted children were sent to Aston Hall from a variety of homes, the report found no evidence of a conspiracy to supply children for abuse.
Barbara O'Hair, who claims she was abused by Dr Milner in the 1970s, told the BBC the report was positive.
"It gives recognition to what happened to me and others at the hospital and I'm very grateful to the police," she said.
At least 30 former patients who have submitted a claim for compensation to the Department for Health through a solicitor. Other claims are also believed to have been submitted.
A spokesman for the department has said the matter was under investigation.
Follow BBC East Midlands on Facebook, on Twitter, or on Instagram. Send your story ideas to eastmidsnews@bbc.co.uk.