Image copyright PA

The security service MI5's handling of large amounts of personal data has been "undoubtedly unlawful", a watchdog said.

A High Court case revealed the Investigatory Powers Commissioner's concerns about data obtained under warrants.

Civil rights group Liberty said it involved the "mass collection of data of innocent citizens".

It said MI5 knew about the issues three years ago but kept them secret.

"MI5 have been holding on to people's data - ordinary people's data, your data, my data - illegally for many years," said Megan Goulding, lawyer for Liberty, which brought the case.

"Not only that, they've been trying to keep their really serious errors secret - secret from the security services watchdog, who's supposed to know about them, secret from the Home Office, secret from the prime minister and secret from the public."

Among the large amounts of data which can be collected by MI5 under the Investigatory Powers Act are individuals' location data, calls, messages and web browsing history.

As well as "bulk data" collection, which can include information about ordinary members of the public, MI5 can use targeted interceptions of communications and computer hacking for investigations such as counter-terrorism.

But the act includes safeguards about how all this information is stored and handled. It is against the law to keep data when it is no longer needed, or to store it in an unsafe way.

MI5 had a "historical lack of compliance" with the law in the way it retained and deleted data, said Lord Justice Sir Adrian Fulford, who oversees MI5's use of data as Investigatory Powers Commissioner.

He criticised "the undoubted unlawful manner in which data has been held and handled" by the security service.

And his ruling, made public for the first time, said the security service would be placed under greater scrutiny by judges when seeking warrants - which the commissioner compared to a failing school being placed in "special measures".

'Serious risks'

Documents presented in court showed that senior members of the security service were aware in 2016 that there were serious issues with the management of data.

In April this year, MI5 informed the Home Office and Number 10 of the concerns, but were criticised by the commissioner for not reporting them earlier.

Information from people's discussions with lawyers was among the data retained without a lawful basis, the court heard.

Liberty said that such material should be protected by legal privileges, but because the systems were failing it was being viewed by people at MI5.

Lawyers for MI5 said they could not explain the exact nature of the breaches in open court, not because they were "embarrassing" but because there were "serious national security concerns".

In a statement last month after he had been informed of the issues, Home Secretary Sajid Javid said MI5 had taken "immediate and substantial" steps to comply with the law.

Julian Milford, representing Mr Javid and Foreign Secretary Jeremy Hunt, acknowledged in court "the existence of serious compliance risks".

But he said that these specific issues were a "complete irrelevance" to Liberty's court case, which was challenging the legality of the whole system of information gathering created by the Investigatory Powers Act.