But they "remain high", its adds in its latest weekly report.
In week four, the agency said there were 52.1 GP consultations per 100,000 in England, compared to 54.1 per 100,00 in week three.
Across the UK there were 177 admissions to ICU/HDU with "confirmed influenza".
:: UK flu outbreak 'worst in seven years'
Overall, PHE said there had been a 7% fall in the rate of hospitalisation because of flu, and a 26% reduction in admissions to intensive care.
The main strains in circulation continue to be A(H3N2), A(H1N1) and Flu B.
It added that in the last few weeks, "statistically significant excess mortality" was continuing in over-65s in England.
But that remained lower than "excess mortality" seen both last winter, and in 2014 to 2015.
Both flu and the cold weather are likely to be "contributing factors".
:: Do you have flu or is it just a cold? Take our test
The best form of protection against flu is to get the vaccine and to practice good respiratory and hand hygiene, Richard Pebody from PHE said, adding that vaccination rates had increased compared to last year.
An additional one and a half million people have had the jab.