Can you remember when pop stardom used to be a glamorous, aspirational career? On No Shame, one of the sparkiest, wittiest figures in British pop confesses to a life of meaningless sex, drug addiction, isolation, distrust, damaged relationships, failed domesticity, inadequate parenting and crippling self-doubt fuelled by a cruelly embattled relationship with a hostile media and internet trolls. “I’ve tried everything to feel something,” sings Lily Allen, her soft, high voice adrift on a wave of distorting harmonies. “But nothing,” she concludes, devastatingly.

Everything to Feel Something is the pitch-black heart of Allen’s fourth album, a tripped-out slow jam on which she surrenders to sexual conquest knowing she is going to hate herself in the morning. “My heart aches,” she sighs with devastating numbness. “All I need is someone to walk all over me/ Close the door behind you, please.” Produced by Fryars, art pop maverick, it may be the most gorgeous and brutal pop song about self-loathing since Robbie Williams’s Come Undone.

Should we be worried about Allen? An early pioneer of curated online stardom, she has always been bold, mouthy and fearless. Top of the blogs at 21, she caused a sensation in 2006 with her cheeky debut, Alright, Still, sold two million copies of 2009’s It’s Not Me, It’s You and declared herself Sheezus in 2014 with her third consecutive chart-topping album. Following a four-year musical hiatus and divorce from the father of her two children, however, her comeback might be subtitled It’s Not You, It’s Me.

The content of No Shame is savagely self-critical, as if her sharp tongue and observational acuity have turned inwards. On songs about her divorce, What You Waiting For? and Family Man, she takes all the blame for the collapse of her marriage while appealing for forgiveness and reconciliation. The album’s most minimalist track, acoustic ballad Apples, contains its bitterest core, when she compares her own broken relationship to her parents’ divorce (“I guess the apple doesn’t fall far from the tree”).

Allen on stage at Mighty Hoopla in Brockwell Park, London on 3 June 2018 Credit: Lorne Thomson/Redferns

Not everything on the album – made in collaboration with cutting-edge producers and guest rappers and singers – is quite so powerful, though. On a few tracks, medium and message seem subtly misaligned. Too often, for example, the balance of an effervescent, upbeat digital pop framework with dark emotional content is upset by a cheesily cheerful chorus. Allen’s doubt and pain can come across as a disorienting after-effect, disguised by the breeziness of her delivery.

Vocally, she barely projects, singing everything in the same sweet, high voice she used to employ for her sarcastic takedowns of pop culture. It is as if one of the saddest albums you will ever hear is masquerading as a set of party hits. Nevertheless, No Shame should be compulsory listening for every young wannabe who still thinks pop stardom will be a panacea for all their problems.

No Shame (Parlophone) is out tomorrow