For hundreds of thousands of commuters, a rail timetable change will never seem innocuous again. Before the schedules were switched three weeks ago, plenty of people had predicted teething troubles: train companies had spoken of the logistical challenge ahead, and commuters were told to expect some initial disruption.

At the time, Govia Thameslink Railway (GTR), a key commuter franchise that runs a sprawling array of routes from Brighton through London to Luton and beyond, seemed to have put recent problems behind it. There had been warning signs on Northern, which covers cities including Manchester, Liverpool and Newcastle. But still, the sheer scale of cancellations, delays, confusion and misinformation that arrived with the new timetable came as a shock.

One rail grandee had predicted problems. But Sir Michael Holden, a former boss of East Coast, said the chaos had surprised him. “Never in my worst nightmares did I imagine it could conceivably be anything like as bad as it is,” he said. Nigel Harris, editor of trade magazine Rail, said it was “the most chaotic, fundamental and humiliating failure it has been my misfortune to witness in 40 years as a rail journalist”.

What had been billed as a boost to services instead saw thousands cancelled: plummeting punctuality statistics for GTR and Northern masked even greater local problems. In some peak periods on the Manchester-Preston line, or on Thameslink mainline trains, two in three trains were failing to run at all. Instead of a few days of hiccups, it swiftly became clear weeks of convulsions were ahead, with emergency timetables being introduced to slash services and stem the chaos.

Around 8,000 services on GTR have so far been cancelled or severely delayed, while some 5,000 Northern trains have suffered the same fate since 20 May – excluding two days of strikes when 2,000 trains were pulled in advance.

As the cumulative fury of commuters swells into a political crisis, how did this self-inflicted disaster come about?

Why was the timetable changed so radically?

This blow to the reputation of England’s railways came, ironically, at what should have been a moment of triumph. Instead of the usual twice-yearly tweaks, the national timetable was to undergo wholesale revisions to take advantage of new technology, new trains and years of engineering work. The Great North Rail project, which included electrification and the construction of a new piece of track, the Ordsall Chord, was designed to allow more trains to travel through Manchester, speed up journeys and make new direct links possible.

In the south, commuters were set to reap the fruits of a project so long in the gestation it was once called Thameslink 2000: a £7bn overhaul that included rebuilding London Bridge station, adding modern signalling and buying new trains so that dozens of services could pass per hour with automated, Tube-style frequency. Every single train timing was redrawn on GTR’s franchise in an attempt to harvest the benefits of that work, add extra services and – laughable though it sounds now – increase reliability.

Why did it go wrong?

The fundamental problem was the lack of drivers. Both GTR and the company that operates Northern’s franchise, Arriva Rail North, claim to be employing more than enough drivers to run the expanded services the May timetable was due to provide. However, what is needed on every train is a driver who has been trained both on the exact model of train and the full route they are to operate. That takes time and resources: new drivers are trained by qualified drivers, taking them out of the action too and compounding the shortage.

That, of course, is foreseeable – which is why services have to be planned many months in advance. Normally, the process of setting timetables starts 16 months ahead, to be thrashed out by train companies and approved by Network Rail – the state-owned operator of Britain’s rail infrastructure – with three to six months to spare. This time, it was a matter of weeks.

Why were the timetables drawn up so late?

There is no single answer, though there are interconnecting issues from north to south. For Northern, Network Rail accepts a large part of the blame for a long-delayed infrastructure upgrade – the electrification of the line past Bolton. Work that should have been long finished has dragged on: a first contractor was replaced by Carillion – but then Carillion imploded. And Carillion’s replacement has had to grapple with ground scarred by old mines, taking weeks to put up power lines that would normally go in overnight.

But Northern has also struggled with industrial relations: goodwill, which has often buoyed up an industry reliant on overtime and rest-day working, has evaporated in parts. All retraining has had to take place on working days, leading to an acute shortage of trained drivers even before the May changeover.

GTR, meanwhile, had been through well-documented problems of its own, with strikes by guards and drivers heaping on commuter despair. The Southern part of the franchise has been a byword for rail failure since 2015 (although right now, almost unnoticed, it is thriving since the timetable change). A set of interrelated problems, on track and trains, had been identified by Chris Gibb, the senior railwayman brought in by transport secretary Chris Grayling to help restore some quality to GTR in December 2016. He warned that a phased introduction of new services would be vital to make Thameslink work.

That view was shared by Network Rail and GTR: but delaying new services needed the Department for Transport’s approval – which came long after Network Rail planners needed to start drawing up the timetable. When the DfT belatedly decided phasing was a good idea, the planners’ work was scrapped and the process started again, four months late already. And yet more issues arose: GTR’s planning team, shorn of numerous experienced staff, requested thousands of changes months later.

What questions remain?

Two inquiries lie ahead: one commissioned by Grayling from Professor Stephen Glaister, chair of the Office of Rail and Road regulator; and an investigation by the transport select committee.

The latter may yet bring the axe hovering closer over Grayling’s head. The transport secretary has angered Network Rail by immediately blaming them. Then in his statement to the Commons, he claimed he had been “personally assured” by GTR that the service would work, and risked alienating allies such as Gibb, who chaired the Thameslink readiness board, by joking about trusting experts.

Grayling’s claim of receiving assurances could yet be called into question by GTR executives. Conversely, should GTR prove to have not informed the readiness board that they did not have sufficient drivers, it calls into question the competence of a company supposedly intensely monitored by the DfT. And Network Rail, as a publicly owned company, reports to the transport secretary – the one accountable person, as Labour points out, in a fragmented industry.

Would nationalisation or a single owner have averted it?

Possibly, yes: even leaving aside questions of state versus private, a single accountable body with proper oversight might have acted sooner. In Scotland, where track and train come under joint supervision, the late delivery of Hitachi trains prompted Scotrail to defer its own new timetable. This was a course of action that England’s fragmented system was apparently unable to take.

A senior industry source said: “It’s a system issue – you’ve got to get all the elements right. We’ve bitten off more than we could chew. We knew that it was going to be the biggest timetable change ever, and the industry didn’t spend enough time checking it would work – until it was too late.”