As holder of the ancient post of lord chancellor, responsible for courts and judges across the land, Derry Irvine had a knack for shutting down irritating interventions from his colleagues. During his time in office, from 1997 to 2003, Irvine’s fearsome reputation preceded him. Minor functionaries and even cabinet ministers would cower when the lord chancellor, who once compared himself to his 16th-century predecessor Cardinal Wolsey, felt they had not mastered their briefs.

Tony Blair might have thought that as the nation’s leader he would escape a lashing from Irvine. But “Young Blair” – Irvine’s name for the prime minister, dating back to 1976, when Irvine gave him his first professional opening as a pupil barrister – was briefly silenced during a fraught meeting early in Labour’s second term in which a worrying rise in the number of asylum cases was being discussed.

The gravest offence, in Lord Irvine’s eyes, was to call into question Britain’s solemn commitments on human rights, notably those made after the second world war in the European convention on human rights (ECHR). When ministers dared to broach the issue of drawing back from some aspects of the ECHR as a way of curbing asylum applications, Irvine’s response was sharp. “I don’t know why you guys don’t just adopt the Zimbabwean constitution and have done with it,” he told Blair and the home secretary, David Blunkett. The discussion was brought to a swift conclusion when Lord Goldsmith, the attorney general, pointed out that Britain would be in breach of its EU membership terms if it sought to wriggle out of its responsibilities under the separate ECHR. New Labour would have to find another solution to its immigration problem.

As today’s generation of political leaders prepares to fight an election that is in part a contest about the mistakes, judgments and assumptions Labour made in government on immigration, it is easy to forget just how much immigration and asylum haunted Downing Street throughout New Labour’s time in office. Between 1997 and 2010, net annual immigration quadrupled, and the UK population was boosted by more than 2.2 million immigrants, more than twice the population of Birmingham. In Labour’s last term in government, 2005-2010, net migration reached on average 247,000 a year.

The dramatic changes have left British politics ruptured. Immigration remains the No 1 issue on the doorstep, according to pollsters – a stream that feeds into the well of mistrust in politics. It has spawned the emergence of Ukip and helped create four- or five-party politics in the UK for the first time.

Despite the Conservative party’s recent travails over its broken pledge to bring net migration down to the low tens of thousands, the issue has been a special bind for Labour. In the wake of the party’s defeat in the 2010 election, there was a brief mass mea culpa about immigration, but even now Labour struggles to explain to a core part of its electorate the decisions that were taken on its watch.

Nigel Farage, the Ukip leader, has made capital out of his claim that the Labour government embarked on a deliberate policy to encourage immigration by stealth. Ukip often cites an article by Andrew Neather, a former No 10 and Home Office adviser, who wrote that the Labour government embarked on a deliberate policy from late 2000 to “open up the UK to mass migration”. Yet where Farage sees a political conspiracy behind the numbers, others veer towards the theory of history identified by the great 20th-century historian AJP Taylor, who always stressed the significance of chance events.

Even the most ardent defenders of the New Labour government acknowledge that such a wave of immigration was not purely down to chance. But the key players of the time show in candid conversations that they were struggling to cope with a new world of rapid population movement across porous borders. At times they felt they were stumbling from one move to another, unsure of the present, let alone the future.

Leap in immigration threatens Labour lead

As the head of Tony Blair’s policy delivery unit during his second term in office, from 2001 to 2005, Michael Barber did not get many chances to lie in on the weekends. One Saturday morning in February 2003, he took the liberty of sleeping until 8.50am. Ten minutes later, his phone rang. On the other end of the line was an anxious prime minister. Blair was once again fixated on the issue that had plagued his first term in office. “He was worrying away about illegal asylum applications,” Barber wrote in his diary.

In his new book, How to Run a Government, Barber recounts how he delivered the bad news: there had been a big jump in asylum applications as refugees from Afghanistan, Saddam Hussein’s Iraq and Kosovo headed to Britain. By Monday morning, having read the 1951 UN convention on refugees, Blair cleared his diary for the entire morning to allow him to spend five hours “forensically” going through the asylum caseload. The following day Blair summoned the relevant ministers to a meeting of the government’s emergency Cabinet Office Briefing Room (Cobra) committee, where he ran rings round most of the people in the room. Barber was deeply impressed with the detailed way in which Blair handled the issue.

Labour feared that the failure to grip the asylum challenge risked making the government look incompetent and – more damagingly – out of touch. “Immigration per se, but non‑European immigration [in particular], was a huge, huge issue for Tony Blair,” recalls Sir Stephen Wall, who was head of the Cabinet Office’s European secretariat between 2000-04. “I remember him saying, very soon after the 2001 election, ‘The one thing that could lose me the next election is immigration.’”

From the first year in office, the issue had hit the Labour government like a whirlwind. In 1997 net migration had been 48,000, but it rose extremely rapidly over the next 12 months, almost trebling to 140,000 in 1998. It was never to fall below 100,000 again.

Ministers and officials from that era recall in painful detail the apparently impossible task of dealing with the surge in asylum applications, as refugees fled to Britain. Almost every day, newspaper headlines would sneer at chaos in Whitehall as the Immigration and Nationality Directorate, later described as dysfunctional by ministers, struggled to cope.

Charles Clarke, who took over as home secretary in 2004, says that from Labour’s first days in office, the system for assessing applications for citizenship, residency and asylum claims was a mess. “We developed a massive backlog, particularly on asylum where we had cases waiting literally five years to be solved,” Clarke says. “That was the core problem that had built up behind an unmanageable set of issues. It was a complete nightmare and led to a sense of complete ungovernability of the whole system and that I think has undermined confidence in it.”