There has recently been built in Merton in south London a “mega mosque” that has inevitably become the focus of much controversy. In his book The British Dream, David Goodhart takes the mosque as symbolic of the unacceptable change that immigration has wrought upon the nation. The mosque, he writes, “replaced an Express Dairies bottling plant which provided a few hundred jobs for local people and lots of milk bottles – an icon of an earlier, more homogenised age.”

There was, in fact, a seven-year gap between the closing of the dairy in 1992 and building work beginning on the mosque. In those seven years the abandoned dairy was, according to local accounts, turned into a crack den. So, one story we could tell is that of economic forces closing down an unprofitable dairy, with the loss of several hundred jobs, and of local Muslims subsequently rescuing the abandoned, crime-infested site, creating new jobs and in the process transforming Merton for the better. Critics of immigration want, however, to tell a different story. The mosque, in their eyes, is symbolic not of the rescue of a site from abandonment and crime, but of the original closure of the dairy and of the transformation of Merton’s old way of life.

The story of the Merton mosque, and the retelling of that story as a narrative of cultural loss, gets to the heart of the contemporary debate about immigration. Immigration is clearly one of the most fiercely debated and toxic issues of today. The debate is, however, less about the facts than about the existential impact. Immigration has become symbolic of the disruption of communities, the undermining of identities, the fraying of the sense of belongingness, the promotion of unacceptable change. For Goodhart, “large-scale immigration” has created “an England that is increasingly full of mysterious and unfamiliar worlds”. He quotes one man from Merton: “We’ve lost this place to other cultures. It’s not English any more.”

The roots of The British Dream lie in Goodhart’s 2004 essay in Prospect magazine, of which he was then editor, called Too Diverse? Liberals, he suggested, had to face up to a “progressive dilemma”. Too much immigration undermined social solidarity, particularly in a welfare state. We had to choose between the two. The essay caused considerable controversy, but the idea that too much immigration undermines social solidarity has over the past decade become almost common sense.

It is a claim that lies at the heart of Paul Collier’s Exodus. Professor of Economics and Public Policy at Oxford University’s Blavatnik School of Government and co-director of the Oxford Centre for the Study of African Economies, Collier has long been concerned with questions of poverty and justice. In Exodus he seeks to unpack the impact of immigration on both the host community and those left behind in the countries of origin. Too much immigration, he suggests, adversely affects both groups. It drains poor countries of human resources and undermines the social stability of rich countries.

Collier, like Goodhart, accepts that economic fears about the impact of immigration on host nations are largely misplaced. But, again like Goodhart, he insists that too much diversity creates social problems, in particular by destroying “mutual regard”, the willingness to cooperate and to redistribute resources. Both authors draw upon the work of the American sociologist Robert Putnam, who has shown that the more diverse a community, the less socially engaged are its members – they vote less, do less community work, give less to charity, have fewer friends. Most strikingly, Putnam discovered that people in more diverse communities show greater distrust not just of members of other ethnic groups but of their own, too.

Putnam’s work has long been used by critics of immigration to suggest that diversity undermines the social fabric. More recent research has, however, questioned his conclusions. The latest such study, led by Patrick Sturgis, director of Britain’s National Centre for Research Methods, investigated the relationship between diversity and trust within London. It discovered the opposite relationship to Putnam. Once the researchers had allowed for social and economic deprivation, they found that “ethnic diversity is … positively related to social cohesion, with significantly higher levels of cohesion evident as ethnic heterogeneity increases.”

We should, of course, no more view Sturgis’s research as demonstrating that diversity creates trust than we should have seen Putnam’s work as having demonstrated that diversity undermines trust. A key problem, as Putnam himself has pointed out, is that such studies offer only a snapshot of attitudes at one moment in time. Diversity, though, is not a static phenomenon but changes over time, as does our political response to it. Over the past few decades, we have witnessed the demise of movements for social change, the rise of identity politics, the atomisation of society, a loss of belief in universal values, all of which has led to civic disengagement and a greater sense of anomie. The real problem being exposed by a study like Putnam’s, then, may not be diversity as such but the political context in which we think about it.

What is missing from Putnam’s data is also missing from the accounts of critics like Goodhart and Collier – a sense of historical context.

The existential fear of immigration is almost as old as immigration itself. Had Arthur Balfour been able to read Goodhart’s account of the creation of an England “full of mysterious and unfamiliar worlds”, of an England that “is not English any more”, he would undoubtedly have nodded in agreement. Balfour was prime minister in 1905 when Britain introduced its first immigration controls, aimed primarily at European Jews. Without such a law, Balfour claimed, “though the Briton of the future may have the same laws, the same institutions and constitution … nationality would not be the same and would not be the nationality we would desire to be our heirs through the ages yet to come.” Two years earlier, the Royal Commission on Alien Immigration (an “alien” was, in the early 20th century, both a description of a foreigner and a euphemism for a Jew) had expressed fears that newcomers were inclined to live “according to their traditions, usages and customs” and that there might be “grafted onto the English stock … the debilitated sickly and vicious products of Europe.”

The sense that Jewish immigration was uncontrolled and that “we’ve lost this place to other cultures” was palpable in the discussions. “There is no end to them in Whitechapel and Mile End,” claimed one witness giving evidence to the 1903 Royal Commission. “These areas of London might be called Jerusalem.” The Conservative MP Major Sir William Eden Evans-Gordon expressed the same sentiment through a quite extraordinary metaphor. “Ten grains of arsenic in a thousand loaves would be unnoticeable and perfectly harmless,” he told parliament, “but the same amount put into one loaf would kill the whole family that partook of it.”

By the 1950s, the Jewish community had come to be seen as part of the British cultural landscape. The same arguments used against Jews half a century earlier were now deployed against a new wave of immigrants from south Asia and the Caribbean. A Colonial Office report of 1955 echoed Balfour, fearing that “a large coloured community as a noticeable feature of our social life would weaken … the concept of England or Britain to which people of British stock throughout the Commonwealth are attached.” There were worries, too, about the uncontrolled nature of immigration. “The question of numbers and of the increase in numbers,” Enoch Powell insisted, lay at “the very heart of the problem”. “Whole areas, towns and parts of England,” he claimed, were being “occupied by different sections of the immigrant and immigrant-descended population”. A decade later Margaret Thatcher gave a notorious TV interview in which she claimed that there were in Britain “an awful lot” of black and Asian immigrants and that “people are really rather afraid that this country might be rather swamped by people with a different culture.” The echoes are unmistakable, both of the debate about Jews before and of the contemporary immigration debate.

Just as Jews became an accepted part of the cultural landscape, so did postwar immigrants, though the acceptance was more grudging, and often not extended to Muslims. Today, the same arguments that were once used against Jews, and then against south Asian and Caribbean immigrants, are now raised against Muslims and East Europeans. A succession of authors such as Mark Steyn, Oriana Fallaci, Melanie Phillips and Christopher Caldwell warn that Muslim immigration is threatening the very foundations of European civilisation. The melodramatic title of Caldwell’s book, Reflections on the Revolution in Europe, is a nod to Edmund Burke and reflects Caldwell’s belief that the impact of postwar immigration has been as dramatic as the fall of the ancien régime in 1789. Muslim migration in particular has been akin to a form of colonisation. “Since its arrival half a century ago,” Caldwell claims, “Islam has broken – or required adjustments to, or rearguard defences of – a good many of the European customs, received ideas and state structures with which it has come in contact.” Islam “is not enhancing or validating European culture; it is supplanting it”.

For Caldwell, prewar immigration between European nations was different from postwar immigration from outside Europe because “immigration from neighbouring countries does not provoke the most worrisome immigration questions, such as ‘How well will they fit in?’ ‘Is assimilation what they want?’ and, most of all, ‘Where are their true loyalties?’” In fact, those were the very questions asked of European migrants in the prewar years. “The notion of the easy assimilation of past European immigrants,” as the historian Max Silverman has written, “is a myth.”

Throughout the 20th century, virtually every wave of immigration, whether of Irish and Jews to Britain, Italians and North Africans to France, Catholics and Chinese to America, was met with the claim that the influx was too large, too culturally distinct, too corrosive of stability and continuity. Come the next, larger wave of immigration, and the previous wave now came to be seen as acceptable in terms of what the nation could absorb but the new wave was not. And it is against this background that we need to understand the fears of Goodhart, Collier and Caldwell. All insist that Europe today faces a unique danger. All the arguments recycle the panic expressed in response to every wave of immigration.

The current debate takes place, however, in a new context. When Balfour warned of the impact of Jewish immigrants, there existed a strong sense of British identity, rooted primarily in the concepts of race and empire. Hostility to immigration was part of a racialised defence of national identity.

Behind contemporary hostility to immigration lies a sense of the dissolution of such identity, of the erosion of common values. There lies also the breakdown of traditional political mechanisms, the growing chasm between the elite and the public, and the abandonment by mainstream parties of their traditional working-class constituencies. As a result, argues Goodhart, what he calls the “left behind” white working class experience immigration “as a loss, either directly because they lived in a neighbourhood that was rapidly changed by it or indirectly because their working class culture and institutions seemed to be pushed aside by the same market forces that then ushered in the newcomers”.

The transformation of working-class life, the erosion of the sense of working-class identity, the breaking of bonds of solidarity, the marginalisation of labour as a political voice – all are real phenomena. But all have roots not in mass immigration but in broader economic and political changes. When the first wave of postwar immigrants arrived in Britain in the 1950s and 1960s, it was a period of full employment, an expanding welfare state and strong trade unions. Today, Britain’s manufacturing base has all but disappeared, working class communities have disintegrated and the welfare state has begun to crumble. Trade unions have been neutered, the Labour party has largely cut its roots with its working-class base, and the very idea of class-based politics is derided. All this has helped erode the bonds of solidarity that once shaped working-class communities, leaving many feeling voiceless and detached from the political process.

Immigration has played almost no part in fostering these changes. It has, however, come to be a means through which many perceive these changes. Partly this is a consequence of the way that the public discussion has been framed, with politicians at both ends of the spectrum presenting immigrants as a problem, even a threat. Partly also it is because the forces of globalisation, or the internal wranglings of the Labour party, are difficult to conceptualise. One’s Bangladeshi or Jamaican neighbour is easy to see. Almost inevitably, immigration has come to be viewed by many not as something that has enriched their lives, but as something that has diminished them.

Goodhart himself acknowledges that “social and economic change would have swept away the old working class ways even if there had been zero immigration”. Why, then, bring immigration into this debate at all? So great has become the obsession with immigration that it has come to be felt as the problem even when reason informs us otherwise.

This is particularly apparent in Exodus. Throughout the book, Collier chastises other participants in the immigration debate for allowing their prejudices to shape their reasoning, and for using reason “to legitimise judgments that we have already made on the basis of our moral tastes”. And yet it would be difficult to find a more apt description of Collier’s own approach. Everything from the changing nature of British criminal culture to recent “policies of reduced taxation and increased reliance on the market” to the London riots of 2011 may be attributed to “the pronounced increase in cultural diversity brought about by immigration”. He provides no evidence. Indeed, he suggests that “the purpose” of his “anecdotes in which immigration appears to have undermined social capital, is decidedly not to strengthen an argument”. But why else introduce them? Much of his book reads like a search for a narrative to bolster an already formulated argument about immigration.

Goodhart and Collier both claim that in “liberal circles” immigration “has become a taboo subject”. “The only permissible opinion,” writes Collier, “has been to bemoan popular antipathy to it.” In reality, though, what is rarely questioned is not immigration but the idea that immigration is responsible for Europe’s social ills.

After the Lampedusa tragedy in October when a boat carrying migrants sank in the Mediterranean, leading to the deaths of more than 300 people, European politicians expressed much anger and grief. What no one was willing to acknowledge was that the tragedy was not merely an accident but the gruesomely inevitable consequence of EU border policies. For more than three decades the EU, driven by an obsession with immigration, has been constructing a Fortress Europe to keep the “unwanted” from landing on the shores of the continent, spending hundreds of millions of euros on external border controls. Since 1988 some 20,000 migrants have died trying to enter Europe, two thirds of them perishing in the Mediterranean. And what have European nations done in response? They have continued to strengthen Fortress Europe and charged fishermen who saved drowning migrants with aiding illegal immigration.

So bedazzled have we become by the existential fear of immigration that migrants have come to be seen less as living, breathing human beings than as so much flotsam and jetsam to be swept away from Europe’s beaches. Fortress Europe has created not only a physical barrier around the continent but an emotional one, too, around Europe’s sense of humanity.