For supporters of immigration in the U.S. and Europe, it’s a hard time to be optimistic. The White House effectively declared the legislative effort for immigration reform dead last week, while looking for another $2 billion to throw at border control. The recent European Union elections, meanwhile, voted in more extreme-right politicians than ever. Yet there are still strong reasons to think the trend is toward a more welcoming West for migrants from the rest of the world. Poll evidence suggests Europeans and Americans are becoming more welcoming of foreigners over time. Add to that the growing economic interest in increased migration to keep up living standards, and a return to the historical norm of open borders in the West is the likely long-term result.

Politically, there’s little question that immigration is currently seen as a losing issue. The U.S. spends considerably more on immigration enforcement agencies than on the FBI, the DEA, the Secret Service, the U.S. Marshals, and the ATF combined. In elections for the European Parliament at the end of May, far-right parties won the most votes in both France and Denmark, while Greece’s fascist Golden Dawn party won three seats. In Britain, Prime Minister David Cameron has proposed reducing immigration to the U.K. from the hundreds of thousands to the tens of thousands, yet his Conservative Party was still outpolled by the UK Independence Party, whose platform is pretty simple: Get out of Europe, close the borders.

The average citizen in both the U.S. and Europe, however, appears to be far calmer about immigration than the heat and light of recent events would suggest. Though there has been an uptick in popular concern about levels of migration in Europe and (to a lesser extent) the U.S., that rise should be seen in the context of a longer-term trend away from nativism. According to World Values Survey data, the proportion of Germans who think that “when jobs are scarce, employers should give priority to people of this country over immigrants” has fallen from 56 percent in the late 1990s to 41 percent more recently. Over the same period in Spain, the proportion fell from 70 percent to 53 percent, and in the U.S., 59 percent to 50 percent.

The U.K. is the only country out of eight European countries and the U.S. surveyed by the German Marshall Fund where the majority of respondents thought there were too many immigrants in the country in 2013. Compare that with 41 percent in the U.S. and only 24 percent in Germany. In the U.S., more than two-thirds view immigration as a good thing for the country. And even in the outlier U.K., the percentage of people suggesting immigration has gone too far has been similar—and if anything a little higher—all the way back to the 1960s. The proportion of Britons who admit they are at least a little prejudiced against people of other races has fallen from 35 percent in 1980 to 30 percent in the latest survey. The downward trend looks set to continue: Opposition to immigration skews old, and young people are considerably more relaxed about migration and race. Thirty-seven percent of British people born before 1929 admit to being very or a little prejudiced against people of another race, compared with 25 percent of Generation Y.

The U.K. also demonstrates the disconnect between attitudes toward immigration and the scale of immigration itself. While rising concern in the U.K. over the past decade has followed an upswing in migration from new member states of the European Union, in 2012 U.K. net migration was at its lowest level since 2008. Prejudice is the least prevalent in the most racially diverse parts of the country. Inner London, perhaps the most diverse part of the U.K., sees only 16 percent willing to admit prejudice—about one half the national average. Similarly, animosity toward immigrants in the U.S. is concentrated in rural areas, according to Katherine Fennelly and Christopher M. Federico of the University of Minnesota. They suggest that might be because of “greater isolation and lesser contact with immigrants and minorities.”

Across Europe, the recent elections may reflect a growing animosity toward immigrants during a downturn, but the far right in Europe did better in countries that suffered comparatively little from the financial crisis. Decades of research suggest views about migration simply aren’t related to self-interested worries about the threat of losing jobs. In their survey (PDF) of public attitudes toward immigration, Jens Hainmueller and Daniel Hopkins suggest the idea has “repeatedly failed to find empirical support,” making it something of a “zombie theory.” At the same time, a cluster of attitudes toward race and nationalism alongside immigration are closely linked.

This all suggests attitudes toward migration are a cultural issue—like those toward guns or gay marriage. And cultural attitudes unmoored from immediate economic concerns can change fast—look at gay marriage, where popular backing for marriage equality increased from 27 percent to 55 percent over the past 18 years. Or take another cultural question about employment: The World Values Survey in Germany in the late 1990s found more than one-fifth of the adult population thought that when jobs were scarce men had more right to a job than did women. That has fallen to 7 percent in the most recent survey. In Spain, that figure has dropped from 27 percent in the 1990s to 7 percent today. No major politician in Europe or America has come out with a proposal to shut women out of the workforce during the recent economic crisis. Hopefully, during the next economic crisis, the same will be true of migrants.

Those politicians fostering “acceptable nativism” might want to look at long-term economic trends. A recent analysis by the U.K.’s National Institute of Economic and Social Research suggests that the long-term impact of reducing net migration by 50 percent—in line with the target proposed by Britain’s David Cameron—would reduce British income per capita by about 2.7 percent by 2060 and force income tax increases of about 2.2 percentage points. That’s largely because most migrants are young—and so populations skew older absent immigration. Lower net migration implies more retirees on pensions with heavy hospital bills and fewer working-age people paying taxes.

You can be the party of low taxes or of low immigration. You can’t be the party of both. Politicians on both sides of the Atlantic should consider that math next time they use immigrants as a convenient scapegoat come election time.