The University of Kansas is the largest university in the sunflower state. Founded in 1865 on a hill in the town of Lawrence, it has since expanded to five campuses, mainly in woodland. Speak to the students, and you get a sense not just of the social buzz of the university, but also its scale. In total, there are almost 30,000 students, hailing from around the world. There are almost 3,000 students that are non-white, almost 6,000 are drawn from beyond Kansas, and almost 2,000 over the age of 25. This is a diverse population.

Over recent years, the way in which social networks are formed has become a major focus of science. There have been many studies, but one of the most fascinating was conducted by Angela Bahns, an American psychology professor, into academic institutions in the state of Kansas.

The researchers observed the students, watched as they hung out with their friends, and then gave them questionnaires so that they could probe the way they built social groups. In addition to the University of Kansas, they also studied five smaller universities in Kansas: Baker University in Baldwin City, Bethany College in Lindsborg, Bethel College in North Newton, Central Christian College in McPherson and McPherson College, also in McPherson.

When I say these other universities are smaller, I mean much smaller. Baker University, a wonderful college with a rich history, has only three residence halls and two apartment buildings for students. Where the University of Kansas has a student population of almost 30,000, the other five universities have an average of barely one thousand. McPherson College has 629 students and Bethel College has just 437 students. These colleges also, by implication, have less overall demographic diversity. Bethel has just 105 students from outside Kansas, while Baker University and McPherson College have no students at all from overseas.

Now, the question Bahns was seeking to answer was how these differences in background conditions influence the characteristics of the social networks within the institutions. How would they shape the way people made connections?

The answer at an intuitive level seems obvious. The University of Kansas, by sheer dint of scale, affords far more in the way of opportunities to meet people who think in different ways, and who come from different backgrounds.

At Bethel College, on the other hand, the institution may be impressive, but its small size implies that there are far more restrictive opportunities to meet people who are different. For all its benefits of intimacy, a tiny student population must surely curtail the possibilities for meaningful interaction with people with think differently, act differently, or merely look a bit different.

When Bahns looked at the data, however, she found the complete opposite. The social networks at the University of Kansas were more homogenous, not just in terms of attitudes and beliefs, but also politics, moral convictions, and prejudices. “It was a clear result, and completely different to what most people expect,” Bahns told me. “When people are a part of broader communities, they are likely to construct networks that are more narrow.”

Think back to the two campuses. At Kansas, there are lots of people. They are diverse, to be sure, but diversity has a paradoxical property. It means there are a lot of different people to potentially interact with, but it also means that there are many people who are very much like oneself. If one wants to hang out with the like-minded, they are not so very difficult to find. Sociologists call this “fine grained assorting”.

At a smaller college with fewer people, on the other hand, there is less overall diversity. But this means that it is almost impossible to find someone who thinks or looks exactly like oneself. You have to compromise, to accept some minimum level of difference.

“It sounds ironic, but it is quite predictable,” Bahns says. “In the smaller universities, there are fewer available choices, and people have to make connections with people who are comparatively more different. When the campus size is big, on the other hand, there is a greater opportunity for students to ‘fine-tune’ their social network. They can pursue people who are very similar to themselves.”

The internet has created a hyperspace that spans the globe and can be triggered instantly. We have unprecedented access to diverse opinions, beliefs and ideas. This was, of course, the original vision of the internet by Tim Berners-Lee: a place where scientists could share ideas. But high diversity in the overall network has the potential to create paradoxical effects in local networks. This is as true in the digital world as in the social world. At a cosmopolitan university such as the University of Kansas, for example, it led to homogenous friendship groups.

These insights help us to grasp one of the defining paradoxes of the modern age: echo chambers. For all its promise of diversity and interconnection, the internet has become characterised by a new species of highly cohesive in-groups, linked not by kin or clan, but by ideological fine sorting.

By getting their news from Facebook, and other platforms, where friends share cultural and political leanings, people are more exposed to people who agree with them, and evidence that supports their views. They are less exposed to opposing perspectives. The dynamics of fine-sorting can be magnified by a subtler phenomenon: the so-called filter bubble. This is where various algorithms, such as those inside Google search, invisibly personalise our searches, making it more likely that we’ll see more of what we want to see, further limiting our access to diverse viewpoints.

This is the digital equivalent of the Bahns experiment, but at a higher level of gearing. The sheer interconnectivity of the internet has facilitated enhanced political fine tuning.

The precise extent of echo chambers is a matter of some debate, with different studies pointing in slightly different directions. The mathematician, Emma Pierson, analysed how the troubles of Ferguson, Missouri, were covered on social media in 2014, after a police officer called Darren Wilson shot and killed a black man, Michael Brown. She found two distinct clusters. “Blue tweets” expressed horror at Brown’s death and criticised the oppressive police response, while “red tweets” argued that the policeman was being scapegoated and the protesters were looters.

As Pierson puts it: “The red group says they would feel safer meeting Darren Wilson than Michael Brown, and says that Brown was armed when he was shot; the blue group sarcastically contrasts Darren Wilson with the unarmed Michael Brown. The red group talks about mob justice and race baiting; the blue group talks about breaking the system.”

Perhaps most tellingly of all, these two groups had virtually no interaction. “When it comes to Ferguson, two groups with very different political and racial backgrounds ignore each other,” Pierson writes. “This seems likely to cause problems, and in fact it does. For one thing, the two groups think drastically different things.”

Other studies led by Seth Flaxman of Oxford offer a different lens on the digital world. These find that when you look at overall internet use, digital users have higher average exposure to the views of their own side, but nevertheless get to see the views of opponents, too. Perhaps that is not surprising. Even in the clan systems that emerged after the agricultural revolution, the in-groups were not completely shut off from each other.

But what is fascinating – and broadly acknowledged by almost all scholars – is what happens when exposure does take place. Now, you might have thought that by hearing the views of opponents, opinions would become less extreme. Views would become more nuanced. In fact, the opposite happens. People become more polarised.

In Pierson’s study, for example, the limited interaction between red and blue tweeters was explosive. “When the red and blue group did talk, it often wasn’t pretty,” she writes. “DeRay Mckesson, a school administrator who has played a central role in organising protests, was described as a ‘commie boy’ who spread hate… saw ‘value in racist drivel’.”

A study led by Christopher Bail of Duke University found a similar pattern. He recruited 800 twitter users to follow a bot that retweeted the views of high profile people from across the political spectrum. What happened? Far from becoming more balanced, the twitter users became more polarised. This was particularly true for Republicans, who became more conservative. It was as if exposure to different views confirmed their prior convictions.

To understand what is going on, and to fully glimpse the logic of echo chambers, we need to draw a distinction between echo chambers and information bubbles. As the philosopher, C Thi Nguyen notes, information bubbles are the most extreme form of isolation, where people on the inside see only their side of the argument and nothing else. These kinds of social groups have rarely existed in modern history except in cults and other “walled institutions”.

Echo chambers, Nguyen argues, are different. They may cut some people off from alternative views through informational filtering but their distinctive feature is that they have not one filter, but two.

What is the second filter? We shall call them epistemic walls.

In their scholarly book, Echo Chamber, Kathleen Hall Jamieson and Frank Cappella, two experts on the media and politics, examine the logic of political polarisation. They do so through the prism of Rush Limbaugh, a successful conservative commentator whose radio show has a cumulative weekly audience of around 13.25 million unique listeners.

They note that Limbaugh doesn’t seek to persuade his audience to cut themselves off from alternative voices. This would be pretty much impossible in such an interconnected world. Instead, he seeks to delegitimise alternative voices. He attacks the integrity of those who offer different views, and defames their motives. His insistence is not (just) that opponents are wrong, but that they are malicious. He argues that the mainstream media expresses a liberal bias. The media has set out to destroy Limbaugh and his followers because it can’t abide the truth he speaks.

“The conservative opinion hosts underscore the notion that the mainstream media use a double standard that systematically disparages conservatives and their beliefs,” Jamieson and Cappella write. They argue that Limbaugh seeks to discredit all other sources of information through the techniques of “extreme hypotheticals, ridicule, challenges to character, and association with strong negative emotion”.

Now we can begin to glimpse the subtly different properties of information bubbles and echo chambers. With the former, informational borders are hermetically sealed. People on the inside only hear co-inhabitants of the bubble. This creates distortions, but it also confers fragility. The moment a member of the in-group is confronted with outsider opinions, they are likely to question their beliefs. The way to burst an information bubble, then, is through exposure. This is why cults take such lengths to deny insiders access to different voices.

Echo chambers, with their additional filter, have fundamentally different properties. People on the inside hear more opinions from the in-group, but these views tend to become stronger when exposed to opposing opinions. Why? Because the more opponents attack Limbaugh, the more they point to the errors in his opinions, the more it confirms the conspiracy against him. Opponents are not offering new insights, but fake news. Each piece of evidence against Limbaugh is a new brick in the wall separating the in-group from outsiders.  

As Nguyen puts it:  “What’s happening is a kind of intellectual judo, in which the power and enthusiasm of contrary voices are turned against those contrary voices through a carefully rigged internal structure of belief. Limbaugh’s followers read – but do not accept – mainstream and liberal news sources. They are isolated, not by selective exposure, but by changes in who they accept as authorities, experts and trusted sources. They hear, but dismiss, outside voices.”

As Nguyen puts it: “Ask yourself: could you tell a good statistician from an incompetent one? A good biologist from a bad one? A good nuclear engineer, or radiologist, or macro-economist, from a bad one? Nobody can really assess such a long chain for herself. Instead, we depend on a vastly complicated social structure of trust. We must trust each other, but, as the philosopher Annette Baier says, that trust makes us vulnerable.”

It is this epistemic vulnerability that echo chambers exploit. By systematically undermining trust in alternative views, by defaming those who offer different insights and perspectives, they introduce a filter that distorts the belief-formation process itself. Alternative views are dismissed not after consideration, but upon contact. Facts are rejected even as they are offered. Perspectives and evidence are repelled somewhat like a magnet repelling iron filings.

“Echo chambers operate as a kind of social parasite on our vulnerability…” Nguyen writes. “An information bubble is when you don’t hear people from the other side. An echo chamber is what happens when you don’t trust people from the other side”.

This is not about conservative radio hosts, still less Conservatism itself. There are echo chambers across politics, including on the Left, not to mention diets, breastfeeding, and even some academic disciplines.

It is the twin filters of information and trust that create an unusually resilient form of in-group cohesion. Where an information bubble is inherently fragile, echo chambers on both sides of the political spectrum are reinforced by the mutual exposure of alternative views, driving polarisation, and leading to competing (and often contradictory) claims of fake news. Each side thinks the other is living in a post-truth age.

As Nguyen puts it – and it’s perhaps worth both sides in the Brexit debate reflecting on the words: “Here’s a basic check: does a community’s belief system actively undermine the trustworthiness of any outsiders who don’t subscribe to its central dogmas? Then it’s probably an echo chamber.”

Rebel Ideas: The Power of Diverse Thinking by Matthew Syed is published in September.