New experiments show that the brain distinguishes between perceived and imagined mental images by checking whether they cross a “reality threshold.”

Is this the real life? Is this just fantasy?

Those aren’t just lyrics from the Queen song “Bohemian Rhapsody.” They’re also the questions that the brain must constantly answer while processing streams of visual signals from the eyes and purely mental pictures bubbling out of the imagination. Brain scan studies have repeatedly found that seeing something and imagining it evoke highly similar patterns of neural activity. Yet for most of us, the subjective experiences they produce are very different.

“I can look outside my window right now, and if I want to, I can imagine a unicorn walking down the street,” said Thomas Naselaris, an associate professor at the University of Minnesota. The street would seem real and the unicorn would not. “It’s very clear to me,” he said. The knowledge that unicorns are mythical barely plays into that: A simple imaginary white horse would seem just as unreal.

So “why are we not constantly hallucinating?” asked Nadine Dijkstra, a postdoctoral fellow at University College London. A study she led, recently published in Nature Communications, provides an intriguing answer: The brain evaluates the images it is processing against a “reality threshold.” If the signal passes the threshold, the brain thinks it’s real; if it doesn’t, the brain thinks it’s imagined.

They’ve done a great job, in my opinion, of taking an issue that philosophers have been debating about for centuries and defining models with predictable outcomes and testing them. - Thomas Naselaris, University of Minnesota

Such a system works well most of the time because imagined signals are typically weak. But if an imagined signal is strong enough to cross the threshold, the brain takes it for reality.

Although the brain is very competent at assessing the images in our minds, it appears that “this kind of reality checking is a serious struggle,” said Lars Muckli, a professor of visual and cognitive neurosciences at the University of Glasgow. The new findings raise questions about whether variations or alterations in this system could lead to hallucinations, invasive thoughts or even dreaming.

“They’ve done a great job, in my opinion, of taking an issue that philosophers have been debating about for centuries and defining models with predictable outcomes and testing them,” Naselaris said.

When Perceptions and Imagination Mix

Dijkstra’s study of imagined images was born in the early days of the Covid-19 pandemic, when quarantines and lockdowns interrupted her scheduled work. Bored, she started going through the scientific literature on imagination — and then spent hours combing papers for historical accounts of how scientists tested such an abstract concept. That’s how she came upon a 1910 study conducted by the psychologist Mary Cheves West Perky.

Perky asked participants to picture fruits while staring at a blank wall. As they did so, she secretly projected extremely faint images of those fruits — so faint as to be barely visible — on the wall and asked the participants if they saw anything. None of them thought they saw anything real, although they commented on how vivid their imagined image seemed. “If I hadn’t known I was imagining, I would have thought it real,” one participant said.

A 1910 study by the psychologist Mary Cheves West Perky found that when our perceptions match what we are imagining, we assume that their inputs are imaginary. Source: https://doi.org/10.2307/1413350

Perky’s conclusion was that when our perception of something matches what we know we are imagining, we will assume it is imaginary. It eventually came to be known in psychology as the Perky effect. “It’s a huge classic,” said Bence Nanay, a professor of philosophical psychology at the University of Antwerp. It became kind of a “compulsory thing when you write about imagery to say your two cents about the Perky experiment.”

In the 1970s, the psychology researcher Sydney Joelson Segal revived interest in Perky’s work by updating and modifying the experiment. In one follow-up study, Segal asked participants to imagine something, such as the New York City skyline, while he projected something else faintly onto the wall — such as a tomato. What the participants saw was a mix of the imagined image and the real one, such as the New York City skyline at sunset. Segal’s findings suggested that perception and imagination can sometimes “quite literally mix,” Nanay said.

Not all studies that aimed to replicate Perky’s findings succeeded. Some of them involved repeated trials for the participants, which muddied the results: Once people know what you’re trying to test, they tend to change their answers to what they think is correct, Naselaris said.

So Dijkstra, under the direction of Steve Fleming, a metacognition expert at University College London, set up a modern version of the experiment that avoided the problem. In their study, participants never had a chance to edit their answers because they were tested only once. The work modeled and examined the Perky effect and two other competing hypotheses for how the brain tells reality and imagination apart.

Evaluation Networks

One of those alternative hypotheses says that the brain uses the same networks for reality and imagination, but that functional magnetic resonance imaging (fMRI) brain scans don’t have high enough resolution for neuroscientists to discern the differences in how the networks are used. One of Muckli’s studies, for example, suggests that in the brain’s visual cortex, which processes images, imaginary experiences are coded in a more superficial layer than real experiences are.

Nadine Dijkstra, a postdoctoral researcher at University College London, led the new study that probed how brains distinguish between what is really perceived and what is imagined. Courtesy of Nadine Dijkstra

With functional brain imaging, “we’re squinting our eyes,” Muckli said. Within each equivalent of a pixel in a brain scan, there are about 1,000 neurons, and we can’t see what each one is doing.

The other hypothesis, suggested by studies led by Joel Pearson at the University of New South Wales, is that the same pathways in the brain code for both imagination and perception, but imagination is just a weaker form of perception.

During the pandemic lockdown, Dijkstra and Fleming recruited for an online study. Four hundred participants were told to look at a series of static-filled images and imagine diagonal lines tilting through them to the right or left. Between each trial, they were asked to rate how vivid the imagery was on a scale of 1 to 5. What the participants did not know was that in the last trial, the researchers slowly raised the intensity of a faint projected image of diagonal lines — tilted either in the direction the participants were told to imagine or in the opposite direction. The researchers then asked the participants if what they saw was real or imagined.

Vivid imagery is more like perception, but whether faint perception is more like imagery is less clear. - Nadine Dijkstra, University College London

Dijkstra expected that she would find the Perky effect — that when the imagined image matched the projected one, the participants would see the projection as the product of their imagination. Instead, the participants were much more likely to think the image was really there.

Yet there was at least an echo of the Perky effect in those results: Participants who thought the image was there saw it more vividly than the participants who thought it was all their imagination.

In a second experiment, Dijkstra and her team didn’t present an image during the last trial. But the result was the same: The people who rated what they were seeing as more vivid were also more likely to rate it as real.

The observations suggest that imagery in our mind’s eye and real perceived images in the world do get mixed together, Dijkstra said. “When this mixed signal is strong or vivid enough, we think it reflects reality.” It’s likely that there’s some threshold above which visual signals feel real to the brain and below which they feel imagined, she thinks. But there could also be a more gradual continuum.

To learn what’s happening within a brain trying to distinguish reality from imagination, the researchers reanalyzed brain scans from a previous study in which 35 participants vividly imagined and perceived various images, from watering cans to roosters.

In keeping with other studies, they found that the activity patterns in the visual cortex in the two scenarios were very similar. “Vivid imagery is more like perception, but whether faint perception is more like imagery is less clear,” Dijkstra said. There were hints that looking at a faint image could produce a pattern similar to that of imagination, but the differences weren’t significant and need to be examined further.

Scans of brain function show that imagined and perceived images trigger similar patterns of activity, but the signals are weaker for the imagined ones (at left). Courtesy of Nadine Dijkstra

What is clear is that the brain must be able to accurately regulate how strong a mental image is to avoid confusion between fantasy and reality. “The brain has this really careful balancing act that it has to perform,” Naselaris said. “In some sense it is going to interpret mental imagery as literally as it does visual imagery.”

They found that the strength of the signal might be read or regulated in the frontal cortex, which analyzes emotions and memories (among its other duties). But it’s not yet clear what determines the vividness of a mental image or the difference between the strength of the imagery signal and the reality threshold. It could be a neurotransmitter, changes to neuronal connections or something totally different, Naselaris said.

It could even be a different, unidentified subset of neurons that sets the reality threshold and dictates whether a signal should be diverted into a pathway for imagined images or a pathway for genuinely perceived ones — a finding that would tie the first and third hypotheses together neatly, Muckli said.

Even though the findings are different from his own results, which support the first hypothesis, Muckli likes their line of reasoning. It’s an “exciting paper,” he said. It’s an “intriguing conclusion.”

But imagination is a process that involves much more than just looking at a few lines on a noisy background, said Peter Tse, a professor of cognitive neuroscience at Dartmouth College. Imagination, he said, is the capacity to look at what’s in your cupboard and decide what to make for dinner, or (if you’re the Wright brothers) to take a propeller, stick it on a wing and imagine it flying.

The differences between Perky’s findings and Dijkstra’s could be entirely due to differences in their procedures. But they also hint at another possibility: that we could be perceiving the world differently than our ancestors did.

Her study didn’t focus on belief in an image’s reality but was more about the “feeling” of reality, Dijkstra said. The authors speculate that because projected images, video and other representations of reality are commonplace in the 21st century, our brains may have learned to evaluate reality slightly differently than people did just a century ago.

Even though participants in this experiment “were not expecting to see something, it’s still more expected than if you’re in 1910 and you’ve never seen a projector in your life,” Dijkstra said. The reality threshold today is therefore likely much lower than in the past, so it may take an imagined image that’s much more vivid to pass the threshold and confuse the brain.

A Basis for Hallucinations

The findings open up questions about whether the mechanism could be relevant to a wide range of conditions in which the distinction between imagination and perception dissolves. Dijkstra speculates, for example, that when people start to drift off to sleep and reality begins blending with the dream world, their reality threshold might be dipping. In conditions like schizophrenia, where there is a “general breakdown of reality,” there could be a calibration issue, Dijkstra said.

“In psychosis, it could be either that their imagery is so good that it just hits that threshold, or it could be that their threshold is off,” said Karolina Lempert, an assistant professor of psychology at Adelphi University who was not involved in the study. Some studies have found that in people who hallucinate, there’s a sort of sensory hyperactivity, which suggests that the image signal is increased. But more research is needed to establish the mechanism by which hallucinations emerge, she added. “After all, most people who experience vivid imagery do not hallucinate.”

Nanay thinks it would be interesting to study the reality thresholds of people who have hyperphantasia, an extremely vivid imagination that they often confuse with reality. Similarly, there are situations in which people suffer from very strong imagined experiences that they know are not real, as when hallucinating on drugs or in lucid dreams. In conditions such as post-traumatic stress disorder, people often “start seeing things that they didn’t want to,” and it feels more real than it should, Dijkstra said.

Some of these problems may involve failures in brain mechanisms that normally help make these distinctions. Dijkstra thinks it might be fruitful to look at the reality thresholds of people who have aphantasia, the inability to consciously imagine mental images.

The mechanisms by which the brain distinguishes what’s real from what’s imaginary could also be related to how it distinguishes between real and fake (inauthentic) images. In a world where simulations are getting closer to reality, distinguishing between real and fake images is going to get increasingly challenging, Lempert said. “I think that maybe it’s a more important question than ever.”

Dijkstra and her team are now working to adapt their experiment to work in a brain scanner. “Now that lockdown is over, I want to look at brains again,” she said.

She eventually hopes to figure out if they can manipulate this system to make imagination feel more real. For example, virtual reality and neural implants are now being investigated for medical treatments, such as to help blind people see again. The ability to make experiences feel more or less real, she said, could be really important for such applications.

It’s not outlandish, given that reality is a construct of the brain.

“Underneath our skull, everything is made up,” Muckli said. “We entirely construct the world, in its richness and detail and color and sound and content and excitement. … It is created by our neurons.”

That means one person’s reality is going to be different from another person’s, Dijkstra said: “The line between imagination and reality is just not so solid.”