Computer Model 'Reads' Letters Directly from the Human Brain

Want to know what someone is thinking? How about what someone is reading? Scientists have discovered a way to analyze MRI images of the brain in order to determine which letter a person is looking at, revealing a way to reconstruct thoughts more accurately than ever before.

Functional MRI scanners have been used in the past to "read" the thoughts of individuals. However, they've mainly just been used to determine which brain areas are active while participants in studies perform a certain task. Yet making these scans even more specific has long been an intriguing goal for scientists.

In order to refine MRI scans, the researchers "taught" a model how small volumes of 2x2x2 mm from the brain scans, known as voxels, respond to individual pixels. By combining all of the information about the pixels from the voxels, the scientists reconstructed the image that the subject was viewing--in this case, handwritten letters. This image wasn't exactly clear, though; instead, it formed in a fuzzy speckle pattern. The scientists knew they had to refine the technique a bit further.

"After this we did something new," said Marcel van Gerven, the lead researcher, in a news release. "We gave the model prior knowledge: we taught it what letters look like. This improved the recognition of the letters enormously. The model compares the letters to determine which one corresponds most exactly with the speckle image, and then pushes the results of the image towards that letter. The result was the actual letter, a true reconstruction"

The latest technique, in fact, is similar to how the researchers believe the brain itself combines prior knowledge with sensory information. For example, you can only recognize letters once you learn to read; the scientists had to essentially teach the models to "read" before it could recognize the letters that the individual was looking at.

In the end, the researchers were able to successfully employ a model to interpret the MRI scans. This, in turn, allowed them to see exactly which letters a subject was viewing at the time of the scan.

"In our future research we will be working with a more powerful MRI scanner," said Sanne Schoenmakers, one of the researchers, in a news release. "Due to the higher resolution of the scanner, we hope to be able to link the model to more detailed images. We are currently linking images of letters to 1200 voxels in the brain; with the more powerful scanner we will link images of faces to 15,000 voxels."

The findings are published in the journal NeuroImage.