LiU Reader Eklund who works at the Department of Computer and Information Science and the Department of Biomedical Engineering, respectively, tested the analysis methods by using them on known, reliable data. The methods showed false activity in the brain in 60 percent of the cases. A reasonable figure is five percent.
The analysis methods were previously only validated against simulated data, but Dr Eklund pointed out back in 2012, in his doctoral thesis, that the results are not always reliable. The statistical methods used are built on a number of assumptions; if one or more of the assumptions are incorrect, the results will also be incorrect. At that time, critics maintained that errors could certainly arise if data from a single person was analysed, but that the errors would even out in group analyses.
In his thesis, Dr Eklund proposed another method in which few assumptions are made and significantly more calculations – a thousand times more – are done, which yields a significantly more certain result. With the help of modern graphics cards, the processing time can be reduced so that the method is usable in practice.
Dr Eklund went off to a postdoc position at Virginia Tech. When he came home in the autumn of 2014, he realised that research was still resting on old, unreliable analysis methods.
“I thought perhaps that someone would pick up the thread and continue validating the methods, but that didn’t happen. That’s why when I came back, I bought three computers and started testing the statistical methods for group analyses. Whatever I found, it would be interesting, I thought.”
An MRI examination can cost SEK 5,000 for an hour, and examining 500 people is most often not financially possible. But over the last few years, more and more researchers have started sharing their collected data, so that more researchers can make use of it.
“Today we can download actual measurement data from healthy control subjects,” he says.
When examinations are done using fMRI, for example, images from 20 healthy people are compared with images from 20 people who have a brain injury or some impaired function.
Dr Eklund now used the current analysis methods and compared 20 healthy people with 20 other healthy people. In other words, there should not have been any differences – or, in any case only the five percent that chance provides. In total he made three million comparisons of randomly selected groups with data from 499 healthy persons.
“The differences were considerably greater than five percent, up to 60 percent in the worst case,” Dr Eklund says.
This means that the analyses could have shown positive results where there shouldn’t have been any, thereby indicating brain activity where there was no activity.
He also analysed the same data set with his more calculation-heavy method and obtained a considerably better correspondence, with differences in the expected five percent of cases.
“Thanks to modern graphics cards, large calculations can be run. It would take 1,000 times longer to run the calculations using a normal computer, but thanks to the graphics cards I reduced the processing time from 10 years to 20 days,” Dr Eklund tells us.
Anders Eklund“If you’ve spent months gathering data at great cost, you should be more interested in letting the analysis take time so that it’s correct,” he says.
An American research group has also repeated his calculations. Dr Eklund, Dr Knutsson (professor at the Department of Biomedical Engineering), and their colleague at the University of Warwick in Great Britain also chose to put out what is known as a ‘pre-print’ on arXiv so that other researchers could present their objections.
But no mistakes have been found in Dr Eklund’s calculations, and the results of his serious toil in research are now being published in the esteemed American scientific journal Proceedings of the National Academy of Sciences (PNAS).
“It really feels great; it’s recognition and I hope we can get a discussion going in research circles regarding how we validate models. Today, there is both data available to validate and enough processing power to perform the calculations.”
How may of the studies done with fMRI need to be redone?
“I can’t answer that; many of them were done 10 to 15 years ago and it’s not certain that the basic data still exists. The important thing is that researchers think about what method they use in future,” Dr Eklund says.
In one of the software programmes used until now, the functions have also been changed so that the results are more correct.
“Changes can happen quickly sometimes, and this shows that they have accepted that the method previously could have shown incorrect results,” Dr Eklund says.
The research was financed by LiU investments in neuroeconomy, by the Swedish Research Council, and through the European ITEA 3 programme BENEFIT.
Article:
Cluster failure: why fMRI inferences for spatial extent have inflated false positive rates., Anders Eklund, Thomas Nichols and Hans Knutsson, PNAS 2016,
The analysis methods were previously only validated against simulated data, but Dr Eklund pointed out back in 2012, in his doctoral thesis, that the results are not always reliable. The statistical methods used are built on a number of assumptions; if one or more of the assumptions are incorrect, the results will also be incorrect. At that time, critics maintained that errors could certainly arise if data from a single person was analysed, but that the errors would even out in group analyses.
New method
In his thesis, Dr Eklund proposed another method in which few assumptions are made and significantly more calculations – a thousand times more – are done, which yields a significantly more certain result. With the help of modern graphics cards, the processing time can be reduced so that the method is usable in practice.
Dr Eklund went off to a postdoc position at Virginia Tech. When he came home in the autumn of 2014, he realised that research was still resting on old, unreliable analysis methods.
“I thought perhaps that someone would pick up the thread and continue validating the methods, but that didn’t happen. That’s why when I came back, I bought three computers and started testing the statistical methods for group analyses. Whatever I found, it would be interesting, I thought.”
An MRI examination can cost SEK 5,000 for an hour, and examining 500 people is most often not financially possible. But over the last few years, more and more researchers have started sharing their collected data, so that more researchers can make use of it.
“Today we can download actual measurement data from healthy control subjects,” he says.
When examinations are done using fMRI, for example, images from 20 healthy people are compared with images from 20 people who have a brain injury or some impaired function.
Compared healthy with healthy
Dr Eklund now used the current analysis methods and compared 20 healthy people with 20 other healthy people. In other words, there should not have been any differences – or, in any case only the five percent that chance provides. In total he made three million comparisons of randomly selected groups with data from 499 healthy persons.
“The differences were considerably greater than five percent, up to 60 percent in the worst case,” Dr Eklund says.
This means that the analyses could have shown positive results where there shouldn’t have been any, thereby indicating brain activity where there was no activity.
He also analysed the same data set with his more calculation-heavy method and obtained a considerably better correspondence, with differences in the expected five percent of cases.
“Thanks to modern graphics cards, large calculations can be run. It would take 1,000 times longer to run the calculations using a normal computer, but thanks to the graphics cards I reduced the processing time from 10 years to 20 days,” Dr Eklund tells us.
Anders Eklund“If you’ve spent months gathering data at great cost, you should be more interested in letting the analysis take time so that it’s correct,” he says.
Repeated calculations
An American research group has also repeated his calculations. Dr Eklund, Dr Knutsson (professor at the Department of Biomedical Engineering), and their colleague at the University of Warwick in Great Britain also chose to put out what is known as a ‘pre-print’ on arXiv so that other researchers could present their objections.
But no mistakes have been found in Dr Eklund’s calculations, and the results of his serious toil in research are now being published in the esteemed American scientific journal Proceedings of the National Academy of Sciences (PNAS).
“It really feels great; it’s recognition and I hope we can get a discussion going in research circles regarding how we validate models. Today, there is both data available to validate and enough processing power to perform the calculations.”
How may of the studies done with fMRI need to be redone?
“I can’t answer that; many of them were done 10 to 15 years ago and it’s not certain that the basic data still exists. The important thing is that researchers think about what method they use in future,” Dr Eklund says.
In one of the software programmes used until now, the functions have also been changed so that the results are more correct.
“Changes can happen quickly sometimes, and this shows that they have accepted that the method previously could have shown incorrect results,” Dr Eklund says.
The research was financed by LiU investments in neuroeconomy, by the Swedish Research Council, and through the European ITEA 3 programme BENEFIT.
Article:
Cluster failure: why fMRI inferences for spatial extent have inflated false positive rates., Anders Eklund, Thomas Nichols and Hans Knutsson, PNAS 2016,
