Printed from http://www.electronista.com

Medical software varies results by OS, hardware, version

updated 11:46 pm EDT, Mon June 18, 2012

Mac, Linux see up to five percent variance in data

Neurological imaging tool FreeSurfer, curated at the Martinos Center for Biomedical Imaging, is a commonly-used open-source software tool employed by researchers to measure cortical thickness and volume of varying brain structures from the MRI scan of a patient. Recent research has shown that the calculations, given the same input, can vary up to 15 percent between different versions of the software, and up to five percent between Linux and OSX. No explanation has been given by the curators for the phenomenon.

To test the software, a study pool of 274 participants was separated into two groups plus a control group. Thirty scans were picked at random, 10 out of each group. The FreeSurfer application was installed on two different iMacs, each with a different processor and operating system, along with three different Xeon-based Mac Pro systems running OS X 10.5 and 10.6, and a single HP workstation under CentOS 5.3, also with a Xeon processor.

Three different FreeSurfer versions were used to count 190 subcortical volumes, and 68 cortical thicknesses. The same brain re-calculated on the same workstation didn't see any variance. All the Macintosh workstations on the same operating system and software revision had identical results, regardless of processor and RAM configuration. Differences, however, were revealed between the HP and any Macintosh station, including the similar architecture Xeon-based Mac Pro. The individual FreeSurfer versions also calculated brain volumes differently. OS X 10.5 also unexpectedly calculated differently than OS X 10.6 with all other variables unchanged.

Depending on brain location, results varied by between five and 15 percent between versions of FreeSurfer. Changes in platform or major OS revisions differed less, between two and five percent.

Electronista contacted a major children's neurology center for insight, and was told that hospital information technology workers take steps to insure that both OS and hardware variances are strictly controlled in the research environment to minimize this effect. Small differences in mathematics have been known to show up induced by hardware or software migrations in disciplines involved in complicated, floating point mathematics -- such as architecture or ballistics.

The developers behind FreeSurfer are aware of this phenomenon. The software license for the package states that the package is for comparative research purposes only, and has not been approved by the FDA for clinical use. Specifically, "clinical applications are neither recommended nor advised," according to FreeSurfer's license.

Electronista has contacted the FreeSurfer curators for comment beyond the EULA language, but has not received a reply thus far. The research tool is intended for like comparisons -- while MRI data can be fed into two different versions of FreeSurfer and two different answers may result, data from different brains can be fed into the same version of the software, and comparable results will result, which is the intent of the open-source tool. No evidence of any research error or misdiagnoses has been reported as a result of the discrepancies. [via PubMed]



By Electronista Staff
Post tools:

TAGS :

toggle

Comments

  1. The Vicar

    Junior Member

    Joined: Jul 2009

    -3

    Nothing like Open Source...

    ...for reliability and utility. I hope I'm never in a life-or-death situation where open source will make a difference.

  1. eggman

    Mac Enthusiast

    Joined: Aug 2002

    +5

    Floating point?

    My guess is that we're looking at a difference between implementation of floating point arithmetic. Increasingly, to boost performance, we're seeing math that used to be done by a CPU offloaded to a GPU to be done in hardware - in parallel - in realtime, and I'll lay odds that the floating point math routines vary.

  1. LenE

    Fresh-Faced Recruit

    Joined: May 2004

    -2

    I lost count

    How many train wrecks are here? What is the point of this article?

    Who did the testing, and why we're their methods so poor? What was the result?

    I work a lot with high school students, and it baffles me how simple concepts like the scientific method seem to easily escape them. Here, the same concept apparently escapes medical researchers, which really calls into question their more important experiments, beyond qualifying medical software.

    For all I know, they have done a good experiment, and the writing staff here butchered it. Based on the haphazard description of the experiment, and total lack of apparent controls, I doubt it.

  1. LenE

    Fresh-Faced Recruit

    Joined: May 2004

    +2

    See the problem

    The image above appears to show the problem. The user is asking for a volume calculation for tissue in the right ventricle (this is a drainage passage for cerebrospinal fluid), and it is only partially highlighting the chamber in purple. The shade of gray and similarity to near-neighbor pixels is how it determines what to select.

    Between 10.5 and 10.6, the display gamma was shifted from1.8 to 2.2. Most applications wouldn't have a problem with this, as they would be using the proper frameworks that handle color management of image data. This program, being cross-platform generically, and occurring on Linux in specific, would most likely not have a clue about color management. This may be why the calculation results are different between 10.5 and 10.6, if the selection criteria depends on a click on the screen to select the gray value to identify in the MRI image.

  1. Arne_Saknussemm

    Forum Regular

    Joined: Apr 2011

    +2

    @The Vicar - AVOID at all cost -

    most email/bank/blogs/news sites, traffic management systems (most roads for that matter), and quite a few services having Linux or one of it's variants at it's core. For they too use either opensourced applications or worse yet OSs!

  1. testudo

    Forum Regular

    Joined: Aug 2001

    +2

    @LenE

    I gather the results were coming from the same 'data' set, not just the same subjects scanned over and over again. But I may misread.

    And the point of the article seems quite clear. What FreeSurfer is good for and what it isn't (which is clear). Running the same data on the same machine gives the same results. Just don't send that data to another user running a different platform and go ballistic when the results are different.

    So, when researchers (which is what the tool is used for) are going through a set of data, the note here is "Do it all on the same machine!". That's all. It's important, but not earth shattering.

  1. testudo

    Forum Regular

    Joined: Aug 2001

    +2

    Re: Nothing like Open Source...

    ...for reliability and utility. I hope I'm never in a life-or-death situation where open source will make a difference.

    Yeah, because if this software was closed source, I'm sure it would be 100% perfect, no bugs, and completely free of variances on any platform.

    Oh, and since it is open source, nothing prevents someone from taking the source, equalizing out the variances on each platform, and getting it to be 100% consistent, either.

  1. testudo

    Forum Regular

    Joined: Aug 2001

    +1

    Re: Nothing like Open Source...

    BTW, this is why a lot of cross-platform companies don't waste their resources using built-in OS frameworks. For example, people whine that Adobe Photoshop doesn't use Core this or Core that on OS X, but they don't do it because they'd still have to write one for Windows, and then they most likely would end up working differently. Thus, they just want to use one set of libraries to try to get the most consistent results, platform be damned.

Login Here

Not a member of the MacNN forums? Register now for free.

toggle

Network Headlines

toggle

Most Popular

Sponsor

Recent Reviews

Kenu Airframe Plus

Simple, stylish and effective, the Kenu Airframe + portable car mount is the latest addition to Kenu's lineup. Released earlier this y ...

Plantronics Rig Surround 7.1 headset

Trying to capture the true soundscape of video games can be a daunting task. Looking to surround-sound home theater options, users hav ...

Adesso Compagno X Bluetooth keyboard

The shift from typing on physical keyboards to digital versions on smartphones and tablets hasn't been an easy for many consumers. Fro ...

Sponsor

toggle

Most Commented

 
toggle

Popular News