fMRI bugs could upend years of research

A whole pile of “this is how your brain looks like” fMRI-based science has been potentially invalidated because someone finally got around to checking the data. The problem is simple: to get from a high-resolution magnetic resonance imaging scan of the brain to a scientific conclusion, the brain is divided into tiny “voxels”. …

[Peter Prof Fox]

Good science

This is how science is supposed to work. Obviously the previously 'good' science is now 'oops science' but not intentionally bad...

Except, wait for the backlash from researchers who find their findings, at the very least 'tainted'. Which option is easier: Re-doing the work based on evaporated data and no grant, or slagging-off those bold folk who pointed out the Emperor had no clothes?

[Geoffrey W]

Re: Good science

And now I suppose we wait to see if your predicted outcome in fact happens or if your hypothesis is also bad science. Real science is durn slow innit?

[Anonymous Coward]

Re: Good science

I'm sure they'll slag off the people who discovered the problem.

On the same note: It turns out people at the USGS have been manipulating climate data since 1996.

They won't say who did it, they won't say if they were punished, they did say no charges will be filed - and I imagine all the studies based on that data will still be cited, quoted, etc.

You'd expect that to be showing up in the news, wouldn't you?

[Palpy]

Re: USGS manipulating data --

-- link to evidence, or it didn't happen. That is all.

[Anonymous Coward]

Link To Evidence

Here is a link:

http://dailycaller.com/2016/06/23/federal-lab-forced-to-close-after-disturbing-data-manipulation/

[Anonymous Coward]

Re: USGS manipulating data --

Link from technical journal here:

http://cen.acs.org/articles/94/i26/USGS-finds-data-fraud-closes.html

[Alan W. Rateliff, II]

Re: USGS manipulating data --

"link to evidence, or it didn't happen. That is all."

Will this do to start?

https://stevengoddard.wordpress.com/tracking-us-temperature-fraud/

(He has since moved here: http://realclimatescience.com/ )

There are plenty more sources if you care to search on your own.

[John Gamble]

Re: USGS manipulating data --

"Will this do to start?"

No, that won't do.

He has a BS in geology from Arizona State University and a Master's degree in electrical engineering from Rice University.

One of Goddard's earliest writings, an article for The Register, asserted that the National Snow and Ice Data Center's (NSIDC) data underlying a chart depicting 2008 Arctic sea ice loss was incorrect and that NSIDC seemed to demonstrate "a consistent pattern of overstatement related to Arctic ice loss." Ten days later, however, Goddard acknowledged that the data on which the graph was based was accurate.

I'd like something from a real scientist please.

[Pompous Git]

Re: USGS manipulating data --

I'd like something from a real scientist please.

Like Al Gore, you mean? :-)

[waldo kitty]

Re: USGS manipulating data --

like this one?

http://www.telegraph.co.uk/news/earth/environment/globalwarming/11395516/The-fiddling-with-temperature-data-is-the-biggest-science-scandal-ever.html

and others found here??

https://www.google.com/search?q=climate+data+manipulation

[Geoffrey W]

Re: Good science

@Meldreth

RE: "You'd expect that to be showing up in the news, wouldn't you?"

If it isn't showing up in the news then where did you hear or read it? Come on Victor, you don't expect us to just accept your statement do you?

"I heard it on the internet" is Donald Trumps methodology. You've gotta be able to do better than that! I don't even demand a direct link to actual data - Just a note as to your general source will do for me so I can go and read up about it myself.

[Anonymous Coward]

Re: Good science

@Geoffrey W

If it isn't showing up in the news then where did you hear or read it? Come on Victor, you don't expect us to just accept your statement do you?

It isn't showing up in the MSM because they only regurgitate science press releases and anyone that goes against the 'settled science' is a non believer and should be prosecuted as several US Aturnies General are trying to do.

If they manage that then we will see all the 'adjustments' used to fit the climate science dogma exposed and the MSM will then get the news.

[Geoffrey W]

Re: Good science

@Ivan 4

I realise that but my point is, if the story of manipulated results isn't in the MSM then where did he read it? If he wasn't personally involved then he must have read it somewhere else. I just hope it isn't somewhere like World Net Daily unless they have other solid references.

[Anonymous Coward]

Re: Good science

Uh oh: think I just heard "...they only regurgitate...":. Another absolute 100% fact, it seems.

[Anonymous Coward]

Re: Good science

Link from technical journal here:

http://cen.acs.org/articles/94/i26/USGS-finds-data-fraud-closes.html

[Anonymous Coward]

Re: Good science / USGS manipulating data

OK, so the same thing as the Denver Post story I linked above. Has nothing to do with climate in terms of global warming, but of measuring contaminants in the environment.

The poster who made the original claim in this thread didn't understand the article and instead saw what he wanted to see, "manipulation of climate science", causing him to make made a false claim. I'm sure he's not the only one making that mistake - he may not have made the mistake himself, but saw an article written by someone who made the mistake (or was trying to deliberately mislead people) and quoted a few blurbs out of context making it sound like the USGS was manipulating climate data like temperature records.

Some on the other side will point to stuff like this trying to make the case that all claims of climate manipulation are false or misunderstandings - even when they know damn well that's not the case.

[Anonymous Coward]

Re: Good science / USGS manipulating data

"Has nothing to do with climate in terms of global warming, but of measuring contaminants in the environment."

Well, that is disingenuous at best.

One of the tenets of global warming is that it is a man-made phenomenon driven by "contaminants in the environment."

Lets pick coal, one of those contaminants specifically cited in the article.

Government policy in this country is that coal is one of the factors driving global warming.

This is a quote from the NY Times from Interior Secretary Jewell saying that the federal policy on coal "...takes into account its impacts on climate change.”

So the main question is:

Why does a government lab knowingly manipulate data?

A logical assumption might be that it is done because the actual data does say what the government wants it to say.

Perhaps someone else has another explanation?

[Palpy]

Re: USGS manipulating data: bogus alarm.

Another explanation, Meldreth: as noted, this is an inorganic chemistry lab. Research included "...uranium in the environment, health effects of energy resources, and U.S. coal resources and reserves".

Nothing to do with global warming was mentioned, implied, or even a consequence of this particular chem lab's malfeasance.

Nothing.

Your claim -- "people at the USGS have been manipulating climate data since 1996" -- is utter whubchunkies. FUD. A lie by misdirection.

Oh, let's just be straight with each other: your claim was a lie.

Sayonara.

[Anonymous Coward]

Re: USGS manipulating data: bogus alarm.

"utter whubchunkies. FUD. A lie by misdirection."

"your claim was a lie."

Oh, dear.

I think after the reading the articles I linked to and the information in my posts, any reasonable person would think my characterization was reasonable.

But we've left reason far behind, haven't we?

This is all about offending the god of the true believer now, isn't it?

Well, I'll leave you to rant and fume - must suck to be you.

[Pompous Git]

Re: USGS manipulating data: bogus alarm.

But we've left reason far behind, haven't we?

Much to The Git's amusement, Australia's climastrologists were so successful in their claim that "the science is settled", that their employer (CSIRO) sacked most of them. Why fund research where none is needed?

[Adrian Midgley 1]

You are in a hole

and digging furiously.

I have concluded that you cannot be relied upon to report anything accurately in any context.

(And if you can't tell the difference between CO2 and coal, don't get involved in science please)

[Edward Ashford]

Re: Good science

It's reported in the Denver Post 1 July 2016.

[Anonymous Coward]

Re: Good science

The only Denver Post story I found on July 1st 2016 was about falsifying test results of chemical analysis for toxic metals, apparently done in an attempt to "correct calibration errors" on a mass spectrometer.

Nothing to do with climate data.

http://www.denverpost.com/2016/07/01/investigators-us-lab-worker-in-colorado-faked-test-results/

[Anonymous Coward]

Re: Good science

THis link: http://dailycaller.com/2016/06/23/federal-lab-forced-to-close-after-disturbing-data-manipulation/

[NotBob]

Re: Good science

All these people asking for proof of climate data manipulation is great, but has everyone forgotten climategate? It happened, we knew it happened, and no one got in trouble because it supported the official line that was to be toed...

[Adrian Midgley 1]

Re: Good science

Climategate was bollux.

You can determine that with a thermometer and sufficient years.

Some interests preferred to pretend it wasn't happening, rather than change anything they did to make it happen.

[Pompous Git]

Re: Good science

Climategate was bollux.

In that case why have the writers of the emails never repudiated their contents? How do you test the validity of a the contents of an email "with a thermometer and sufficient years"?

[Adrian Midgley 1]

So the claim it was _climate_ data is completely false...

What was the cause of the error in reporting?

Did you want it to be climate, to the extent that you see a report about a bad lab practice with metals and read it as climate, and fail to retain or indicate that it is one lab rather than the whole orgnaisation (and people rather than person),

... or are you just unreliable?

[Code For Broke]

Re: Good science

In medicine, as is increasingly true in other branches of science, research is largely conducted in the interests of corporations. This too often results not in a drive to learn and improve from mistakes, but to profit before the mistakes become obvious.

[John Brown (no body)]

Re: Good science

"This too often results not in a drive to learn and improve from mistakes, but to profit before the mistakes become obvious."

Or, more likely, to produce the "desired" results.

[Uffish]

Re: Good science

No, it's bad science. The measuring instruments were all (?) badly calibrated. With all due respect to the people involved in the experiments, they didn't check their equipment properly.

That's a problem nowadays, you get a big shiny beige box with knobs on it and you think it must be outputting the truth. Well it ain't necessarily so and the shiny beige box will always need some sort of calibration validation traceable back to something real, not just something repeatable.

[Version 1.0]

Re: big shiny beige box

I see this all the time - coming soon to a research department near you, users wanting to run their MRI analysis on their cell phones with an app.

[Stoneshop]

Re: Good science

The measuring instruments were all (?) badly calibrated.

The problem is not the scanner; going from MR data to images does produce images that look quite like slicing your body and taking piccies of that, but it's less invasive. Living people tend to prefer it that way. The problem is the statistics software used to correlate bits of brain activity, and while it's essential to the research mentioned, I wouldn't classify it as 'instruments', those being something you can kick.

[Uffish]

Re: You can kick instruments

Point taken but consider a multi-million pound telescope, if it ends up taking a photo of the wrong galaxy it is probably the fault of the software in the pointing mechanism but it is still the telescope that is malfunctioning and I bet the telescope operators take a few calibration photos now and again just for a reality check.

[tony2heads]

Re: You can kick instruments

Calibration is a regular part of any scientific instrument, both internal (do sensors respond in a reasonable way) and external (check against various known results from a totally different instrument using different software).

[Stoneshop]

Re: You can kick instruments

MRI, as a tool for looking inside your noggin while it's still more or less functioning*, is quite OK. Which can, and has been, calibrated by scanning corpses and cutting them up**.. What happens to be a problem is interpreting correlations in activity in different brain areas, for instance the physical stimuli as caused by consuming Coke (or coke), and the associated feelings.

* mine nearly wasn't, and MRI showed the cause, allowing the correct medical treatment. Guessing would have had well over 50% chance of being dead wrong.

** Or so I've been told by someone working at Philips Medical Systems.

[Alan Brown]

Re: Good science

"The problem is not the scanner"

The problem is that the raw data was not preserved. This is a fundamental violation of repeatability principles in scientific work. If this kind of "scientist" was around in the 1970s, the Ozone hole would have taken another 20 years to confirm(*)

(*) The software processing NOAA data dumped ozone readings that were "too high or too low", which is why the ozone hole took too long to be confirmed, but because the instrument raw data was kept, once the software error was realised, the entire dataset was rerun and the ozone hole's evolution became blindingly obvious. Without that data it would have taken at least another decade to get enough data to verify that it was growing and that it was man-caused.

[Adrian Midgley 1]

raw data ... what is raw, and what is data?

My area of interest is photography.

The sensor in a camera gets photons, and something happens because of that, which produces voltages, which are recorded.

In order to produce a picture - what people actually want from a camera in the end - those voltages have to be processed into something to keep. Raw camera files are big - and less raw than the direct effects on the sensor, fairly direct though they are. The camera processes them into a JPEG image.

If you retain the raw file (often given as RAW, for no reason clear to me) you can repeat that processing, and you can do darkroom tricks on it. (In film days or terms the undeveloped negative is the raw data, but you can't even read it until you process it, which requires decisions which influence the picture produced. Operator effort and skill is involved)

Reuters was recently reported to have required photographers no longer to record raw files, in search of unedited unaltered images from wherever they are being taken.

SImilarly, I suspect there is rather a lot of raw data off those big magnetic whirry 3-D cameras and even though storage is cheaper, keeping it forever or even long enough to get it off the box is not entirely likely.

Bottom Line: only select Open Source software for medicine and science if you want to make it difficult to be prevented from detecting faults.

Now, Reuters

[JimC]

Re: raw data ... what is raw, and what is data?

Is it just me, or is that last but one paragraph (Bottom Line:) a monumental non sequitur? I use Open source software and recommend it, but I don't see it has much connection with whether or not the unprocessed data is discarded or not...

[tom dial]

Re: raw data ... what is raw, and what is data?

Open source software not only has nothing to do with whether raw data was/was not retained, but also cannot be assumed to be more correct or free from error than closed source. I also use, and recommend it, but do not delude myself that it is free from error, and I have plenty of examples to show it is not.

[Alan Brown]

Re: Good science

" you get a big shiny beige box with knobs on it and you think it must be outputting the truth."

_THIS_ in spades. The current crop of researchers just blindly accept whatever "computer says" without crosschecking results or understanding _how_ those results were obtained.

Or even if they're actually using an appropriate tool. These are the same people who in woodwork class would be attempting to bang in nails using a carpenters' plane because it was on the workbench, or trying to add water to concentrated sulphuric acid in chemistry class after not reading the safety warnings.

[Ian Michael Gumby]

Re: Good science

Here's the $64K (USD) question...

Is the underlying data bad, or just the interpretation done by the software?

If the underlying data is good and preserved, its possible to redo the analysis, however I get the impression that the data wasn't saved.

[Alan Brown]

Re: Good science

"Is the underlying data bad, or just the interpretation done by the software?"

This question cannot be answered, as the underlying data (the raw input) has been dumped.

[tom dial]

Re: Good science

Not saving the raw data would be a definitely Bad Thing. One of the natural things, especially for unexpected or novel results would be for other researchers to want to analyze it in different ways or perform consistency or sanity checks using different analytic tools.

[Draco]

Re: Good science

I would rather think it is more akin to "newspeak" and "doublethink" where you believe as true what is told to you by the "authorities" (in this case "scientists") - whether or not it changes from day to day (or decade to decade).

How can people say, "See, this is how science is supposed to work" when the previous results and research are all batshit from the researchers arses? Ah! It is because today's "facts" and "research" overturn yesterday's truths.

I think it would be better if, rather than pretending it is fact / truth / knowledge, we all say instead, "I BELIEVE!" with the proviso: "My beliefs are subject to change / adaptation / maturation as I see fit."

[Schultz]

Cargo Cult Science

Cargo Cult Science is a term coined by Physicist Richard Feynman to describe the lot of scientists that go through the motions of producing science, but they are not really contributing anything. It's usually caused by scientists who don't quite understand the underpinnings of their field (i.e. statistics). If you didn't yet read the book "Surely You're Joking, Mr. Feynman!", then go get it, it's a delight.

Unfortunately it's not easy to identify cargo cult science -- in this example it apparently too 15 years. So don't get your hopes up for saving the corresponding scientific funds.

[a_yank_lurker]

Re: Cargo Cult Science

Cargo cult science is a bit harsh for most. I doubt very few of the researchers would or should have a clue about validating third party proprietary software. That is on the vendor to get it right since they wrote it and sold it. The researchers at some point have to trust others that they did their parts correctly. The bugs generate false positives apparently with some consistency makes spotting it more difficult. Random false positives (and negatives) will be washed out be careful analysis over time.

[Schultz]

"Cargo cult science is a bit harsh for most."

Is it really too harsh? A whole research field went astray for 15 years, huge amounts of money were burned, scientific careers were made and unmade -- and nobody validated the experimental approach? Those scientists had 15 years of praise, now they deserve their five minutes of shame.

[DavCrav]

Re: "Cargo cult science is a bit harsh for most."

"Is it really too harsh? A whole research field went astray for 15 years, huge amounts of money were burned, scientific careers were made and unmade -- and nobody validated the experimental approach? Those scientists had 15 years of praise, now they deserve their five minutes of shame."

Yes, it is. A similar example: I was laying a laminate floor the other day and the edges of the pieces I was cutting never seemed to line up with the wall completely, it was fine because of the gap, so I ignored it, putting it down to either the wall not being straight (which it isn't) or the floor itself being laid slightly off straight (which it obviously will be). I then at one point made another line with two different pieces of equipment and they were different.

Turns out that my right angle I was using for marking the boards was, in fact, not a right angle.

Could I have checked that when I bought it? Yes, of course, but you assume that the company selling right angles sells right angles, and not sort-of right angles. Do I "deserve [my] five minutes of shame"? I am not sure.

Their experimental approach was validated, but it depended on a machine giving correct readings.

[Pompous Git]

Re: "Cargo cult science is a bit harsh for most."

Could I have checked that when I bought it? Yes, of course, but you assume that the company selling right angles sells right angles, and not sort-of right angles. Do I "deserve [my] five minutes of shame"? I am not sure.

No, you don't deserve shame, but you have learnt a valuable lesson: A careful workman never assumes; he checks. First building site I worked on I was taught two things:

Measure twice and cut once.

If you've got time to do it twice, you've got time to do it right the first time.

Easy way to check a carpenter's square is to draw a line while holding it against a straight edge. Flip it and draw another line a smidgeon* away from the first. If the lines are not parallel, the square is useless.

* Technically, a red cunt hair, but if there's no red cunt available, a smidgeon is near enough.

[a_yank_lurker]

Re: "Cargo cult science is a bit harsh for most."

Granted someone deserves being named and shamed for this fiasco. There is a point at which one has to trust someone else to their job correctly. The vendors apparently did not bother to properly test or more telling did not apparently publish their data. To me, the principle blame falls on the vendors because they claimed they did their job correctly and obviously failed. The researchers get a much smaller portion for not asking the correct questions up front which may be obvious in hindsight.

I remember a metallurgy professor commenting about crack detection equipment. Vendors tout the smallest crack it can find. But he said one should ask what is the biggest crack it will miss. The size of the crack is critical when trying to determine if a fatigue failure will occur. And I wonder how many fatigue failures occur because this overlooked. This situation sounds similar. The researchers may not have know the right questions to ask until now.

[Voland's right hand]

Re: Cargo Cult Science

Cargo cult science is a bit harsh for most.

Sort-a. They underlying root cause is that statistics is not taught to biology/med science majors properly around the world.

You need to learn probability theory all the way inclusive of its dependencies from a mathematical standpoint such as calculus to be able to make sense of modern statistics used in Biology and Medicine. I do not know of a country which has that in the biology major curriculum. Chemistry usually studies a basic course which is clearly insufficient by today's standards. Biology and medical sciences - not a chance.

As a result, they cannot make sense of a false result even if it slaps them in the face like a Monty Python wet fish.

[Primus Secundus Tertius]

Re: Cargo Cult Science

While I was a physics undergraduate, I had courses on statistics. I concluded that it was a difficult subject to teach.

To prove the results that applied stats uses needs very advanced maths, which only a mathematical specialist will master. So those results are one-off recipes that have to be memorised individually. Contrast this with the classical geometry of Euclid, where the proofs are relatively straightorward and can easily be checked and re-proved if one needs to do that.

We therefore reach the situation in statistics where in the physical sciences they are usually reasonable, in the biosciences they are wobbly, and in the social "sciences" they are non-existent. That last is particularly grievous because a lot of public expenditure is at stake there.

[Voland's right hand]

Re: Cargo Cult Science

To prove the results that applied stats uses needs very advanced maths,

Yes

To prove the results that applied stats uses needs very advanced maths, which only a mathematical specialist will master

No.

Based on my personal suffering from the same subject (I am not a math graduate either), I can conclude that same as the majority of non-math graduates you have suffered from Idioticus Statisticus. I am quoting the species name by translating a good friend of mine who used to run the prob and stats dept in a major Eu university for several decades. That is how he referred to those of his colleagues which sucked so bad that they had to go teach non-math majors. That is curable.

Go read Feller, "An Introduction to Probability Theory and Its Applications". Amazon finally has it paperback so you do not have to shell out the relatively obscene 100 or so quid for the harcover (I have both the original and the English translation). Once you have read it, you are likely to change your opinion about "only advanced can grok it". It still holds a premium spot on my bookshelf till this day (many years after I did anything in proper math or sciences). Most of it reads like a novel by the way, you can enjoy reading it (that says a lot about a math book). I am probably going to add the English version of Gantmacher next to the original to it at some point so I can tell the kids to sod off and go read it when they ask math questions.

[Pompous Git]

Re: Cargo Cult Science

Go read Feller, "An Introduction to Probability Theory and Its Applications". Amazon finally has it paperback so you do not have to shell out the relatively obscene 100 or so quid for the harcover

A lot less than 100 quid from Abebooks. Thanks for the pointer :-)

[Anonymous Coward]

Re: Cargo Cult Science

Go read Feller, "An Introduction to Probability Theory and Its Applications". Amazon finally has it paperback so you do not have to shell out the relatively obscene 100 or so quid for the harcover (I have both the original and the English translation).

It's also on archive.org for free:

https://archive.org/details/AnIntroductionToProbabilityTheoryAndItsApplicationsVolume1

[Pompous Git]

Re: Cargo Cult Science

It's also on archive.org for free:

An Introduction To Probability Theory And Its Applications Volume 1

Yer blood's worth bottlin' as we say in these parts :-)

[LionelB]

Re: Cargo Cult Science

Well said. Statistics is difficult and the maths is hard and subtle. But the hardest part of applied statistics is figuring out which tests to use, whether your data satisfies the underlying assumptions behind the tests, how to apply them correctly and how to interpret what the results are (and are not) telling you.

[Doctor Syntax]

Re: Cargo Cult Science

"They underlying root cause is that statistics is not taught to biology/med science majors properly around the world."

In my day, if we had a statistical problem we went and asked a statistician. I recall one such visit where a very simple test gave a 1 in N chance of a random pattern match. When we left he was still shaking his head and muttering, about N "That's a very big number".

In this case it's a matter of not so much understanding the statistics but the code of a function. It's not clear from the report whether or not the function was open source or, even if it was, whether users would necessarily know it was incorporated into proprietary software.

[PNGuinn]

Re: Cargo Cult Science

"I doubt very few of the researchers would or should have a clue about validating third party proprietary software."

If you don't have a clear audit trail of your methodology, measurements and analysis it's not science - it's bad magic at best. Go get a real job annoying chickens.

If that sounds a little harsh, think of the damage false or fake results cause, especially when the the wider world makes vital decisions based on them.

If that proprietary software is not FULLY open source and 3rd party audited (note that I didn't say free and open source) It MUST be ASSUMED TO BE DODGY and no results based upon it can be taken as anything more than "interesting".

The whole point of science is to be able to question EVERYTHING.

[DavCrav]

Re: Cargo Cult Science

"The whole point of science is to be able to question EVERYTHING."

Except you don't. You assume that the scales and thermometers are accurate. You are heading towards Descartes's evil demon with your thinking. It's 'standing on the shoulders of giants', not 'standing behind them constantly checking their shoes'.

Science works by construction and destruction. You have to assume that most people are doing their jobs properly, because you cannot question absolutely everything. If I had to check every piece of research before using it I wouldn't be able to do any research. Sometimes errors are discovered, generally because new research contradicts old research or experiments don't match theory, and we have a big bonfire and throw a load of research on it. But I claim that this is by far the most effective method, rather than everyone being paralyzed through the requirement to replicate every single experiment.

[Pompous Git]

Re: Cargo Cult Science

"The whole point of science is to be able to question EVERYTHING."

Except you don't. You assume that the scales and thermometers are accurate. You are heading towards Descartes's evil demon with your thinking. It's 'standing on the shoulders of giants', not 'standing behind them constantly checking their shoes'.

I think you missed the word ABLE.

You can assume that "thermometers are accurate" all you want, but that doesn't mean you are correct. Different sensors may well agree that water freezes at 0 C and boils at 100 C at a pressure of 1 atmosphere, but will disagree on air temperature when placed in a screen.

The energy balance of small temperature sensors was modelled to illustrate the

effects of sensor characteristics, particularly size, on the accuracy of readings in the presence of

strong shortwave or longwave radiant loads. For all but extremely small sensors, radiant

exchange may lead to unacceptable errors. The common practice of using passively ventilated

instrument screens was evaluated in a series of comparative measurements. The differences

resulting from the use of different models of shields may be an order of magnitude greater than

the error resulting from sensor calibration. In the absence of technological innovation capable

of reducing the error due to radiant exchange to negligible proportions, it is suggested that a

standard methodology for calibrating and labelling the error resulting from the characteristics

of the screens be adopted, to allow comparison of new data with long-established records.

http://www.fau.usp.br/aut5823/Medicoes/Erell_Leal_Maldonado_2005_Abrigos_Radiacao.pdf

assume = to make an 'ass' out of 'u' and 'me'.

[Alan Brown]

Re: Cargo Cult Science

"Except you don't. You assume that the scales and thermometers are accurate. "

I don't. I was brought up with the notion that you CHECK their calibration against known standards, which are in turn crosschecked against other standards and you periodically RECHECK the devices to ensure they have not gone wonky. You also need to know their limitations - thermometers being one example where they may be accurate in liquids but not in air.

I was also taught the difference between RESOLUTION and ACCURACY.

What that means is that an uncalibrated device can often tell you that there is a difference between two measurements to a high degree of resolution, but you can't tell what the absolute values really are. both are useful traits. A calibrated device which has low resolution might give an accurate reading for measurements, but the higher resolution uncalibrated device allows you to tell the differences between the readings (for this reason, a digital multimeter might give 3-4 digit resolution but the reading should never be considered accurate to more than 1 significant place even after calibration - nonetheless, it's still useful to have those extra digits when tuning for peaks or minimums.)

[Adrian Midgley 1]

Assuming scales???? No

When I was taught to weigh things I was also taught to check the scales - balance beams with brass pans - by swapping the sides round.

[Anonymous Coward]

Non open source "must be assumed to be dodgy"

I sure hope you don't mean to imply that open source software can be assumed to be not dodgy. That didn't work out too well for people relying on e.g. openssl, which in the past year or two was found to have some bugs that were just as old as the bugs in this research software - despite being used by FAR more people and being FAR more critical to get right. Yeah, openssl code was rather messy, but research software is generally the ugliest code you've ever seen.

There are only two ways to validate the output of research software like this. One, have a second version of it developed completely independently to act as a check - but who wants to "waste resources" on writing and maintaining it? Two, have someone occasionally work out the results "by hand" to check. I realize some may object it say "what if it needs a million calculations, it would take someone a lifetime to do that" but you can still do it with computers, you just need to break it down into its components and do the calculations separately piece by piece without referring to the standard research software at all.

Even these aren't foolproof - what if the method is based on a paper someone wrote 20 years ago and it turns out there was an error in the paper that no one ever caught? In such a case the software would be "right" in terms of following the formula exactly, and separately developed software or calculations performed manually would show a matching - but wrong - result. Science is hard.

[Adrian Midgley 1]

Not automatically reliable, but

very difficult to prevent anyone from discovering the faults.

Proprietary software often seems designed to cover up faults.

[Arthur the cat]

Re: Cargo Cult Science

Can also be anthropological studies of millennial belief systems in post-WW2 Melanesia.

[oldtaku]

I'd have hoped most people twigged to this when that 2010 fMRI study found statistically significant activation in the brain of a dead salmon.

[Anonymous Coward]

Dead Salmon

"brain of a dead salmon"

Doesn't count - that was the control group.

[Mark 85]

Re: Dead Salmon

I guess they couldn't get a politician for the control group then?

[mosw]

Re: Dead Salmon

"I guess they couldn't get a politician for the control group then?"

Actually it was a politician. It was just the salmon coloured comb-over that fooled them.

[David 132]

I'd have hoped most people twigged to this when that 2010 fMRI study found statistically significant activation in the brain of a dead salmon.

Now that you mention it, I did think it was a bit fishy...

[Anonymous Coward]

Might have been a red herring.

[Code For Broke]

The whole series of experiments were nothing more than a fishing expedition.

[Captain Badmouth]

They were just trawling for data, obviously, but the netting didn't quite balance things.

[Petrea Mitchell]

Salmon's off

My first thought as well. (Though the study was actually in 2009.)

http://blogs.discovermagazine.com/neuroskeptic/2009/09/16/fmri-gets-slap-in-the-face-with-a-dead-fish/

[allthecoolshortnamesweretaken]

I really like the "This is what your brain looks like on bad data" line.

Question: any chance the raw data is still stored somewhere and can be processed again using revised software? (I guess not, but I've never worked with MRI machines.)

BTW, why the many downvotes for the cargo cult reference? It may be a bit harsh in this context, but the basic point is valid. If you use something in your work that is a "black box" to you, you take a risk.

[You aint sin me, roit]

If you use something in your work that is a "black box" to you, you take a risk

I imagine the whole MRI scanner is a black box to most biologists.

Even if you wave your hands and say "nuclear magnetic resonance" you're using the black box of quantum mechanics - so do the biological researchers now need to be physicists? And those physicists... few [i]fully[/i] understand the rigorous derivation of all of the black box mathematical methods they use. So do the biologists now need to be mathematicians too?

In the case of this research, would the researchers be expected to have reviewed the software? Would the software developers be expected to review the compiler, the operating system, the computer hardware, the software used to design chips?

However, it is a bit worrying that the software was faulty... I wonder how many other researchers, heavily dependent on third party analysis packages, are quaking in their boots.

[EddieD]

Re: If you use something in your work that is a "black box" to you, you take a risk

"I imagine the whole MRI scanner is a black box to most biologists."

I don't think so - even back in the 80s when I was studying towards a Biology based degree, and NMR was a fairly new phenomenon, we were taught very carefully the underlying principles of both NMR and ESR spectroscopy. I don't think that that will have changed.

And the University insisted that all science subjects had Maths and Stats courses.

[a_yank_lurker]

Re: If you use something in your work that is a "black box" to you, you take a risk

You were taught enough to have a fair understanding of how it worked commensurate with the amount of math and quantum mechanics you studied prior. This is true of all areas of science and engineering, you learn your field reasonably deeply but you are relatively weak in related fields.

The problem is when one must rely to on "black boxes" that they did not design, build, and program. One will always be at risk either misusing the black box or, as in the case, the black box has a serious flaw that corrupts the data and invalidates the experiments. I seriously doubt any of these researchers could design, build, and program an MRI machine.

[Alan Brown]

Re: If you use something in your work that is a "black box" to you, you take a risk

"However, it is a bit worrying that the software was faulty..."

Software is ALWAYS faulty.

One shining example being the way that floating point calculations are handled. With the best will in the world there are always rounding errors introduced and the way you write your software can actively seek to minimise them, or possibly ignore them and compound the errors.

We ran into real-world examples of this with IDL programs giving different answers in 32 and 64 bit environments. The assumption was that as the 32-bit software had been around for years, the 64bit environment was wrong.

It turned out that BOTH were wrong, just in differing degrees. The resulting insights meant that more accurate (and consistent) results were achieved in both environments(*) AND the old raw data got reprocessed to give "better" answers.

(*) Hint: Don't take the answer from a single calculation and use that as input for the next calculation, ad nauseum. This compounds the rounding errors.(**) Always recalculate from your basepoint.

(**) It's akin to using pi=22/7 or 3.142 and then iterating over several million subsidiary results. Whilst both are a useful approximation, over a few million cycles you won't get circles anymore. ALWAYS do all the integer stuff _first_ when calculating on a computer. Real numbers need to go into the calculation mix last.

[Destroy All Monsters]

Only 20% of a software development team's brains are used at any moment!

What would happen if they used 100% (as Einstein famously not said). Click "HERE" (site related to Scientology) to find out!!!

[Anonymous Coward]

Any decent scientist checks the tools for accuracy

Just because the tool is software does not abdicate them of that responsibility.

[Anonymous Coward]

Re: Any decent scientist checks the tools for accuracy

Just because the tool is software does not abdicate them of that responsibility.

This is not 1 + 1 = 3. This is millions of lines of software carrying out complex calculations, and widely used by a large community. It's like saying that any decent driver tests the software in their car for bugs.

[Doctor Syntax]

Re: Any decent scientist checks the tools for accuracy

To a very limited extent I agree with you. I used to be a Tutor Counsellor back in the early days of the OU. The kit sent out to foundation science students included a simple beam balance. Two of my students were teachers and they objected to this because they had nice digital balances at school - the lure of the numerical readout. I pointed out that with a beam balance they at least could see what their standards were. When my lab acquired its first digital balance (we'd lost our original kit in a fire) I took advantage of the fact that the local Weights and Measures folk were next door and went to borrow a standard mass from them to check it.

Apparatus that you put together yourself (which you might well do if you're in a field where COTS kit isn't available) can and should be carefully tested and calibrated.

But given the complexity and quantity of commercial stuff in any lab these days it's not feasible and the manufacturer's reputation has to be relied on.

[DavCrav]

Re: Any decent scientist checks the tools for accuracy

How? This isn't a slide rule. This is a piece of software that interprets a couple of terabytes of information into pictures. What do you test it with, and against?

[Anonymous Coward]

Re: Any decent scientist checks the tools for accuracy

Trust but verify.

Science based on unvalidated software is not science.

[John Smith 19]

"“lamentable archiving and data-sharing practices”"

Yes that did put me in mind of The Climate "Research" Unit and the harryreadme file.

I've heard people comment that functional MRI has been used by drinks companies to identify brain activation and response to various drinks. Yes that really is your brain on Coke.

That research has been very quietly done and is obviously worth billions in a global market.

"Know your tools" is a good moto but of course with tools this complex there is a reason for "we have to trust the developers knew what they were doing."

This looks like the situation in CFD,but without the approach. In that field no software (or new major release) gets accepted without multiple test runs amongst known test cases (many from live wind tunnel tests). First they test, then they trust.

BTW doing it with dead samples brought up the interesting factoid that about 500 genes were firing after death, presumably as part of the organisms last ditch attempts to self heal.

[Vic]

Re: "“lamentable archiving and data-sharing practices”"

with tools this complex there is a reason for "we have to trust the developers knew what they were doing."

And yet - most of us here have worked in complex development environments, and we know the shite that goes out the door on ship day...

In that field no software (or new major release) gets accepted without multiple test runs amongst known test cases

The trouble with that sort of test-driven development is that you can easily end up in the situation where you're developing a product to pass those tests and do essentially nothing else; if the tests are not renewed frequently, your solution ends up touching the problem surface at a few discrete points only.

Vic.

[Alan Brown]

Re: "“lamentable archiving and data-sharing practices”"

"The trouble with that sort of test-driven development is that you can easily end up in the situation where you're developing a product to pass those tests and do essentially nothing else;"

I've seen this too. Products getting fudge factors inserted by the programmers so that they passed the tests (what's happening in the real world) vs notifying the project initiator so they can go back to find what the root cause of the discrepency was and incorporating this into the original hypothesis.

People can and have been sacked for this kind of shenanigan and it's the kind of attitude from staff that results in things like accusations of deliberate climate model fudging for political ends, etc.

The fundamental problem is that the slacker attitude that you see in many movies is alive and well even in well-funded science environments (and thriving in badly funded ones). Noone wants to admit that they can't trust _all_ their staff to produce the right results (worse still you don't know which staff are unreliable until long after the horse has left the barn), in the same way that noone really wants to admit the biggest threat to any business is from within.

[Steve Crook]

Wee 'P' values

There's an idea that a small 'P' value always means something, and also, an idea that statistics isn't something that needs a specialist, that your graduate course in stats and Excel functions are enough to see you through...

I can't help feeling that every science team needs a statistician or three to analyse their data and produce results. Sort of like system/acceptance testing, members of the team, but kept apart so they can at least make a show of independence.

As for publishing code and data to allow reproducibility, why would anyone want to do that unless they're forced to? IMO for peer review to be worth anything, every published paper should have had its code and data analysed by at least one independent reviewer and should be published to allow anyone to reproduce the results.

Remember a paper published last year that claimed to show that people who held right wing views correlated with people with psychotic tendencies? Someone got hold of the data and showed that a bug in the researchers Excel spreadsheets meant they'd got the 'results' the wrong way round.

http://retractionwatch.com/2016/06/07/conservative-political-beliefs-not-linked-to-psychotic-traits/

If forcing reproducibility during review means that guff like this never gets published, job done.

[Alan Brown]

Re: Wee 'P' values

"Someone got hold of the data and showed that a bug in the researchers Excel spreadsheets "

I think you mean a bug in the researchers. Using Excel for detailed analysis should be a sacking offence.

[Doctor Syntax]

Who'd 'a'thunk it?

Clustering software produces artefacts.

[demat]

Where's the problem?

Provided the proper controls/baseline scans were in place then any systematic errors in the analyses will be accounted for.

[Uffish]

Re: Where's the problem?

I agree - but what is a 'proper' baseline scan (or set of scans)? Totally MRI ignorant nerds would like to know.

[demat]

Re: Where's the problem?

The design of appropriate controls will completely depend on what the experiment is. The point is that any systematic errors will be present in both datasets.

[cantankerous swineherd]

loads of neuro"science" goes down the drain, and a good thing to.

side note: statistics is inherently hand wavy, involving approximations and estimates. add a lot of prerequisites for the theorems you're using but don't understand and any statistical argument is prima facie dodgy.

[DavCrav]

"statistics is inherently hand wavy"

You are wrong. Statistics is not inherently hand wavy. People who don't understand statistics think it's hand wavy.

"involving approximations and estimates. add a lot of prerequisites for the theorems you're using but don't understand and any statistical argument is prima facie dodgy."

I can tell you are not a mathematician or statistician by that argument, which boils down to: "I don't understand stats. Therefore nobody does".

[EddieD]

"You are wrong. Statistics is not inherently hand wavy. People who don't understand statistics think it's hand wavy."

Oh I dunno. Having something called "Tukey's honestly significant difference test" implies that there are dishonestly significant difference tests out there somewhere...

[DavCrav]

"Oh I dunno. Having something called "Tukey's honestly significant difference test" implies that there are dishonestly significant difference tests out there somewhere..."

Oh, I didn't say you couldn't lie through your teeth with statistics. But if you are doing post-hoc analysis of data, it's so much easier to be honest about it. So much easier to pretend you were looking for that all along.

[LionelB]

loads of neuro"science" goes down the drain, and a good thing to (sic).

How on earth is that a "good thing"?

side note: statistics is inherently hand wavy, involving approximations and estimates. add a lot of prerequisites for the theorems you're using but don't understand and any statistical argument is prima facie dodgy.

You clearly have absolutely no idea about the theory or practice of statistics. A statistical test does exactly what it says on the box - no more, no less. Approximations are usually delimited and their consequences frequently quantifiable. That is not "hand-wavy". Correct application and interpretation of results is difficult and subtle - but essential; without statistics there is no science.

[Pompous Git]

without statistics there is no science.

Given that statistics didn't really take off until the 19th C I take it you believe there was no science before then. Many historians of science wouldn't believe you. Aristotle for example was doing some very interesting stuff in marine biology two and a half thousand years ago. Not to mention formalising the logic that allows our computers to do their stuff.

[Anonymous Coward]

If these are 3 separate packages that have problems, what are the chances that part of each one's validation process involves comparing results with one or both of the others ?

[a_yank_lurker]

"If these are 3 separate packages that have problems, what are the chances that part of each one's validation process involves comparing results with one or both of the others ?" - If they produced similar results it is highly likely everyone would assume all was good. One would assume that the three packages would produce the same errors at similar rates.

[Anonymous Coward]

I have worked in my career on several simulation software tools. In certain cases we tried to compete with a market leader.

In that case, we are of course always benchmarked against the market leader. And occasionally we found deviations, and we discovered the market leader had bugs. Sometimes pretty basic ones, which seemed to suggest that not only did the market leader not do sufficient testing, but also the customers accepted apparently anything at face value.

In such situations, there has occasionally been commercial pressure to be "bug-compatible" with the market leader. Technical explanations go over the head of the managers making buying decisions. And the big tool is trusted by default.

As such, I wouldn't put too much stock in using, say, three tools to analyse the same data.

Also, if a tool is market leader, it may not mean much about its quality. Chances are the developers stopped bug-hunting years ago, after all nobody likes bad news. ("We found a bug and actually all these simulation results you trusted for years were crap".)

Anonymous to protect the guilty. Trust your slide rule.

[Pompous Git]

the big tool is trusted by default.

That's seriously funny!

[Anonymous Coward]

Trust your slide rule.

The nice thing about slide rules: You have to know where to put the decimal point, so you've generally engaged your brain and have an answer somewhere in the ballpark.

Whereas I see people take that the result they see on a computer screen as gospel, no matter how ludicrous that result might be.

[Pompous Git]

Re: Trust your slide rule.

The Git whips out his Accu-Math 400 and says: "By golly; he's absolutely correct". :-)

[c4m1k4z3]

As I understand it, first the system would run the experiment and acquire the raw data files. That data is then processed by some means.

So the question is does this affect the raw data? If not, they can just reprocess with the fixed software...

[Pompous Git]

So the question is does this affect the raw data? If not, they can just reprocess with the fixed software...

You are assuming here that the original data still exists. Sometimes it "goes missing" as in the case with the Climatic Research Unit at East Anglia U. This is also known as "the dog ate my homework" excuse. Paleoclimatologist Lonnie Thompson (ice cores) has never archived his data to the best of my knowledge. IOW don't hold your breath. Sometimes you can get lucky and find data that "no longer exists" on an anonymous FTP server. Scientists aren't necessarily computer literate or security conscious.

[Anonymous Coward]

"Paleoclimatologist Lonnie Thompson (ice cores) has never archived his data to the best of my knowledge. IOW don't hold your breath. Sometimes you can get lucky and find data that "no longer exists" on an anonymous FTP server. "

I have spent most of the last 16 years arguing the case about backup of data with people who should know better but refuse to pay for the backup equipment.

In almost every single case when I've been overruled they've eventually ended up trashing their datasets and come to me demanding that XYZ disk be restored. Sometimes it's easy (undelete) and other times you have to send stuff out for disk recovery at £2k a pop. Those same groups who refuse to pay for backups also refuse to pay the necessaries for recovery of their "critical data" and frequently bludgeon management into submission with "this failure will affect the reputation of the university" - when they're singlehandedly responsible for the mess they've created AND are repeat offenders.

They're also the same kind of people who send an email at 4am wanting a VPN setup at 6am for a conference they're attending 8 timezones away, when they knew they needed it for weeks beforehand and then send blistering missives to management at 8am because it didn't happen - never mind that noone able to setup the VPN is in until 9.30 and doing so would be a disciplinary offence in any case.

Anon, because those offenders are still around and I want to keep my job.

[Adrian Midgley 1]

Reference please...

What data went missing?

My recollection was that the complaints were around the programmer-like language in some of the rather convoluted recipes used to process the data, not that data was missing.

(I'd be a little surprised if all data on measurements forever had been retained - the previous Canadian government made a large effort to destroy quite a lot of maritime data, did it not?)

[Pompous Git]

Re: Reference please...

What data went missing?

The data that the IPCC relies on for its Climate Assessment Reports.

We are not in a position to supply data for a particular country not covered by the example agreements referred to earlier, as we have never had sufficient resources to keep track of the exact source of each individual monthly value. Since the 1980s, we have merged the data we have received into existing series or begun new ones, so it is impossible to say if all stations within a particular country or if all of an individual record should be freely available. Data storage availability in the 1980s meant that we were not able to keep the multiple sources for some sites, only the station series after adjustment for homogeneity issues. We, therefore, do not hold the original raw data but only the value-added (i.e. quality controlled and homogenized) data.

Source:

https://crudata.uea.ac.uk/cru/data/availability/

[DavCrav]

"So the question is does this affect the raw data? If not, they can just reprocess with the fixed software..."

The raw data in fMRI is massive though, I think a friend of mine in chronic pain study told me it was about a terabyte or two per scan. Nowadays that's possible to store, although remember you have lots of these per study, but ten years ago that was a serious amount of data.

[Alan Brown]

"I think a friend of mine in chronic pain study told me it was about a terabyte or two per scan. "

Raw data yes. It's highly compressible though (even with lossless methods)

[Anonymous Coward]

Re: You assume that the scales and thermometers are accurate.

" You assume that the scales and thermometers are accurate. "

You might. But that makes you ill informed, a fool, a bad engineer/scientist, or some combination thereof.

Sensible organisations which are taking measurements that matter will take them using tools that have a regular and auditable calibration programme traceable (directly or indirectly) to national and international approved standards. Uncalibrated (or out of calibration programme) instruments and sensors are not appropriate for definitive use. That applies whether you're measuring length, mass, current, voltage, temperature, or any other sensibly measurable quantity.

Readers might wish to look up the history of the subject of "metrology". It goes back a long way, but the SI system of weights measures etc brought it increased prominence in most parts of the world (with one obvious exception), though in fairness it wasn't always called metrology.

For further light entertainment, readers might also want to look up the history of incidents with the Therac-25 to see what can happen when naive people combine hardware, software, and a lack of understanding of the importance of proper closed loop calibration. People died. See e.g.

http://sunnyday.mit.edu/papers/therac.pdf (full version, 49 pages)

Then think about your self driving cars, and why their requirements are different from VCRs.