Sci/tech MPs want peer review, not pal review

The House of Commons Science and Technology Select Committee has called for greater integrity and data disclosure in peer-reviewed literature. It recommends that all UK research institutions should have "a specific member of staff leading on research integrity". MPs are also concerned that research quangos should be wary of …

[Anonymous Coward]

Not the greatest threat

The greatest threat to research integrity in the UK isn't peer review complacency, it's a meddling government hacking at science funding and changing research priorities to namby pamby stuff like the "digital economy", and removing such unpopular areas as "chemistry" and "mathematics".

And the MP questioning the peer review process got a degree in industrial relations and then worked as a technician in a geology lab for ten years. He's the chairman of the Science and Technology select committee. One of my friends appeared before that and told me the whole lot of them are useless and don't understand anything about the UK science landscape.

[LPF]

@Anonymous Coward

Nice go at deflection and not an ounce of baised in your post huh? :P

Yeah they dont understand that its a case of you scratch my back and I'll scratch yours!, how day they point out that getting your mates to review your work and to only particially release data and hiding behind FOI requests exemptions should be considered unethical.. damned plebs!

[Anonymous Coward]

Not a UK thing

The peer review process is international, has nothing to do with the UK government. It is conducted by independent companies, not under the jurisdiction of the UK government, not regulated by the UK government, so they can feck right off.

It's not deflection to say that the EPSRC reforms are completely batshit insane, and will do incalculable damage to UK science. See http://gowers.wordpress.com/2011/07/26/a-message-from-our-sponsors/ for Tim Gowers's blog. He is a pre-eminent mathematician, Fields medallist, and his opinion is worth more than the entire S&T committee's.

As for peer review: if you have any better ideas, I'd love to hear them. I've done peer reviewing for top journals and it's not foolproof, particularly with computer simulations and calculations, where it's impractial to make the data available (think tens of gigabytes per calculation: who is going to pay for putting it all up online?). If a better system exists, I would move to it in a heartbeat. But governmental fiddling has never helped in the past, and who do you trust more: scientists or politicians?

[Loyal Commenter]

Sorry LPF

But I have to agree with the AC here. This country used to be a world leader in science and technology. Think of the vast number of things that were invented here. However, investment in science and technology is a long term investment, and the political class rarely see beyond the four year re-election horizon. They aren't interested in the potential benefits if they fall beyond that timeline. The solution really is for funding to be properly decoupled from the influence of MPs, and handled, as it should be, by the research councils without interference from meddling politicians who are only after thier next vote.

As for peer review; yes, it is generally acknowledged that this isn't a perfect system, in much the same way that our voting system isn't perfect. I would ask you to suggest a better alternative to either that is generally acceptable. I remember reading somewhere an article about why a perfectly representative voting system isn't possible, the peer review system is good enough, has stood teh test of time, and is one of several checks and balances that keeps scientific research honest. Other checks include the need for reproducibility, ethics committees, and the unwillingness of funding bodies to give money to researchers who can't show that their work is serious and novel.

Having said that, the researches in the so-called 'climategate scandal' may have been guilty of being somewhat circumspect about releasing their data. You have to remember, however, that being human, they are not perfect, just like you and I, and most people wouldn't want to spend a significant portion of their working time pandering to nutjobs, adn lobbyists who are trying to undermine your work by grabbing your research data and selectively quoting the outliers.

The problem really isn't that science is dishonest; on the whole it is very much the opposite, but that human beings are dishonest, particulalrly those with vested interests. In this case, I'd put money on the MORE dishonest parties being the AGW deniers funded by large oil producers and their lobbyists.

[Sigfried]

Keeping to the Party Line...

Oh yeah, those naughty skeptics all getting tonnes of money from the oil producers ! Wow, batshit insane anyone ? There are literally billions of $$$ being provided for "climate research" which are only available to those who will report on new and very scarey (of course) scenarios based on cherry picked data which they refuse to show anyone else "in case they find something wrong with it".

Just think re the "Climategate" case, the emails set out "scientists" clearly conspiring to pervert the peer review system as well as other extremely dodgy behaviour. They also tried to avoid FOI requests by claiming confidentiality agreements, but when they received further FOI's for those agreements they couldn't show any worth noting. Dishonest much ?

You need to actually read an unbiased account of what was happening instead of simply repeating the tired and blatantly untrue canards that the PR spin-meisters employed by the climate establishment (who just happen to be the same people involved with the NOTW scandal, surprise surprise) put out for useful idiots to employ.

[zooooooom]

most journals

are international - not subject to uk whims

[Anonymous Coward]

Peer review = friend review. Not where I publish...

The general consensus amongst scientists I mix with is that science is so specialised that the only people who can properly review your work are your direct competitors. Which generally means a rough ride in peer review - and in a rapidly moving field like computer science sometimes your ideas appearing somewhere else before you get them published.

Of course peer review is blind, but since you know who your competitors are, you have pretty strong suspicions from the style of prose.

While a personal tragedy for some researchers, none of this reduces the quality of the science itself - you need a clear advance in thinking or a really important result to convince your competitors to avoid being on the receiving end of an 'Oxford Sandwich' - no commital praise followed by a single inciteful but possibly trivial observation calling into question your whole paper, followed by more faint praise and a weak reject.

Another problem is that doing a proper review costs a huge amount of time for which you don't get much credit. When you are up against a deliverable deadline (yes, we have them in academia) then guess what suffers?

Of course, for short periods of time peer review can be gamed. But the truth outs before too long http://en.wikipedia.org/wiki/Sokal_affair. The problem with climate science is the normal haphazard way science progresses is simply not fit when you want direct science-driven policy. For all of the faults, academic led clinical trials show how this should work so it is possible to get it right.

So peer review remains least worse alternative to quality control in science.

[Sam Liddicott]

Making data available

To carry on from Aurthur C. Clarkes comment about advanced technology being indistinguishable from magic, those so-called scientists who don't make their data available are indistinguishable from witch-doctors or wizards.

[Anonymous Coward]

yeah

in the same way that programmers who don't make their source code available are indistinguishable from witch-doctors or wizards.

In any case the universe rather stubbornly refuses to go away - so you are all free to do your own experiments, take your own measurements, assemble your own data, y'know. And if you don't, you can't even claim the exalted title of witchdoctor.

[breakfast]

A problem of scale

Peer review was designed for a time when there were few enough papers published that other researchers could look at a paper and actually reproduce the results for themselves. With so much research and so much data available now, how often does this actually happen and what are the incentives for it to be done? Especially when most research scientists seem to be judged by the papers they have published rather than those they have rigorously reviewed.

[Anonymous Coward]

The UK does "do" science ...

it does "media"

[Anonymous Coward 101]

The UK does "do"...

...epic fail grammar.

[Ru]

The UK does "do"

Epic fail idiom.

[David Harper 1]

Scientists are human too

It's hardly surprising that there will be the occasional case of fraud or misconduct in the world of scientific research. Scientists are only human, and they are under enormous pressure to publish as many papers as possible by the funding agencies.

But we're in danger of castigating the entire scientific community because of a handful of well-publicised cases of bad behaviour. It's like saying that all journalists are lying scum, just because of a few bad apples at News International.

We should instead be celebrating the fact that the vast majority of publicly-funded scientists would never even think of committing fraud or misconduct.

We should also bear in mind that the few individuals who did commit fraud were found out by their fellow scientists, and were swiftly thrown out of the scientific community.

[JimC]

> all journalists are/bad apples

You mean you're seriously advancing the proposition that phone hacking or other immoral behaviour is limited to News International, and not found anywhere else in the press, and say, other tabloids or programmes like Panorama are completely immune from it?

I admire your optimistic view of journalistic integrity, but I fear I don't share it...

[Tim Parker]

@JimC

"You mean you're seriously advancing the proposition that phone hacking or other immoral behaviour is limited to News International"

..perhaps not the best analogy, but I didn't read it that way myself. David said it was like extrapolating to *all* journalists from some at News International but that doesn't preclude other journalists in other areas of the press - I think it's pretty much a given - just not every last one of them.

[Tom 13]

The problem is the occasional case,

it's the increasing frequency and the sheer audacity of those committing the fraud. Particularly unbecoming is their Eddie Murphy imitation whenever they are called on it. I'm quite certain it cost Copernicus more as a percentage of the money available to him to make his data available to Kepler than the corresponding numbers for CMU to retain the raw data for their claims.

[Sigfried]

Wrong !

Wrong ! Most of the climate scientists guilty of unprofessional, misleading, and even scientifically fraudulent behaviour are not "thrown out", but indeed are promoted to become leading authorities and lead authors in IPCC reports.

[Anonymous Coward]

I'm confused as to why they think it's expensive to make their data available.

Storage is dirt cheap (3TB for £70 these days!) and all universities and most research establishments have massive bandwidth.

Just dump it into a semi-public read-only FTP server, job done. Nobody's asking for it to be beautifully catalogued.

There's no excuse not to - it costs you next to nothing to keep the data nearly forever, and you're doing yourself a favour anyway because you probably will want that raw data again in a few years time for some related research.

[Spoobistle]

data availability

Maybe the data isn't yours to make available: take look at

http://www.bbc.co.uk/news/science-environment-14315747

- the Trinidad & Tobago govt evidently made a little pin-money out of charging for their climate data. So you could find yourself caught between a FoI commissioner order to release and a Justice Arnold order blocking your ftp server...

[Blofeld's Cat]

@Spoobistle

You make a valid point, although the T&T government would presumably sell their data to whoever asked for it.

I suspect that in many cases it is the *method* used to process the raw data that requires the peer review.

There's an old saying that If you have enough data you can prove anything with the right selection criteria - and judicious use of "Skinner's constant".

[Sigfried]

Trinidad data has been released before...

Get serious about it though... the T&T data was released publically many years ago, so it's kinda late to be claiming some sort of confidentiality. Also, all of the data was released to a selected academic (someone who they could be sure wouldn't rock the boat) shortly before it was refused to another academic (who they thought would "rtb").

Honesty and consistency are virtues apparently entirely unknown in some quarters. But money to pay for PR spin doctors, there's plenty of that !

[Anonymous Coward]

@AC

Data storage is not cheap, those £70 3TB disks are not designed to be run in servers, they have a low duty cycle, slow bus and spindle speed and just can't hack the workload of even spinning 24/7. You need better and therefore more expensive disks. Then you need a disk array to host them, a server to serve them and a SAN to connect them and a network to interconnect the servers. These all need to be supported, which costs, even if you just have hardware support it still costs, a lot. Once you've got the server infrastructure together you also need WAN bandwidth onto the Internet, sure some universities already have lots of bandwidth, but that it usually ear-marked, you don't buy up bandwidth you aren't going to use, so you need to buy more. Then you need to think of backup and DR, support staff, electricity and cooling, building maintenance costs, etc. etc.

The basic upshot is that it is very, very expensive indeed to host large datasets for all and sundry to access. This is not to say that it shouldn't be done, it just needs to be understood that the disk you see on the shelf in PC world doesn't even touch the tip of the iceberg of data hosting costs.

The big question is: Who funds the data, should the research project fund the data, if so, what happens when the project finishes? What happens to the cost of projects when data publishing is factored in, do we do less science and make it more open, or more science and accept that it will be harder to get hold of data? If the project doesn't fund the data being hosted, who does? Should it be the probably already cash-strapped university?

[Steve Gooberman-Hill]

Peer Review and Grant Application Review

It's not just Peer Review of papers - the same thing happens with Grant applications. Many grant applications involve peer review of the proposals. If your friends are doing the peer review, then it is very easy.

And it's not just science. It is all forms of academic research, including social sciences and humanities, which suffer from these forms of unethical and nepotistic behaviours

[Anonymous Coward]

just because they're your friends

doesn't stop them giving an accurate assessment.

[Tom Kelsall]

Title? What title?

...and I always thought science went "hypothesis, observation, interpretation, <repeat with controls as required>, publish". Why on earth would a set of scientific research based on this paradigm need peer reviewing before publication?

Once published, it will either get repeated by others (and therefore corroborated) or raped and vilified into obscurity. Yes, you'll get a lot of shit coming out this way, but the shit will soon been seen as such when it's not repeatable or when the interpretation doesn't stand up to additional tests. The "good science" will rise to the top. This approach would also see a more cautious media; reporting only those papers which have been independantly verified.

No - the reason for the current system is commercial funding - commercial organisations won't allow publication of research they have funded until it's independantly verified... and that's a shit way to operate with so many conflicts of interest as to make it almost unworkable as well as OPEN TO ABUSE.

[Tim Parker]

@Tom Kelsall

"No - the reason for the current system is commercial funding"

No - it's not.

[Some Beggar]

Change the record.

This report has nothing at all to do with the debunked accusations of dishonesty at UEA. It came out of the very real fraud committed by Andrew Wakefield. Why are you still wittering on about climategate?

[JimC]

> debunked

Have they been debunked? All the reports I've read seemed to say that the alleged misbehaviour was outside the scope of the report so they weren't going to investigate it.

[Some Beggar]

re: debunked

"we saw no evidence of any deliberate scientific malpractice in any of the work of the Climatic Research Unit"

"careful examination of the e-mails and their full context shows that the petitioners' claims are exaggerated and are not a material or reliable basis to question the validity and credibility of the body of [climate] science"

Seems fairly unequivocal. To continue to bang on about it and to imply that this current report has any relationship to it is disingenuous to the point of being ridiculous.

[Feralmonkey]

WOW

The reason they "saw no evidence of any deliberate scientific malpractice in any of the work of the Climatic Research Unit" is because the people doing the examination were on the UEA payroll and had been a part of the CRU, in other words a whitewash from start to end.

[Some Beggar]

Lord Oxburgh was on the payroll of the University of East Anglia?

Hahaha. Priceless.

[Anonymous Coward]

@ Some Beggar

No, but UEA did pick him.

I know that newspapers aren't the most reliable sources, but look here

http://www.timesonline.co.uk/tol/news/environment/article7071751.ece

It would appear that a conflict of interests is/was present.

[ChilliKwok]

Lord Oxburgh: Biased? Moi?

Lord Oxburgh: President of the Carbon Capture & Storage Association, Chairman of Falck Renewables (windfarms) and Blue NG board member. Paid advisor to Climate Change Capital, the Low Carbon Initiative, Evo-Electric, Fujitsu, and Deutsche Bank carbon trading.

Yes - he sounds like an ideal candidate to lead a supposedly impartial, independent inquiry into the can of worms exposed by the CRU email leak. He had absolutely no financial stake in the outcome, did he?

He's got fingers in so many green pies he had to type the report with his toes. No suprise he only bothered to write a 5 page white-wash. What a joke. And yet the warmists on here attempt defend the indefensible. Tells you all you need to know: Watermelons !!

[Tim Parker]

@JimC (debunked)

"All the reports I've read seemed to say that the alleged misbehaviour was outside the scope of the report so they weren't going to investigate it."

Some Beggar was talking about "debunked accusations of dishonesty" - all the reports i've seen have dealt with the potential 'dishonesty' - moral and/or legal - and either

(a) thoroughly investigated it and exonerated the personnel

or

(b) rubber stamped everything their mates did

depending on your view-point. I thought the thing that wasn't investigated due to 'scope' issues was the science - again a good/bad thing depending on your preference... I may be wrong on that and welcome corrections.

[Anonymous Coward]

Theoratical research versus applied research, not really a new arguement.

"...it's a meddling government hacking at science funding and changing research priorities to namby pamby stuff like the "digital economy", and removing such unpopular areas as "chemistry" and "mathematics"."

Really means:

Government withdrawing funding from theoritical research which is failing to deliver on its promises, and putting the funding to applied research that has a tangible and measurable output inline with current government priorities.

Most academics think money is a bottomless pit and that their research should be funded above anyone elses. Mention measurement of success and deadlines and you get a confused and slightly scared look or outright acusations that you don't understand what they are trying to achieve.

That being said:

1. I dont believe most academics are liars, or set out to decieve anyone.

2. Peer-review does generally work well.

3. Media has a tendency of taking scientific research and trying to "simplify it" for the average viewer or reader. In the process often loosing much of the detail and accuracy.

[DavCrav]

Deadlines

"Mention measurement of success and deadlines and you get a confused and slightly scared look or outright acusations that you don't understand what they are trying to achieve."

Yes, academic doing research, you must have a groundbreaking idea by Friday lunchtime, we have a deadline.

[Alex 14]

Not so simple

"Government withdrawing funding from theoritical research which is failing to deliver on its promises, and putting the funding to applied research that has a tangible and measurable output inline with current government priorities."

If you thought any particular theoretical field was "promising" anything, you misunderstood. By definition, it's impossible for research to promise anything - you don't know whether theoretical research will even produce results that might potentially be useful until after you actually do the research. That's the whole point of doing research.

The idea that you can simplistically trade off theoretical research for applied research is dangerous because applied research builds on theory. If it weren't for theoretical research into electromagnetic radiation (lasers), we wouldn't have applied research into higher capacity optical storage media (DVDs etc). No-one could have predicted that all that theory would be as useful as it has been when Einstein set it out.

"Most academics think money is a bottomless pit and that their research should be funded above anyone elses. Mention measurement of success and deadlines and you get a confused and slightly scared look or outright acusations that you don't understand what they are trying to achieve."

I take it you've never had to apply for funding then. Academics are acutely aware of limited resources every time they have to compete with legions of their peers for research grants. Moreover, deadlines are an issue. Trust me. Money runs out.

[David Harper 1]

Public data release

In some areas of science, public data release is the norm.

When the Human Genome Project was launched in the early 1990s, the participants agreed to make all of their data publicly available. Organisations such as the Sanger Institute devote considerable effort into providing free public access to terabytes of genome sequence data through web sites such as ensembl.org

In astronomy, all of the major international observatories give visiting astronomers sole access to any observational data that they obtain for a very limited period, typically six months or a year. After that, it is automatically made public on the observatory's data archive web site, and anyone with an Internet connection can download it and use it however they want. That includes data from the Hubble Space Telescope, by the way.

But this does require a huge investment in storage hardware, software and network bandwidth, not to mention the personnel required to keep it all running. That places an impossible burden on small research groups, because they can hardly get adequate funding to do the science in the first place, let alone set up a long-term public archive.

It's only large, well-funded organisations such as the Sanger Institute or the Space Telescope Science Institute which can do this kind of thing, because they have sponsors (the Wellcome Trust and NASA, respectively) which support public access to data.

[Wommit]

This is the reason...

"The inquiry into peer review came after the release of the Climategate files, which revealed academics at the University of East Anglia selectively disclosing data needed to replicate their results, hiding from Freedom of Information Act requests, recommending destruction of email trails, and vowing to "redefine" the peer-review process to keep papers they disagreed with out of the publication system."

This and this alone should have been enough to have the scientific world screaming at the CRU.

Golbal warming is far too serious _NOT_ to publish full data, show the algorithms and assumptions used, and present conclusions.

The 'Hockey Stick' graph, would have been laughed out of existence _IF_ the full data and algorithms had been published.

So we now have 'scientists' who are ready to block access and pervert the usual quality processes, I cannot help but wonder WHY.

This time, _WE_ really do need to know.

[airbrush]

Research abuse

The media's search for a good story does tend to give research a bad name, often papers are published with small data sets which are statistically invalid but may attract more funding to do further research. Tellingly these are the papers that make the news, mmr vaccines and fish oil (snake oil salesmen) for example. This is an example of why we need laws to ensure newspapers tell the truth.

Drug companies are notorious for only publishing studies that show more positive results (for some reason they don't publish the studies showing the drug is worse than an existing cheaper drug). Peer review isn't the problem here they must publish every study for peer review to be possible.

Peer review where its open to political or commercial interference is basically corrupt. Peer review that stops a paper being published that goes against received wisdom may have left us believing the earth is flat. So how does this leave us better off? Knowing this government it will leave research more corruptable and lead to bad decisions affecting everyone.