Top doctors slam Google for not backing up incredible claims of super-human cancer-spotting AI Distinguished doctors have publicly criticized Google and others for making grand claims about AI research and then not sharing the source code and models to let others replicate and verify the experiments. In January, a team led by Google Brain's Scott Mayer McKinney published a paper in Nature boasting that its artificial …
It is more like not publishing the details concerning how the sequencing was done, how the organism was identified, how contamination was avoided, what reagents were used, etc. In other words, just publishing the results without the underlying data and methodology, which happens to be a very hot topic in biology and medicine currently. The whole Surgisphere fiasco provides a great example of what can happen when the underlying data (lying being the operative word) is missing in action.
That is the reason there is a large effort to have research journals require that the supporting data and methodology be made available along with the published study, with appropriate controls to protect the privacy of study subjects. That way, other researchers will be able to verify that the reported results are consistent with the data, and have the information necessary to replicate the study.
So it's just classic corporate science then? "We have this cool thing, trust us, it's soooo cool. No you can't see, but it's super cool. Trust us. Changes everything. Nothing will ever be the same."
[Product immediately ceases development once no profit is identified; never mentioned in public again.]
"It's possible that Google might be holding back the code for commercial reasons."
Ya think?
I can fault Google for many things but trying to develop commercial medical software is not one of them. I doubt Stanford or MIT is any more forthcoming on their commercial products.
Besides, there's nothing to prevent the boffins from developing their own cancer identification algorithms and if they do I assume that their respective institutions would patent and commercialize it.
I'd welcome if Google or any tech giant developed a better breast cancer screening mechanism.
Now if they started spamming the patients with reconstructive surgery ads, I'll happily put back on my Google bashing hat.
The problem is no one can verify if it really is a "better" screening mechanism. Google have said "it's better", and some data might temporarily suggest that is the case, but without a review there is no knowing whether or not there is a systemic problem undiscovered so far. Something like that can easily make it a "worse" screening system, which would harm patients.
It's the same with the Covid19 tracking apps we're all supposed to be using. Neither Google nor Apple have actually said what they've done with the Bluetooth bit, so every nations' app built on top is having to trust that Apple and Google have got it right. For all we know it's rubbish, thus rendering the apps built on top of it worthless. In a global pandemic, that's not great.
The main point for me was that "the web giant doesn't want its source code to be released until it's gone through a QA process"
Then don't publish results on it if they cannot be trusted.
No nature in Science and no science in Nature. They aim for hype and controversial articles.
At the time I expressed sceptism and said all these things are only pattern matching curated by experts.
So even if it worked, you need to keep training the experts.
So are Google trustworthy? Have they ever lied?
IBM famously misled about Watson, which wasn't even the same Watson that won Jeopardy and didn't really work for Cancer diagnosis.
AI is just hyped pattern matching using often biased and usually human curated data. If the so called "training data" (really a special kind of database) isn't expertly human curated confidence should be low.
The Google comment “It’s important to state that this is early stage research” does not indicate any imminent commercial application. So I'd consider it more probable that they sprinkled a generous helping of unicorn dust over their data and algorithm (maybe some p-hacking, aka tuning of the algorithm / data until the result conforms to expectations).
Second law of scientific publishing: nobody is allowed to see the unicorn dust.
I agree that if Google develops commercial medical diagnostic software that is wonderful. Far better than earning 75% of their income from advertising. However, "Nature" shouldn't be an advertising vehicle. It should be a vehicle for open exchange of practical scientific information and always enable verification of experiments.
They could set up a long term joint study with some cancer institute, running in parallel with existing diagnostics until those doctors became confident enough to incorporate the AI into their diagnoses.
Then those doctors could publish their experience and result in a medical journal, and AI project could be expanded to other hospitals, without ever divulging the technical secrets. (Having someone else report your success would make a bigger impact.)
Or they could have published in "Wired", who would be happy to hype the hype without supporting details.
The training material: were these from those health care records supplied to Google without the patient's notification and consent?
Did that 'anonymized' set of data include date/time and location information?
If so, can Google match a patient's data to an individual using cell phone and app location data?
Is Google's end game that healthcare providers start providing Google with patient info for evaluation?
That may be so - but how they're joined together is what matters, along with the training data.
In other news: this morning I have worked out how to travel faster than light, how to solve world peace, and how to play the flute. All the components of my method are open to the public already.
I have worked out how to travel faster than light, how to solve world peace, and how to play the flute.
So you were stumped when you attempted to solve the post Brexit frictionless border issue?
-->a beer for you, if you do solve it
There seems to be more and more of the wealthy Brexit proponents who heavily backed Brexit with cash and resources who, having achieved their aims, seem to be now leaving these shores to set up home in warmer climes, often tax havens.
PS, no that downvote wasn't me. I don't know if you are Brexit supporter or wealthy :-)
Neither. My plans were made long before the brexit fiasco crashed the currency, which has rendered my retirement potentially rather less comfortable than it might have been. Of course, if the UK rises to the challenge and the pound goes up, I'm better off - woohoo - but most of the more vocal pro-brexit industrialists seem to think that a low value pound is a good idea. Which it is, unless you want to spend it in, say, anywhere else in the world...
"
So you were stumped when you attempted to solve the post Brexit frictionless border issue?
"
I've already figured out a method to do that. I just need to solve a couple of minor technical issues. Such as how to time travel.
The general problem when custom computer in research is the the quality of the code and the more importantly the quality of the algorithm. If either one is flawed then the ultimate conclusions are very likely flawed. When others can review both there is more confidence in then the conclusions. Admittedly, errors in the algorithm are often harder to spot and are likely to linger longer. Often now there is a tendency to treat computer code and algorithms as proprietary, intellectual property that is never published. It appears too many think they can commercialize the code and make a lot of money if they can keep it private.
I remember a few years ago there was an economics paper (using an Excel spreadsheet) that had a typo in one of the calculations the authors missed. When it was pointed out the authors realized their mistake and pulled the paper. I believe the paper was peer reviewed so several people missed the error until after publication. The error was discovered, if I remember the details correctly, by someone reading the paper and digging into the spreadsheet and spotting the typo.
Should never have been published.
Certainly not in Nature!
How was it peer reviewed without the source code? Otherwise this is hocus pocus science at its worst.
Actually unbelievable that it was accepted never mind published. I’d never have got away with that and I’m first named author on 20 papers....
Who's to blame?
The article makes out that the publication of the article without the source code is Google's fault.
But I think Nature is equally responsible for accepting and publishing an article without the supporting reproducibility information. If they refused to publish, there wouldn't be a problem. Google does private research and keeps the results. Not an ideal world IMHO, but it is the existing one.
Surely if Google published a webservice that you could submit a sample and receive a non clinical assessment on that sample, then we could effectively run an extended trial of this service over time.
That being google, one could still wonder whether the service software and model data remain constant over time, or were constantly being "improved"
Claimed super-duper-astonishingly-good-you-won't-believe how good fractal compression.
Had half a dozen pictures with massive compression ratios.
Algorithm? Actually grad students stuck in office for months.
Finally released software was basically brute force search for the coefficients and was sloooooow. Let's call Michael Beasley's claims (for legal reasons) "optimistic"
This has a similar whiff. Missing detailed structure and critical parameters not listed. This should never have been given major attention in a scientific journal. Early publication credit for preliminary results fine. But where's the f**king follow up?
My BS meter is redlining like a Geiger counter in the engine compartment of a Cold War era Soviet nuclear submarine.
This is just another example of a common outcome of the progressive commercialisation of scientific research over few decades. Almost all research is now grant funded or otherwise paid for by a party with a vested (typically financial) interest in the results. Consequently, complete openness (the essence of true science) is contrary to the sponsors' perceived advantage. For the same reason, it's also considered bad form to fail, so results tend to be hyped.
... which Google claimed after they built a quantum-computer-like thing which basically changed state randomly, with no conceivable use, then claimed that it would take a conventional computer 10,000 years to simulate their device changing state randomly?
They're a publicly traded company. Publicly traded companies create buzz to influence their stock price. Elon Musk is an expert at it, but they all do it.
I'll believe the diagnostic AI when I see it do as well as a person which, claims to the contrary, it hasn't done yet.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
