Square Kilometre Array precursor shrinks 5TB of data to 22MB – every second! Australia's precursor to the Square Kilometre Array has gone from sitting on the slipway to shedding champagne-bottle shards and sliding gracefully into action. The Australian Square Kilometre Array Pathfinder, ASKAP if you're thrifty with syllables, is doing its first science for a project called WALLABY, an all-sky hydrogen …
impressive compression
Intuitively you'd assume that "lots-of-black lots-of-black white-dot lots-of-black" bloody well should compress pretty well, but having had a stab at (optical) night-sky photography myself, I can say from experience that there's almost no black out there at all.
Zoom in on any "dark" patch of sky and you'll just see more stars. Zoom in to the dark bits in-between and guess what? It's full of stars.
"Zoom in on any "dark" patch of sky and you'll just see more stars. Zoom in to the dark bits in-between and guess what? It's full of stars."
It still compresses pretty well. I deal with some archives of glass plates and they use surprisingly little space per image (the factor of there being a few tens of millions of them is another matter)
This sounds like synthetic aperture imaging. The antennas are receiving exactly the same signals but some components of the signal have incredibly slight phase differences. Those phase differences are the imaging data so the raw samples can be discarded.
What are the odds that re-analyzing the raw samples would reveal new details that are no longer present in the sky? Not good enough to justify archiving 5TB/s with current levels of technology.
Was going to say it is highly undersampled, so it is not just a steered array, modern estimation techniques for sparse sampling of images, and the fact that one is often looking at point sources (we know something of the nature of the image from optical sources in other situations) are involved here.
" How much did they spend on the antenna and all the other setup that they can't afford to store ~2TB a day?"
2TB/day is a new storage array every 3-6 months, PLUS the power to keep it online.
Or a very large tape silo.
And unlike satellite data (where it's hard to go back to 1970 and rescan atmospheric data), the original source is still out there if you need to redo things.
2TB/day is a new storage array every 3-6 months, PLUS the power to keep it online.
This is one of the scenarios where scale-out NASes work rather well. I'd rather hope they have some kind of data management software to control what gets saved and archived etc.
What gives that they can't even afford to keep the compressed data even at the reduced rate?
If the statements made in the article are correct, sheer volume of data. It would seem that each antenna generates 5.2 TB/sec; currently they have 12 installed and will ultimately have 36. A volume of data that makes commercial "Big Data" solutions look decidedly small...
So their ultimate daily data volume is circa: 22mbps x 36 x 60 x 60 x 24 which is 68.5TB per day...
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
