back to article Square Kilometre Array precursor shrinks 5TB of data to 22MB – every second!

Australia's precursor to the Square Kilometre Array has gone from sitting on the slipway to shedding champagne-bottle shards and sliding gracefully into action. The Australian Square Kilometre Array Pathfinder, ASKAP if you're thrifty with syllables, is doing its first science for a project called WALLABY, an all-sky hydrogen …

  1. pro-logic

    I invented a program that downloads porn off the internet one million times faster.

    Ahh reminds me of Doug from the Simpsons:

    Doug: I invented a program that downloads porn off the internet one million times faster.

    5TB to 22MB is impressive compression!

    1. Little Mouse Silver badge

      Re: I invented a program that downloads porn off the internet one million times faster.

      impressive compression

      Intuitively you'd assume that "lots-of-black lots-of-black white-dot lots-of-black" bloody well should compress pretty well, but having had a stab at (optical) night-sky photography myself, I can say from experience that there's almost no black out there at all.

      Zoom in on any "dark" patch of sky and you'll just see more stars. Zoom in to the dark bits in-between and guess what? It's full of stars.

      1. hplasm
        Alien

        Re: I invented a program that downloads porn off the internet one million times faster.

        " Zoom in to the dark bits in-between and guess what? It's full of stars."

        Oh my god! It is...

      2. Alan Brown Silver badge

        Re: I invented a program that downloads porn off the internet one million times faster.

        "Zoom in on any "dark" patch of sky and you'll just see more stars. Zoom in to the dark bits in-between and guess what? It's full of stars."

        It still compresses pretty well. I deal with some archives of glass plates and they use surprisingly little space per image (the factor of there being a few tens of millions of them is another matter)

  2. Kevin McMurtrie Silver badge

    This sounds like synthetic aperture imaging. The antennas are receiving exactly the same signals but some components of the signal have incredibly slight phase differences. Those phase differences are the imaging data so the raw samples can be discarded.

    What are the odds that re-analyzing the raw samples would reveal new details that are no longer present in the sky? Not good enough to justify archiving 5TB/s with current levels of technology.

  3. PNGuinn
    Black Helicopters

    Firehose of data ...

    I hope they keep backups in case their data get hosed ....

    Couldn't they keep it locally in a disused dunny with a notice on the door saying "Beware of ol' Redback"?

    1. Adam 1

      Re: Firehose of data ...

      I'm sure HPE are on top of it.

  4. Mikel

    Sparse arrays

    Most of the picture is black.

    1. Anonymous Coward
      Anonymous Coward

      Re: Sparse arrays

      Was going to say it is highly undersampled, so it is not just a steered array, modern estimation techniques for sparse sampling of images, and the fact that one is often looking at point sources (we know something of the nature of the image from optical sources in other situations) are involved here.

      1. Anonymous Coward
        Anonymous Coward

        Re: Sparse arrays

        Also forgot to mention that if you keep the time data for all sensors, you can "focus" offline in different directions by matching correct time delays in the data to where you want to look.

  5. Anonymous Coward
    Anonymous Coward

    My desktop computer has 4TB and can easily write 22mb a second

    What gives that they can't even afford to keep the compressed data even at the reduced rate? How much did they spend on the antenna and all the other setup that they can't afford to store ~2TB a day?

    1. Alan Brown Silver badge

      Re: My desktop computer has 4TB and can easily write 22mb a second

      " How much did they spend on the antenna and all the other setup that they can't afford to store ~2TB a day?"

      2TB/day is a new storage array every 3-6 months, PLUS the power to keep it online.

      Or a very large tape silo.

      And unlike satellite data (where it's hard to go back to 1970 and rescan atmospheric data), the original source is still out there if you need to redo things.

      1. druck Silver badge

        Re: My desktop computer has 4TB and can easily write 22mb a second

        Not if it's a fast gamma ray burst, or other short lived phenomenon.

        1. John Mangan

          Re: My desktop computer has 4TB and can easily write 22mb a second

          Would you be looking for a GRB at radio wavelengths?

      2. Korev Silver badge
        Boffin

        Re: My desktop computer has 4TB and can easily write 22mb a second

        2TB/day is a new storage array every 3-6 months, PLUS the power to keep it online.

        This is one of the scenarios where scale-out NASes work rather well. I'd rather hope they have some kind of data management software to control what gets saved and archived etc.

    2. Roland6 Silver badge

      Re: My desktop computer has 4TB and can easily write 22mb a second

      What gives that they can't even afford to keep the compressed data even at the reduced rate?

      If the statements made in the article are correct, sheer volume of data. It would seem that each antenna generates 5.2 TB/sec; currently they have 12 installed and will ultimately have 36. A volume of data that makes commercial "Big Data" solutions look decidedly small...

      So their ultimate daily data volume is circa: 22mbps x 36 x 60 x 60 x 24 which is 68.5TB per day...

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon