back to article Facebook building 'on-demand executable file format' that self-inflates using homebrew compression

Facebook has developed promising asymmetric compression technology and aspires to share it with the world. Dubbed "Superpack" and detailed in a post by Facebook Engineering software engineer Sapan Bhatia, the tech "combines compiler analysis with data compression to uncover size optimizations beyond the capability of …

  1. DrXym Silver badge

    I have a better way to save space on phones

    Stop shipping Facebook garbage in the firmware so the read-only partition can be smaller.

    1. sw guy
      Pint

      Re: I have a better way to save space on phones

      I came here to post same comment, but you beat me to it.

      Have a (see icon), later in the day, though.

    2. elsergiovolador Silver badge

      Re: I have a better way to save space on phones

      But kickbacks...

  2. GreenJimll

    Start your engines virus folks...

    I see another fertile battle ground emerging for the virus/anti-viral code hackers to start working on... ;-)

    1. The Man Who Fell To Earth Silver badge
      FAIL

      Nothing to see. Move along...

      As usual, there's nothing original in what Facebook is doing.

      In Windows, compressing dll's and putting the compressed result in an executable's resource area has been around almost as long as Windows itself. Used more back in the days of floppies when space was at a premium. When the exe starts, inflate the embedded compressed dll's in the exe's local directory, and when the exe finished, delete them. Allows you to still architect using custom dynamic-link libraries yet give the appearance of the program being only a single exe, and more importantly, saves space on the media. Only penalties are the main exe has to contain the decompression routine (or calls to Windows Compression API if Win 8 or later), which isn't a big deal, and the program temporarily will require room for the decompressed dll's.

      1. Dale 3

        Re: Nothing to see. Move along...

        Packing the executable with DLLs wasn't the novelty. The novelty was in how they packed it, removed redundancy and encoded entropy to improve compression ratios.

        1. gerdesj Silver badge
          Linux

          Re: Nothing to see. Move along...

          Is it a better mousetrap?

          It depends. Arch Linux recently moved to .xz from .bz2 or .tar.gz (can't remember). Anyway, the additional compression is a roughly 10% saving. However, these things take something like 15x longer to create.

          Now, the distro sources are created once and then downloaded millions of times, so a small saving in size repays the additional effort of compression, many times over. I think they also take longer to decompress too but with SSD these days - who knows? I don't really notice. The even more recent parallel download thing offsets even that.

          1. emfiliane

            Re: Nothing to see. Move along...

            .xz (lzma2) is *significantly* faster to decompress than either .gz (gzip) or .bz2 (bzip2), and it improved install time along with ISO size. Big win all around, the old standbys might be familiar but they're still 70's tech.

            But they moved to .xz in... 2010? The one the recently moved to is .zstd, at the end of 2019, which is very slightly larger than .xz but ten times faster to decompress, again making for faster installs, especially on high end SSDs that can soak up data that quick.

            1. chroot

              Re: Nothing to see. Move along...

              In addition: Zstandard (zstd) is also from Facebook and is known for being fast. It compresses and decompresses really fast, unless you use the higher level compressions, beyond 15, more or less. Default compression level is only 3.

              I highly recommend it as a replacement for gzip. I use it to compress 16 GB files at level 6 with multithreading (-T0).

              If you're running Archlinux and building packages yourself, you may want to lower the default compression level and enable multithreading in your /etc/makepkg.conf with:

              COMPRESSZST=(zstd -czv -T0 -9)

        2. juice Silver badge

          Re: Nothing to see. Move along...

          > Packing the executable with DLLs wasn't the novelty. The novelty was in how they packed it, removed redundancy and encoded entropy to improve compression ratios.

          From reading the article, it sounds like Facebook have come up with the idea of embedding some sort of programming language into the compression system, so that it can run decompression routines which have been perfectly tailored for the content, rather than a generic "one size fits all" algorithm.

          But that's not something new - the RAR format has had a VM system for years, that lets people hand-craft code to improve compression ratios. And as is ever the case, someone even worked out how to write a Hello World program within said VM.

          https://sudonull.com/post/136971-Hello-world-for-RAR-virtual-machine-RarVM

          https://blog.cmpxchg8b.com/2012/09/fun-with-constrained-programming.html

          To be fair, when you've got around 2 billion people using your app on a daily basis, even knocking a few kilobytes off each download makes a difference. Especially when at a glance, they've rolled out over 30 updates just for Android phones in the last 9 months.

          https://androidapksfree.com/facebook/com-facebook-katana/old/

          And it's not like Facebook is short on resources. But I'd still be inclined to get someone to take a long hard look at why the app itself is so "bloated", rather than faffing around with ways to reduce the impact of said bloat.

      2. emfiliane

        Re: Nothing to see. Move along...

        I thought they only started writing to disk when page protection FINALLY appeared on XP SP2. Before that, they would just decompress to RAM, and change the pointer to the EXE/DLL to the new location, since there was zero protection and all memory was read/write/execute outside of the kernel. Meant they couldn't page it out, so it took up more memory, but it's a tradeoff when disk is also at a premium.

    2. Dan 55 Silver badge
      Meh

      Re: Start your engines virus folks...

      It's probably more about slowing down Google Play app scanning, knowing Zuckerborg.

      1. Anonymous Coward
        Anonymous Coward

        Re: Start your engines virus folks...

        You beat me to it. It is probably also going to make it harder to work out what their apps are really doing versus their proclaimed purpose, and I trust Zuck about as much as the average heroin dealer. Not that O know any, but I don't know Zuck either..

  3. sreynolds

    I guess they needed it...

    If they are offloading all the monitoring and decoding of your facial and voice expressions to your device. assisting you when you go to to the toiler (private time for you to be with Zuck) there is a lot of code that just won't fit on the traditional flash based devices.

  4. Anonymous Coward
    Anonymous Coward

    Its a shame Facebook hires genuinely good engineers, I wonder how they feel about their working being used by perverts and psychos

  5. Tromos

    20%? That's nothing...

    ...I've found a way of compressing Facebook executables 100%.

    1. Francis Boyle Silver badge

      I think you forgot to mention

      that the process is lossless.

      1. Korev Silver badge
        Pint

        Re: I think you forgot to mention

        Bravo both of you -->

      2. Artzig

        Re: I think you forgot to mention

        Not at all. It's a 100% gain.

  6. _LC_ Silver badge
    Holmes

    I still remember this well from the C64

    I can still remember this well from the C64, where every bit counted. Android is so “efficient” that this is coming back.

    Only a few years ago, a machine with 10 cores, 12 GB of RAM and 256 GB of disk space, would have been called a “mainframe”. It would have had dozens of users working on it. ;-)

    1. Plest Bronze badge

      Re: I still remember this well from the C64

      Just what I was thinking. If FB wants to learn about packing get some of those coding/hacking groups from the 80s/90s in to teach these guys how to "hack, crack 'n pack" something.

      Those guys were incredible, they could take games with 100+ files over 2 floppies, pack them down into a single executable file that ran off 500kb without the need for any temp unpack space, all done in memory, in real time on CPUs with a fraction of the power in a modern phone.

      1. Version 1.0 Silver badge
        Boffin

        Re: I still remember this well from the C64

        When I started writing programs I was working in about 48kb of memory, the BIOS and CP/M using the rest of it and virtually everyone in that world wrote programs that worked well, even if they had to use half a floppy disk to store the data - I was writing in assembler with a Z80 and using an 8048 when I built hardware devices.

        There are quite a few factors that have resulted in massive program size increases - for example think about adding 1+1+1+1+1 in a 64-bit environment where the default calculations are always floating point.

        1. _LC_ Silver badge

          Re: I still remember this well from the C64

          "where the default calculations are always floating point"

          Huh? You might want to reset your compass on that one. ;-)

  7. Andy The Hat Silver badge

    What goes around ...

    First there were applications and data and they were small to fit the space available

    Then there was too much data so compression was invented

    Then there was more storage and bloatware was developed to fit the space available

    Then there was slow internet and compression was reinvented

    Then there was broadband and bloatware development was enhanced to fit the bandwidth available

    Then there was too much data and compression was reinvented invented ...

    1. red19

      Re: What goes around ...

      All things are wearisome, more than one can say.

      What has been will be again, what has been done will be done again; there is nothing new under the sun

  8. Steve Kerr

    give developers minimum spec

    Giving developers super fast machines with max everything means they don't care about efficency generally.

    Give them yesteryears base models to work with and that will reduce bloat as they will have to work within the framework of what they have.

    That's what I'd do!

    back in the days of spectrums, c64s etc....software developers had to be so much more efficient to work within the limitations they had.

    1. MOH

      Re: give developers minimum spec

      If they use their own software, it might be more effective to give the CEO a bog-standard device.

      I'm sure the message will filter down to the devs pretty quickly.

    2. IGotOut Silver badge

      Re: give developers minimum spec

      This is exactly what they did for the game MDK many years back. They developed it on good machines, but then had to test it on machines that were bottom rung of the ladder. If it ran crap, they had to go back and work out why.

      It's why it was one if the smoothest games of its time.

    3. Anonymous Coward
      Anonymous Coward

      Re: give developers minimum spec

      Giving super fast machines isn't a problem, the problem is either:

      1) Crap devs

      or

      2) Crap management overruling the devs doing things right in order to ship the lowest quality product the company can get away with.

      You give devs low quality machines they will walk, or your costs will go up as everything will take longer due to either the hardware, or the devs moaning about how shit it is.

    4. Irongut

      Re: give developers minimum spec

      And your developers will leave for better jobs.

      Making developers lives harder with slow dev machines is not the answeer, especially since we're talking Android here and you don't develop Android apps on Android. Mid-range devices and older devices are useful in the test drawer but you also need the latest devices so you can test against the latest OS features.

      The answer is to teach junior devs to care about resource usage, not tie one hand behind their back to hamper them.

    5. Anonymous Coward
      Anonymous Coward

      Re: give developers minimum spec

      AFAIK that's actually the core reason why Russian coders are so much more efficient. The US embargoed the supply of the newest PCs to Russia, so they had to get better using what they have and they definitely did.

    6. ShadowSystems

      Re: give developers minimum spec

      At Steve Kerr. I'd give them decent machines to code on, but then force the resulting software to be run on the lowliest machine possible. If they get the job done faster because of their high powered machines, so much the better, but if it runs like shit (or not at all) o, say, a Raspberry Pi with 1Gb of RAM & a 32Gb SD card for file storeage, then make them recode the program until it runs (properly/at all).

      "But we're running it on modern smartphones with a zillion cores able to deal with multiple threads & half a Tb of RAM, so why code for something so crap?" Because you can't ensure that the person that wants/needs to run your software will be using it on such a device. What if it's in a third-world-nation where the average smartphone is over a decade old? Do you personally pay for each of those people to get a modern handset, or do you write your code so it runs worth a damn on said limited hardware?

      *Hands you a pint* Drink up. It'll help lubricate your brain & limber up those coding muscles to do some serious optimization. =-)

    7. smot

      Re: give developers minimum spec

      Indeed. In my earlier life, we used a combination of Huffman and library tokens to compress both text & graphics to allow some seriously large adventure games to run on Speccies. 48k seemed like a bottomless pit.

  9. elsergiovolador Silver badge

    RLE

    Did they reinvent RLE?

    The most important project Facebook isn't working on, is how to close all its operations and delete all data.

    1. Plest Bronze badge

      Re: RLE

      Given the latest revelation that FB has 5.8 million VIPs on a whitelist to whatever they liked without caring about FBs terms and conditions, I think if you don't have an account don't get one, if you have get rid of it and tell your friends and family that you all need to move to something that isn't a cesspit of pond scum that is FB.

  10. KittenHuffer Silver badge
    Mushroom

    I'm surprised ....

    .... that they haven't invented Blipverts yet!

    --------------> T-t-t-t-too easy-e-e-e-e!

  11. minnsey231

    Surely Pied Piper have prior art?

    Sounds like 'middle out' to me.

  12. FIA Silver badge

    Sounds great for apps, not so good for long term storage though if you're having to literally execute your archive to unpack it.

    1. AMBxx Silver badge

      Not so good for batter life either.

      1. The First Dave

        Battery life is somebody else's problem.

  13. Howard Sway

    Mmmm, lots of free space caused by the new executable file format

    Surely they won't mind if we insist on building in a couple of extra data slurping and advertising APIs into every app in return..................

    1. MrReynolds2U Bronze badge

      Re: Mmmm, lots of free space caused by the new executable file format

      Proprietary compression means it may take a while for AV scanning services to be able to accurately scan inside these executables for malicious code. Best to just blocklist anything with that file signature / magic number.

      1. Korev Silver badge

        Re: Mmmm, lots of free space caused by the new executable file format

        Like when you open a Zip file with McAfee on the machine...

  14. Mike 137 Silver badge

    Priorities

    "Bhatia explains that Facebook needs compression because its apps keep bloating"

    Maybe that's the first problem to address. I have a few well written specialist applications that have excellent performance and will run on almost any Windows box and they have one other common property - they have a really small code base. For example a powerful optimising C compiler for PIC with an advanced drag and drop IDE - the whole things is a few tens of megabytes apart from the device specification files, and it runs on everything back to XP SP2.

    Bloat primarily occurs as a result of "tweaking" as opposed to re-engineering. In modern applications there's often a lot of code that never gets executed, either because it's part of a massively redundant library or because it got left in when it was no longer needed by the next version. In either case it never gets called at run time, so it might as well not be there.

    Compressing the redundant stuff - however expertly - seems to be merely first aid rather than medicine.

  15. fidodogbreath Silver badge

    "has uses beyond that"

    Self-expanding proprietary compression could make it easier to hide the full scope of their spyware SDK in other apps.

  16. Anonymous Coward
    Anonymous Coward

    Just ditch React

    Maybe if they just wrote thin, efficient code in the first place. It amazes me how poorly FB actually works, on rare occasions I visit their website. Half the time it doesn't work without a few refreshes.

    1. Anonymous Coward
      Anonymous Coward

      Re: Just ditch React

      Everyone could profit from this example and while they're at it ditch AngularJS too

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2021