back to article Facebook says vendor secrets forced it to homebrew switches

It's four months since Facebook first launched its Wedge switch and accompanying FBOSS operating system. Some forms of Wedge are in production and others are in testing, so El Reg decided to talk with Facebook's VP of network engineering, Najam Ahmad, to see where The Social Network is at with its software-defined networking ( …

  1. Mr C

    FB is in a position to build a homebrew switch

    throw enough resources at it and you can do things outside of your usual field of expertise (which for FB is social stuff)

    Good for them that they stepped away from big vendors who think they need secret codes to hold on to their position. Needing months to fix bugs is sometimes also an eternity.

    I've long seen such deplorable behavior and some vendors can get away with it in 99% of the cases since the little guy (little company) doesn't have the resources to take them on.

    How ironic would it be if FB ever released these switches with their own secret code build in

    1. Preston Munchensonton

      Re: FB is in a position to build a homebrew switch

      And on the flip side, most enterprises don't have a large enough scale for any hosting to trigger the sort of unusual ASIC hiccups that FB encounters regularly. In other words, FB is filling a very small niche and most enterprises will never need it (or afford it).

    2. Bernard

      Re: FB is in a position to build a homebrew switch

      I'd say this kind of tech is sort of core for them in the sense that at this scale there isn't any service they can buy in that will do it better (granted that's not an orthodox definition, but I do think that tasks become non-core only when buying it in would work better than doing it yourself).

      It could well be that if they get good at this stuff they think about spinning it off in an AWS way to other web scale brands. If they don't, their staff who perfect it will definitely leave and set themselves up to do this elsewhere.

      1. phil dude

        Re: FB is in a position to build a homebrew switch

        happens on supercomputers more often than you would like to know...


  2. Anonymous Coward
    Anonymous Coward

    ...big vendors who think they need secret codes...

    Is this news? Every big software and hardware vendor has been doing this FOR EVER. An early example would be back in the 1990's with the MSDOS versus DRDOS business when BillG is alleged to have manipulated Microsoft APIs in Microsoft's favour.

  3. Anonymous Coward
    Anonymous Coward

    At the opposite end of the scale

    There's BT (OpenReach) and their approach to faultfinding on the local loop for DSL faults.

    An ADSL modem is a marvellous piece of technology, and the people that did the early standards did a grand job.

    The hardware (the DSP) has effectively already got a time domain reflectometer for locating discontinuities in the cable.

    The low level DSL protocol has "last gasp" messages so the owner of the exchange kit can tell the difference between a controlled shutdown and an unplanned loss of signal e.g. due to a fault.

    The low level DSL protocol also exchanges details of signal quality, error counts, useful stuff like that, and the DSL kit at each end makes the numbers available to the user (your SNR, CRC, FEC, HEC, etc).

    All this regardless of vendor.

    Posh kit might even have SNMP (e.g. the DSL Line MIB) in addition, but it's not sufficiently common to be widely useful for diagnostics.

    All in all, a veritable galaxy of faultfinding assistance just waiting to be used, generally accessible equally well from the consumer kit or from the exchange kit (if anybody's sufficiently motivated).

    What's the typical approach to DSL faultfinding at OpenReach?

    Ignore all the above, wait for an unhappy customer to phone in, ignore all the above, threaten to charge £100+ for a faultfinding visit, ignore the builtin diagnostics and hope that if a technician does show up the fault is visible at the appropriate time, and then rely on adequately clued-up technicians to use their portable testgear to do much of the same diagnostic stuff that is inherently and unavoidably built into the DSL modem.

    That manual intervention approach might have worked for an installed base of a few hundred modems per exchange.

    I think we got a bit further than that in recent years,

    Just like FaceBook, manual intervention doesn't seem appropriate on that scale. Unlike FaceBook, the DSL diagnostic stuff is already built in **and published**. Unlike Facebook, where broken kit may mean lost revenue, BT see "rent-a-technician" as a revenue opportunity.

    Well done Facebook.

    1. Anonymous Coward
      Anonymous Coward

      Re: At the opposite end of the scale

      Yes, the standards detail a vast number of brilliant diagnostic features which would make any network management company's life easier; unfortunately, I have never seen these features integrated into any consumer or business class kit for one reason alone - cost. The backend systems exist, the technology exists, the will to build doesn't.

  4. Paul Smith


    "...Finally, and this is advice of the most sound sort, if it looks like, in spite of all efforts, your kludge will begin to resemble an ordinary computing machine, it is time to put more men on the job...."

  5. Californiacation

    That's great and kudos to the FB engineers.

    However, engineers always love to build stuff. Once it's built it needs to be supported in all perpetuity and that's where issues arise. FB now needs a SDN team to keep their thing running, up-to-date, following innovation and industry standards, respond to zero-day-exploits, and all other things associated with BEING IN THE NETWORK SOFTWARE BUSINESS.

    Give it a few years and that team/division will become so big that it will warrant a spin off or they will just go to an established product.

    1. Bernard

      Agree, but if there's nothing they can buy in that works as necessary at this size then scale out and spin off could be the more likely option. It would make tech a revenue source and not just a cost centre.

    2. Anonymous Coward
      Anonymous Coward

      Sounds familiar

      "Give it a few years and that team/division will become so big that it will warrant a spin off or they will just go to an established product."

      Company I'm a contractor at will stand up some PoC that everyone forgets about and then a year or two down the road they find it is a critical piece of the daily work and it never got changed over to a production system.

      Anon so I can't be yelled at...

  6. Anonymous Coward
    Anonymous Coward

    I suppose this may help Linux play catchup

    This may help linux play catchup to freebsd in the networking stakes, though obviously facebook is infected with linbois, as common sense would dictate they went with freebsd in the first place.

    betamax vs vhs all over again

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like