Inside Facebook's engineering labs: Hardware heaven, HP hell – PICTURES Facebook's hardware development lab is either a paradise, a business opportunity, or a hell, depending on your viewpoint. If you're a hardware nerd who loves fiddling with data centre gear, ripping out extraneous fluff, and generally cutting the cost of your infrastructure, then the lab is a wonderful place where your dreams …
> Googling metal lathe appears to shoe El Reg is correct :(
Short of it being some sort of combo unit, I am pretty sure that's a milling machine. As a general rule if the motor is on top it isn't a lathe.
Lathes spin the piece against cutter whereas milling machines have a spinning cutter. If that is a lathe then it's an alarmingly badly designed one...
Definitely Bridgeport, I've got one sitting in my house complete with power table feed.
Like to get rid of it though... good steel, scraps for around 600. Just don't use it enough to justify the 150 a year it costs in taxes on the space it takes up.
Yeah, I think you missed the point. Unlike the rest us poor schmoes who are lucky if we're allowed to print a sheet of A4 without some HR/stationary nazi coming down on us for not thinking, these FB lads can print what they like, when they like and in 3D if they so choose and no one will tell them off.
There is actually considerably less innovation shown here than it seems. Most vendors have drive sleds with multiple drives deep, although I think most vendors avoid attaching that many SSDs because in normal land we use SSD for bandwidth and that many SSDs on a SAS loop would kill this PDQ. It's currently not possible to have sufficient CPUs in a box that size to allow enough SAS loops to saturate those disks either. Given that Facebook are after massive capacity and lower power they would be considerably better off manufacturing new SSD disks a few inches thick with more chips in them. Fewer SAS connections would lower the power requirements further while a 5" thick SSD would be large enough to incorporate its own cooling solution. It would also reduce their costs for engineers replacing them because they would fail less often and there would be fewer to swap out on failure. Given their desire for extremely low access speed and high capacity I'm also amazed they don't write custom firmware (maybe they do?) to utilise the spare capacity in SSDs which is usually reserved for replacing dead blocks. In enterprise drives there is double the stated capacity usually so this would be a much more worthwhile project for them than cocking about with 3D printers and quadcopters. Of course, I don't have billions of dollars to my name so clearly they are smarter than I am :)
The point of the exercise is to knock things up quickly and cheaply from commodity components.
Yeah, but they did it by defining their own commodity components in way of the open compute project. All that stuff is custom, but its now "facebook custom".
If facebook said "We're going to order $100m worth of 5" SSDs each year for the next 10 years at least", there would be companies falling over themselves in Taiwan to assemble them for them, and anyone else using OCP spec kit.
".....When these designs are published, the Asian companies are usually the ones selling the designs – at the cost of the profits of HP, Dell, Lenovo, and so on....." Yeah, right up until the customer stops to think about support, and what it means when their cheapo datacenter falls over and they can't get support unless they talk Chinese and can afford to wait around whilst half-a-dozen component manufacturers try and work out how their products fit together. At that point, the reality is businesses go buy hp, Dell, Lenovo, NetApp, EMC, etc., etc., who supply properly tested, integrated and supported products.
Well..... sort of.
Seeing as it's built out of commodity parts, you can either have a bin full of spares to cope with hardware failure, or since you built the thing, you'll have a really good idea how to fix it when it breaks. That having been said:
YES, It's certainly not for small businesses or for those without a clue (or for medium to large enterprises that demand 24/7 support from their vendors and also demand tried and tested hardware with similar support from the vendors.
However, if you have the talent in-house and they are willing to support it at 2 am when it falls over and needs fixing ASAP, then that's a different story.
>However, if you have the talent in-house and they are willing to support it at 2 am (or 24x7x52) when it falls over and needs fixing ASAP, then that's a different story.
Or alternatively, if you need to have talent sitting around at 2 am to fix stuff ASAP, perhaps it is cost effective to use their skills to make it easier to fix stuff ASAP...
Or you have so much kit that if device X breaks, the system recovers and keeps going via automated processes. Manual intervention should only be required to fix the broken device and restore that device to service - the fix may involve replacing a few parts or replacing the unit with the current model.
Maybe a new definition is needed for "cloud services" - a service consisting of clusters of data centres where losing any physical service up to and including a data centre does not result in a loss of service. Not all cloud service providers will meet this, but they should get somewhere close.
"Isn't Lenovo a chinese company Matt?" Yes, but they actually have a 24x7 support service that actually knows about more than just their own kit, documentation in English, labs that do proper integration testing (going on hearsay here), and provide COTS stacks.
I have done plenty of Linux DIY projects in enterprise settings, the problem usually starts when you hit an issue that affects all your systems, which means just dragging out another box out of the cupboard is not going to solve your problem. Usually it's because you updated something to get a new feature or because a vendor of one part of your solution released new drivers to fix a possible problem (such as a security hole). So you go to the maker of that product and - if you're lucky - they say stick a fix on, only when you test it something else on another vendor's bit of the stack breaks. So you go talk to that vendor (if you can), and they say they have never seen the problem, remove the other vendor's update or go without support. But you can't, because the business wants that new feature or security fix, so you have to hack a workaround that may break something different elsewhere, and so on and so on. That is the reality of kludging together untested stacks. Sure, if you can find and employ a team of gurus to build and maintain your stack you may survive, and if you're doing nothing more than posting webpages, and you can afford the time to play around with plywood trays, you could prosper, otherwise COTS, tried-and-tested stacks with proper support make a lot more sense to the business.
>the problem usually starts when you hit an issue that affects all your systems, which means just dragging out another box out of the cupboard is not going to solve your problem.
We've already seen this problem with cloud providers eg. Azure last autumn...
Not half as cool as going to the next stage and actually building your own motherboards etc.
One of things about the IT industry in the late 70's and early 80's was the number of companies who were building their own hardware and hence also required their own software engineering team. Hence to me the absence of scope's, ICE and other test systems from the Facebook lab's is quite telling.
Talking about external suppliers and 'In-House' expertise is an interesting exercise in economics: we know it makes sense for small and medium-sized companies to use external expertise for almost every 'non-core' skill and activity.This is less true for larger companies - but big companies still contract-out everything from catering to networks and desktop support.
Are they wrong? Or is there a size of company that's so big that they are a nation unto themselves and their internal IT functions are bigger than IBM? Or, at least, bigger than any hardware supplier and service provider that would take on an S&P500 company's IT?
It's staggering that an IT software and service provider - and that's what Facebook is - is so large that it 'in-sourcing' the hardware supply budget can generate internal economies of scale that exceed the efficiency of the world's biggest hardware suppliers. That's supposed to be an economic absurdity - one company, one hardware customer, is a bigger market than the entire customer base of the world's biggest hardware supplier, or their second and third-largest rival?
Is there no advantage to specialisation at all? Or some kind of economic 'event horizon', beyond which space and time and economics curve around and disappear when a gravitation amassment of money exceeds some critical threshold?
Basic economics reads the FaceBook 'in-house' hardware supplier as a signal that the IT hardware industry has a *lot* of room for consolidation, and efficiency gains, and for trimming excessive profit margins. Except that competition should've done that already, if economists are right. And we've already got a heavily consolidated industry with dominant suppliers and razor-thin margins.
Could it be that consolidation and shrinking margins have left us with unresponsive monopolists and no money for investment in cheaper products and more efficient production?
Meanwhile, management: is a company that size actually manageable? As in: controllable in terms of internal direction and supervision?
It goes without saying that we've long since passed the event horizon for external supervision by governments and the rule of law to prevent abusive monopolies, regulatory capture, and tax evasion.
That's supposed to be an economic absurdity - one company, one hardware customer, is a bigger market than the entire customer base of the world's biggest hardware supplier, or their second and third-largest rival?
It's called "specializing for a special task". It is possible if commodity stuff can be thrown together to build a specialized tool as materials and know-how become commponplace. It's the same reason why there are not only 5 companies building houses in the world.
It goes without saying that we've long since passed the event horizon for external supervision by governments and the rule of law to prevent abusive monopolies, regulatory capture, and tax evasion.
I don't know what world you are living in, but we are frankly into Nazi territory and moving deeper regarding:
1) Arbitrary antitrust shit whenever some (generally leftish) cunt in government feels like it "Uh... I think people are collunding ... how horrible. Let's reduce the price forcefully for the peons!!" Sure a formula for Great Success. As opposed to letting the monopoly die on its arse by enabling some competition.
2) Regulatory capture being ENABLED by government at full throttle. Checked whether you company is "green enough" lately? Yep, you are being forced to dump the lightbulbs for no good reason.
3) "Tax evasion" being chased down like Himmler's mystical projects depended on it. Which is why you hear about it so much. These taxfeeders are voting themselves pay increases and are HUNGRY.
Nah. Just too many 'safe-pair-of-hands' middle managers. Who will only buy ripoff Cisco and HP kit because they like crawling handcuffed into sacks to take a kicking. Buying commodity stuff without outrageously overpriced warranties or useless support contracts is a no-no. Safe is expensive.
I suppose if you're spending $3bn in opex, sometimes it can be worth speculatively spending $50m in capex to try and reduce that to $2bn, but delivering 4 times the capacity.
Plus facebook imagines itself a disruptive company. You don't pay IBM $50m when you can put some geniuses in a shed with some credit cards - much more disruptive.
I actually like what FB are doing, and should be commended for allowing their staff to be creative and find solutions to problems without having to have degree's in who knows what and somehow get through to someone at the top of the top of the chain. Encouraging staff to FIND and create a better system for them is great for them and at the same time a real boost for staff when their idea becomes used and is successful. Well done FB, a lot of respect earnt by me for this .
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2025
