IBM lifts lid, unleashes Linux-based x86 killer on unsuspecting world

IBM is mounting its strongest challenge yet to x86 hegemony with the unveiling of its spanking new Linux based S822LC system. What makes this system different is that it’s based on a new processor/motherboard configuration, complete with a sporty NVIDIA NVLink connector. According to my research, close to half of you HPC users …

[Bob Rocket]

Awesome

No idea what it is, what it does or what it could be used for but it sounds amazing.

The IBM Power Systems S822LC Technical Overview and Introduction Red Book has the following on page 12

THIS PAGE INTENTIONALLY LEFT BLANK

so we know this is a serious bit of kit, I want one (whether it plays crysis or not)

http://www.redbooks.ibm.com/redpapers/pdfs/redp5283.pdf

[Ralph B]

Re: Awesome

The documentation of really serious kit uses "THIS PAGE INTENTIONALLY LEFT WHAT WOULD HAVE OTHERWISE BEEN BLANK WERE IT NOT FOR THIS MESSAGE"

[Primus Secundus Tertius]

Re: Awesome

Or, to put it more briefly:

THIS PAGE IS INTENTIONALLY ALMOST BLANK

After reading a fewo of those, one's mind begins to feel almost blank.

[Lusty]

Re: Awesome

I recently read the ATC regs (CAP670) and finally worked out what the intentionally "blank" pages were for. Gotta be honest, it's a load off my mind and I've no idea why more hefty documents don't explain properly.

[m0rt]

Re: Awesome

"I want one (whether it plays crysis or not)"

Heretic.

[P. Lee]

Re: Awesome

>>"I want one (whether it plays crysis or not)"

>Heretic.

Heretic runs on Linux - you'll be fine.

[Bloodbeastterror]

Re: Awesome

"finally worked out what the intentionally "blank" pages were for"

Please share - I'm genuinely interested, having been puzzled by this apparent stupidity for years.

[allthecoolshortnamesweretaken]

Re: Awesome

I don't know whether this is the intended effect, but I find this rather helpful when making copies of something. Doesn't leave you puzzling (and shuffling paper) trying to determine whether that page is supposed to be blank or the copy machine fucked up again, especially on double-sided jobs.

[Lusty]

Re: Awesome

It's for when you print a document which is designed to be put in a ring binder which has tabbed dividers to allow sections to be found quickly. With the "blank" page, a new section is guaranteed to be forward facing. It's also used to allow sections to be replaced within a document easily after printing. Cap 670 comes out at a couple of inches when duplex printed, so the liklihood of someone printing it and wasting paper with a "blank" is quite low. There is a section in there explaining how to organise the doc (which is made up of several "books" into a large binder which should be kept at the ATC.

I'd recommend it as a good read, but it's almost entirely non-IT so unless you're really, really into air traffic then probably give it a miss. ED109A is a good IT read though, but expensive.

Do I win king of nerds for enjoying regulatory documents?

[the spectacularly refined chap]

Re: Awesome

It's for when you print a document which is designed to be put in a ring binder which has tabbed dividers to allow sections to be found quickly. With the "blank" page, a new section is guaranteed to be forward facing.

I'd always understood it to be for selective printing - i.e. you have a 100 page document and need only chapter six at 50 pages - you can print and bind that separately without messing up the pagination. Also works if you need to break the document up into volumes because of limitations on binding capacity.

[Bloodbeastterror]

Re: Awesome

@Lusty

"Do I win king of nerds for enjoying regulatory documents?"

Yes, you do, and I'm glad you are, since this is another lifetime puzzle solved... Have an upvote... :-)

It's obvious when you explain it, but I guess it just never occurred to me that someone would want to print something which is already on paper in front of me - I hadn't considered the electronic versions.

[Charles 9]

Re: Awesome

"It's for when you print a document which is designed to be put in a ring binder which has tabbed dividers to allow sections to be found quickly."

I thought so.

In layman's terms, they want sections to always start on a right-hand page. When printed out under normal duplexing, right-hand pages are always odd. "Intentionally blank" pages are always even (left-hand) and would be covered by the divider when someone picks up the tab and flips it.

[Bob Rocket]

Re: Awesome

at my first proper computery job they had two IBM 8100's (serious bits of kit with 10 meg hard drives), one for development and one for production, both were commissioned at the same time.

The IBM engineers had assembled the manuals for the dev machine (they filled a full height double fronted metal cabinet) and my task was to assemble the production machine manuals so that we had two identical cabinets filled with identical manuals.

This was where I first saw 'this page has been left intentionally blank' message, I queried whether I had to include these pages and the explanation was as Lusty pointed out, it tickled me that they still put those pages in a pdf document.

[Dave 126]

Re: Awesome

-Good luck on your voyage Blackadder. The finest cartographers in the land have prepared this Atlas for you.

-Thank you Melchie... Wait, the pages are all blank!

-Yes. They're hoping you could fill it in for them as you go along.

[John Sanders]

Re: Awesome

Always used them to make notes about erratas:

Page 35 where it says "rsync -aviw ...." they mean "rsync -aviW..."

And so on, in the book where the fault is found I write an asterisk + E next to the piece "*E"

Simples.

[John Savard]

Re: Awesome

I've always felt they should use small type at the bottom of the page to say "The relative blankness of this page is intentional, for formatting reasons, and should not be construed to indicate that a printing error has left text missing from this document". Then no one could criticize the statement for being less than 100% accurate.

[Anonymous Coward]

Re: Awesome

That is how the IBM RedBooks operate. No marketing materials here. This is serious, military grade stuff.

I think the rise of the cloud providers gives IBM Power with Linux a new chance to succeed. The problem for Power Linux was always the ISVs. "Does it support Oracle? No. Does it support MSFT? No.... Thanks for coming in." Google doesn't care, at all, if it supports legacy DB and OS vendors. They going to write their own, next gen versions of all of the above. Google probably also doesn't want the IBM case and would rather just take a license for the IP and build their own servers... but IBM will take that deal.

[Daniel von Asmuth]

Does it support Manchester United?

It runs Linux, so a ton of Open Source applications should be able to run on it.

I just wonder what this NVLink is for. Pacal GPU? You mean this is Playstation Blue that will play Angry Birds with you?

[Schultz]

Alternatives to: THIS PAGE INTENTIONALLY LEFT BLANK

THIS PAGE INTENTIONALLY not LEFT BLANK

Are you reading this? Don't read this!

Move on, nothing to see here

Your kids can draw here

This page is just a waste of paper and the this text is just a waste of ink

[The Indomitable Gall]

THIS PAGE LEFT INTENTIONALLY BLANK

You'll also see it at times on pages that are after the end of the document, and just included because standard binding is based on sheets that each account for four pages.

Anyone who took school exams in Scotland will be more than familiar with "this page intentionally blank" (or whatever), cos just imagine the consternation if that wasn't explained to you in an exam, and you spend the rest of the summer worried you'd missed a question or hadn't written enough.

[grumpyoldeyore]

Re: Awesome

I once put "This Page Left Blank by the Photocopier" into a large design document to see if any of the reviewers queried it.....

[Michael H.F. Wilkinson]

Very, very interesting

Memory bandwidth and latency often kill performance in many applications, particularly the bottleneck between GPU memory and main memory. This kind of boost would really help

[Ken 16]

Real time trading and some scientific modelling maybe

Most people will be OK with commodity hardware, I don't see it taking off big. It's nice to have another outlet for Power PC CPU though.

[razorfishsl]

Re: Real time trading and some scientific modelling maybe

Ah yes..... Power Pc

that complete dog turd of a CPU that Apple stupidly used in their products.

[chris 17]

Re: Real time trading and some scientific modelling maybe

It did and has found huge traction in the embedded market. Switches, routers, Mars rovers satellites etc.

You may not buy a pc with it in but other high performance appliances will have it in.

[John Sanders]

Re: Real time trading and some scientific modelling maybe

Essentially relegated to the status of glorified micro-controller that doesn't run as hot as a x86, because exotic things can't match the rhythm of the x86 industry.

Eventually if ARM is not stupid and keeps up they will erode into the PowerPC niche of moderately powerful CPU that doesn't get too hot.

This is an old story.

[asdf]

nice try

>x86 killer

That would be ARM responsible for the mass cull of layoffs at Intel. This just like all IBM kit these days will serve a very specific niche and though with nice margins for IBM hardly will affect x86 volume.

[Anonymous Coward]

Yes, but how much does it actually cost?

For at least the past 20 years, every time IBM released a new generation of Power systems I would salivate over their bandwidth, floating-point units, cache sizes, and generally meticulous engineering. Every time, it would also come with the promise of the TCO being lower than a comparable commodity solutions.

A few times, I was sufficiently impressed to get a quote or two - at which point I invariably came to my senses and realized that none of these systems were even remotely in the price range usable for the garden-variety small-scale HPC, and were not intended for the likes of me. Most of the time, my entire acquisition and maintenance budget (which does allow us to operate a quite nice, if not spectacular commodity facility and keep it updated with new hardware) won't even cover an entry-level support contract for one of these beasts.

Until I see the actual upfront acquisition and maintenance contract prices for these systems, I will have to assume that the TCO and TCA claims are simply the marketing speak. A.k.a. a lie.

[asdf]

Re: Yes, but how much does it actually cost?

> Yes, but how much does it actually cost?

Oldest saying in the luxury market but still true, If you have to ask you can't afford it.

[Dan Olds]

Re: Yes, but how much does it actually cost?

I know what you mean about their pricing in the past, but they showed some significantly lower pricing in their analyst deck and some pretty good compares to HP/Dell TCA and TCO. I think they're willing to take much lower hardware margins in order to make as big a dent in the market as possible. But the proof is in the pudding...and street pricing.

[Anonymous Coward]

Re: Yes, but how much does it actually cost?

https://www.ibm.com/marketplace/cloud/big-data-infrastructure/us/en-us

[Anonymous Coward]

Re: Yes, but how much does it actually cost?

https://www.ibm.com/marketplace/cloud/big-data-infrastructure/us/en-us

Thank you for the link. This is indeed a very good price for a power-based system.

However, to put things in proper perspective, the price of the "essential" model is not too different from what we pay for a supermicro twin2^2 system, which has *four* dual-Xeon nodes in the same rack space, each with specs not too different from the "essential" power system. Each of the Xeon cores is slower, but there are twice as many; the aggregate memory bandwidth to both sockets is actually slightly higher in the DP Xeon system than in the single-socket power box. Overall, one might expect noticeably better price/performance ratio for the twin2 box.

Of course, as always in HPC YMMV. At least these prices are of the same order of magnitude as commodity boxes - so for some workloads a power-based system may turn out to be a win. That's a nice change.

[asdf]

Re: Yes, but how much does it actually cost?

Yes but can you buy one without having to talk to a sales drone? That is when you know an IBM system is trying to go more mainstream.

[ben_myers]

Re: Yes, but how much does it actually cost?

When can I buy one on eBay? I have a lot of PayPal dollars to spend.

[Lusty]

Re: Yes, but how much does it actually cost?

Never compare list prices in IT. The price on the web is for competitors to see, not customers to pay.

[Skoorb]

Can this do things that the new Unisys clearpath x86 kit can do though, like hard partitioning (LPARs) and other traditionally "big iron" features?

[Peter Gathercole]

Haven't checked, but...

If this is a member of the Power family, even though it won't (yet?) run AIX or IBM i, it will be LPARable like all the rest (assuming you buy the entitlement).

You've been able to LPAR Power servers, even the Linux only ones, for longer than Unisys have been using Intel processor in Clearpath servers.

IBM pretty much wrote the book on partitioning servers.

[stewart-ibm]

Re: Haven't checked, but...

It won't run AIX or IBMi as these require the PowerVM hypervisor which is available on other POWER systems from IBM.

This box instead runs OPAL firmware and Linux runs in hypervisor mode, with Linux having complete control of the hardware.

I gave a talk on the firmware stack and how it fits together at linux.conf.au 2016 (link: https://www.youtube.com/watch?v=a4XGvssR-ag ) and you can head over to https://github.com/open-power/op-build/ and compile firmware for this machine yourself (even cross compile from x86)

[stewart-ibm]

Does the processor in the machine support Logical Partitioning? Yes.

Is it the same as on POWER7 and earlier and the POWER8 systems that run AIX/IBMi/Linux? No.

It's to do with the firmware stack that's being run. The -L and -LC boxes run a firmware stack called OPAL, the OpenPower Abstraction Layer, where Linux runs in Hypervisor mode and you can use KVM to create virtual machines (using the LPAR functionality of the chip). Paul Mackerras gave a great talk at linux.conf.au 2015 on KVM on POWER that's up on youtube here: https://www.youtube.com/watch?v=GJWe8dUADXg

For non-L non-LC machines, they run PowerVM, the IBM proprietary hypervisor. This enables those machines to run AIX, IBMi and Linux (either one whole machine partition or combinations thereof). In this setup, Linux runs as a guest of the PowerVM hypervisor. PowerVM and Linux on bare metal (running KVM or not) serve different markets.

Now... can you run a guest on the S822L for HPC? Sure, all the bits are there. However, there are limitations (e.g. GPU passthrough) on that and it's not the target market for *this* machine. This machine is designed for HPC workloads that use a lot of GPU.

Hope this helps,

Stewart Smith

OPAL Architect, IBM

(Usual disclaimer: I don't speak for IBM)

[Christian Berger]

We've come a long way...

...since the Cray I which essentially consisted of 5 off the shelf ECL logic chips.

[Anonymous Coward]

Re: We've come a long way...

"since the Cray I which essentially consisted of 5 off the shelf ECL logic chips"

The Cray 1 wasn't a sea slug brain. It used lots of ICs, just of only a few types. In theory of course you can build any possible computer design with nothing but NOR (or NAND) gates and a storage element; the history of computer design has basically been about putting more and more of them in a single package.

ECL, lad? We used capacitors, diodes, resistors and 2N404 transistors. And lots and lots of interconnect wire, paxolin board and solder.

[Christian Berger]

Re: We've come a long way...

Sorry, should have said that I meant 5 different kind of chips. I thought it was obvious that, given the low density of ECL you couldn't have things like single chip microcontrollers in ECL.

For those not fluent in IC designs, ECL essentially works by using transistors not as "fully switched" switches, but as amplifiers. So (in a nutshell) a one is one transistor having a higher output current than another, and a zero is the other transistor having a higher output current. Since both transistors will let current through, those chips burn _lots_ of power. However you can get them running at many gigahertz easily.

Burning that much power is what makes ECL rather useless for general purpose computers. It's hard to get that much heat away from those chips if you pack them densely. However you need to pack them densely to not have long transmission lines in between which introduce a delay into the computation. However there are specialist applications which are not general purpose where you can simply have a "pipeline" of stages processing some data. ECL is rather suited for this if you have a layout with controlled impedance and controlled line lengths.

[Measurer]

Re: We've come a long way...

Going back to my mid 90's HNC in electronics....

ECL = Emitter Coupled Logic?

[Destroy All Monsters]

Re: We've come a long way...

Yes that. Burns power like there is no tomorrow but basically fast.

No longer the economically sweet spot for supercomputing today...

[Anonymous Coward]

Re: We've come a long way...

I know what you meant but this is the Internet, give snark the smallest opportunity and it will get in.

There were 8 bit ECL bit slice components, I once researched a possible design for some hand held military equipment which basically powered up the compute unit for a millisecond each time it was needed and would have run off lithium batteries, while a CMOS microcontroller sequenced everything. The bit slice cpu did around 100MIPS, and so was able to do rather a lot of computing in the short time available. Not much use using a low power i²l design if it was still thinking when the bad thing happened that we were trying to prevent.

Owing to a minor cockup in my past I was very aware of the cost of Fluorinert, so when I saw my first Cray in the flesh what really awed me wasn't the sixties-sci-fi look of the hardware, but seeing so much Fluorinert in one place.

[razorfishsl]

Re: We've come a long way...

Er no.......

ECL is STILL the fastest non-exotic logic product in the world.

[pyite]

NVidia? How many FPS

If it can give me 120 fps on my 4k monitor in WoW: Legion, I'll take one!

[Ashley_Pomeroy]

I've been waiting ages for the G5 PowerBook - is there any news?

[Runilwzlb]

space for up to 16 drives (96TB total)

Back in days-of-yore when I worked at [Huge Japanese Manufacturer's Name Withheld] IBM OS/MVS shop 'we' were so proud of having almost 1 TERA BYTE (total) of spinning rust; fitted in six (I think) refrigerator-sized cabinets to run the empire on. How things have changed.

[Christian Berger]

A friend of mine...

...has a late 1980s digital VCR changer. Since it stores uncompressed video on 250 (up to) one hour tapes, the whole thing could have a storage capacity of up to about 3 Terabytes.

[Malcolm Weir]

Re: A friend of mine...

It's not an ACR 225 is it, by any chance?

[Nate Amsden]

HP beat em already?

IBM is comparing to HP DL360 when Apollo is the big data solution from HP. I'm not in this space but a quick search came up with

"The HPE ProLiant XL270d Gen9 Accelerator Tray provides up to 56 Tflops of single precision performance per server with eight NVIDIA® Tesla M40, and up to 15 Tflops of double precision performance with the NVIDIA® Tesla K80 GPUs and two Intel® Xeon® E5-2600 v4 processors in a 2U server."

https://www.hpe.com/h20195/v2/GetDocument.aspx?docname=c05069270&doctype=quickspecs&doclang=EN_US&searchquery=&cc=us&lc=en

(technically it is a 4U enclosure that looks to house 2 x 2U servers)

I am not sure if they can fit 8 K80s in a 2U package (vs 4 in IBM?) or if it is 8 M40s and a lesser number of K80s. Nvidia says K80 does up to 2.9Tflops of double precision so I would assume they have roughly 8 K80s in a 2U package.

Though HP is using PCIe I think and not NVlink.

[Peter Gathercole]

Re: HP beat em already?

Remember that IBM has produced some of the densest servers already with the 9125 F2C Power7 775, aka p7IH (the 'failed' Blue Waters machine, that still got sold to serious HPC shops).

When Power 8 came along, with the drop of the GX++ bus, there was not an p8IH, because of the redesign work necessary for the Torrent hardware (AFAIK), but the last time I spoke with people in the know, they were intending to take a shot at a p9IH server for HPC, but that is no guarantee that a product will be see the light of day.

You've also got to look at the density of the BlueGene/Q, now pretty old, but still scattered through the Top500 list, and peaking at position 4 in the June list.

[DrBandwidth]

Don't forget Xeon Phi x200 (Knights Landing)

Mainstream Xeon processors have lower peak bandwidth than the Power8 discussed here, but I routinely sustain >400 GB/s from the 16 GiB MCDRAM memory on my Xeon Phi 7250 systems. The best numbers I have seen are just under 490 GB/s, but that takes a bit of extra tweaking. STREAM Triad gets >470 GB/s with no unusual fiddling required (Flat-Quadrant mode, transparent huge pages enabled, compiled for AVX-512, run with 68 OpenMP threads and launched with "numactl --membind=1").

The Power8 has immensely more memory capacity than the 16 GiB of MCDRAM on the Xeon Phi 7250, but only a subset of jobs need both huge bandwidth and huge capacity. Some of these can use x86 via scalable systems like the SGI^H^H^H HPE UV systems. There are other high-bandwidth, huge-capacity solutions as well -- the Oracle T5-8 from 2013 delivered over 640 GB/s from an 8-socket server with 4TiB capacity.

[Anonymous Coward]

Re: Don't forget Xeon Phi x200 (Knights Landing)

routinely sustain >400 GB/s from the 16 GiB MCDRAM memory on my Xeon Phi 7250 systems

Don't forget that Knights Landing also has six DDR4 memory channels on-chip, so that with the right memory they can have pretty decent main memory bandwidth. The datasheet gives 115MB/s, but that's the theoretical speed-of-light kind of performance; I would be curious as to what STREAM Triad numbers are for the main memory (my bet is on 90+MB/s).

S812LC red book lists 170MB/s peak bandwidth for an optimal memory configuration (and only 85 MB/s for the small-memory configurations) - so Knights Landing does not look too bad here either, considering that it sells for a small fraction of the price.

[SeanC4S]

I'd like to run denoising autoencoders on it, but no one will even pay for a raspberry pi zero.

[Grunchy]

Isn't Intel just gonna put out some new iteration of itanium then?

Intel tit for IBM tat.

Something like that.

[John Savard]

But, but

The NEC SX-ACE system uses four-core processor chips that have 64 GB/second of memory bandwidth per core, so that's 256 GB/second, still ahead of IBM's 230 GB/second. Of course, that's a high-end supercomputer, not a mere server.

[oldcoder]

If I had $6,000 lying around for a new server, I WOULD get one. Even the smallest one outperforms anything with Intel with the same number of cores by a wide margin - and is smaller as well.

[Christopher E. Stith]

TCO on the server is great and all, but where's the workstation?

Great TCO on a server is one thing. Where's the fully compatible, if a bit slower, $200 impulse buy desktop? Where's the $400 low-end laptop? Where is the $3000 professional workstation version? One of the reasons x86_64 is a server hit is because code for it can be banged together and sanity tested on any number of other, cheaper systems. The final performance tweaks might need access to actual server hardware.

With lots of development in Python, Java, Clojure, Perl, Ruby, et al being a Linux box is nearly sufficient. Even then, though, if there's no C or C++ library to call from those languages then you'll be porting those libs or coding around them, and spending a bunch on programmer salaries in the process. If IBM wants to make IBM great the way IBM previously made IBM great, they need to intentionally do what they did accidentally the first time: align a desktop architecture with what they want to sell in entry-to-mid tier racked servers.

[Anonymous Coward]

Re: TCO on the server is great and all, but where's the workstation?

You really can't do that anymore. The hardware may be of the same class, but the similarities end there. Many server innovations in the Intel/AMD architecture have been kept in the servers because they're the only machines that can really make use of them. The article notes various high-speed buses in use that aren't going to be on the desktop anytime soon. Server-class GPUs that are tuned differently from even gamer-class GPUs. Multi-core multi-socket setups. Specialized memory sticks to handle things like buffering or power-failure-tolerance.

It's practically comparing a graphics artist to a construction worker. They may both be human, but their skill sets are entirely different.