2 more degrees and it's lights out: Mercedes-Benz Grand Prix's toasty mobile bit barn

There's something reassuring about the techies that support a championship-winning Formula 1 team resorting to basic cooling methods to prevent their mobile data centre from melting. Matt Harris, head of IT at Mercedes-Benz Grand Prix Ltd, gave a few insights into his life at the business, which recently picked up its sixth …

[Anonymous Coward]

I'll never really understand...

Some years ago, one of my best friends worked at an F1 team - not so successful these days, but back then they were winning championships. He did some kind of sofftware engineering for the systems in the cars.

I will never forget the look of incredulity on my wifes face (she was a big F1 fan) when my friend revealed that he had been offered the 'mobile IT' opportunity for some very big races that year but had turned it down> in favour of staying at home.

I mean, lost bragging rights or wot?

[Pascal Monett]

Re: I'll never really understand...

I have a hard time understanding that as well, but I no nothing about either the conditions he was living in, who is he like and what made him make that decision.

Unless he was just lazy, I'm guessing it was a difficult decision to make. He knew what he was turning down after all.

[Anonymous Coward]

Re: I'll never really understand...

At the time he was very much young and single and it did seem to us outsiders like the sort of opportunity you shouldn't pass up if you are intending to make a career in the industry. He still writes software for vehicles but in a much less high-pressure world with (presumably) fewer foreign trip opportunities. He's not a particularly open type but he seems happy enough. Some people are stuck in low-paid dead-end jobs.

[Intractable Potsherd]

Re: I'll never really understand...

"... he seems happy enough. Some people are stuck in low-paid dead-end jobs."

Some of us prefer to be left alone to get on and do the job we are paid for, not endlessly chasing the next "career goal". Let's put it this way - which path is most likely to lead to burnout and dissatisfaction? I've done both, and, whilst I miss the travel in some ways (e.g. to have a good excuse to escape the children occasionally, if nothing else!*), I really don't miss the endless reviews and bloody interviews that come with chasing the next level of distance between what I'm good at and what the post requires. Good luck to those who want to do that, but it isn't for me, or your friend.

* A simplified life of work, food, read, sleep, even in a hotel room I didn't choose, sounds very attractive at the moment :-)

[KittenHuffer]

Re: I'll never really understand...

In the past (same era as OP) I had a colleague whose husband w**ked for one of the teams, at a facility in the South of England. And tickets were offered to various events in the calendar, with their availability inversely proportional to their distance from the UK.

[big_D]

Re: I'll never really understand...

It depends on his personal circumstances and also which countries he was offered... For the actual back-office staff on site, it isn't so glamorous as being in the pits or sitting and watching the race. All the time the cars are being prepped or are on the track, you are probably stuck in a hot and sticky trailer, making sure everything is working. No going to the hospitality and saying "page me if something goes wrong".

I was offered the choice of 2 travel opportunities in the early 90s, IT support for the team running the elections in war-torn Angola or war-torn Bosnia... I somehow managed to lose my passport.

The guys that did go to Angola were accused by the rebels of collusion with the regime and there was a running gun-battle to the airport to evacuate them... I'm so glad I lost my passport.

[A Non e-mouse]

Re: I'll never really understand...

I've read interviews with a couple of different team members about life during a racing season. The common theme was that it's not a fun or glamorous world. Very long hours, totaling several months away from home, family & friends.

[Boris the Cockroach]

Re: I'll never really understand...

Which is pretty much the story I've heard from a couple of people who went to work for F1 .

Only relaxing time (unless you're pit lane crew) is during the actual race itself. rest of the time you're stuck in the garage wrestling with the spanners...

And dont even think of working in the supply chain, "we make F1 parts" sounds so glamorous when going for that dream job.

Not so fun when a F1 team phone up at 4pm wanting a new design of gearbox housing made, and you know its going to take 8 hrs of solid programming to turn the model into machine code, then 20 hrs of machine time to turn the code into metal..... but the F1 team want it on the plane to god knows where at 5am the next day....

Glad I'm out of that really....

[Anonymous Coward]

Re: I'll never really understand...

Yes, a friend in the aero department and one in the tech department of a certain F1 team which may or may not provide wings, found there to be a lack of respect for your private life and very poor pay.

[trevorde]

Re: I'll never really understand...

Many years ago, I worked with a bloke whose brother worked in the wind tunnel for the Jordan F1 team. They were on a bonus of £1000 for every point the team got. Unfortunately, that year the team didn't score a single f*&%$g point!

[commonsense]

Re: I'll never really understand...

Jordan scored points in every season they competed.

[tony trolle]

Re: I'll never really understand...

92 was their worst year and that was 1 point.

92 second worst 3 points.

[I ain't Spartacus]

Re: I'll never really understand...

Business travel is over-rated. You've always got some place to be - and so you're rarely doing anything interesting. And while eating out is nice with friends, it's nice because it's a change from your normal everyday dining. When you're away for 5 days each "race weekend" (i.e. a week) - 21 times a year (23 this season) - then the charm wears off rapidly.

Plus hotel rooms are just too small - not that F1 crew probably spend much time in theirs - they've got to unpack and re-assemble and test 2 cars, plus setting up servers and the pit area - then take it all down again afterwards and re-pack it. Plus fix the car overnight, if the driver is so careless as to put it into a wall at 200mph, or if the engine decides to blow up of its own accord.

Then a few times a season they have back-to-back races in say Singapore and China - where they never go home, but fly all the kit out to one, then move straight to the other. So they're actually away from home for 2 weeks.

I'm sure it's great fun if you love it, but if you don't it probably gets really fucking annoying, really quickly. Not to mention all the time away from the family.

[big_D]

Re: I'll never really understand...

So true. I haven't worked in F1, but I did work as a consultant for many years and I would often spend 6 nights a week in hotels for 18 months at a stretch. The lack of "simple" food, never being in your "own" bed, not near your friends (what friends? You spend you whole life living out of a suitcase in strange towns, you are only ever at "home" to do the laundry, mow the lawn, pay the bills and then bugger back off to the next hotel room), it gets you down after a while.

[I ain't Spartacus]

Re: I'll never really understand...

Also, hotels seem to have a no comfy chairs policy. They often have lovely comfortable chairs in their bars and nice airy reception areas. But the ones in the rooms always seem to be horrible. Even in posh hotels where you get a desk with a decent upright chair, so you can do some work in your room, the sofa will be rock hard or have no depth so you always feel like you're perched on it.

This means you can't just sit in your room and read, unless you like sitting on the bed. And unless in a posh big room, there's often not the room between sofa and bed to stretch your legs out.

So you're living in the equivalent of your bedroom at home, with mostly the sort of things you have there - and no access to the stuff that might be in your sitting room. And to make it worse hotel TVs seem to be set up for viewing from lying in the bed - which almost nobody actually does by choice.

Plus fiddly little kettles and no easy way to wash stuff up - means that in-room tea making facilities are an annoying faff, rather than a pleasant convenience. And snacks means popping to the shops in a place you don't know, or horrible mini-bar prices.

Sorry, do I sound bitter? I'll shut up now.

[J. Cook]

Re: I'll never really understand...

No sir, you described it perfectly.

I worked for a multi-national ISP back in 2000 to 2002, and it was interesting. Good pay and benefits, decent co-workers, and the change to play around with some leading edge tech. The flip side to that was getting woken up at 1 am, having to drive to the office, grab spare hardware, and hop on the next available plane to BFE to swap a failed line card in a remote site.

Sure, I traveled the country, but the only thing I saw were the insides of data centers, which TBH, no one cares about until it all falls over. :) I did manage on a couple occasions to sneak in some free time on two of my trips to play tourist, which was interesting in an of itself. (Notably, I toured NASA's Houston space center, which was pretty cool. It included a guided tour of some of the back areas, one of which was a full scale mockup of the KIBO module for the ISS.)

[Anonymous Coward]

Re: I'll never really understand...

I was offered a job by the head of a small(ish) company. When I quickly turned it down, the head asked why: I said I didn't want to spend my life living out of a suitcase.

A few weeks later, I was recalling this conversation with one of the company's engineers, who happened to be onsite. He said he'd lost two girlfriends from being on the road all the time and only ever saw his flat on a Saturday night when he was home to do the laundry, ready to go away again Sunday afternoon.

Sure, I could have earned a shed load more money working for that company, but what's the point of earning all that dosh if you never have any time to spend in - either yourself or with friends or someone special?

I don't own a fancy car. I don't live in a mansion. But I do have a happy life.

[Ol'Peculier]

I bet one of the countries he wouldn't mention is Azerbaijan, I went there for the race last year and EE was tempting me with 5MB of roaming data for £36.

Anyway, I love reading about the stuff that goes on behind the scenes in F1, so more of the same please! I do remember a few years ago Red Bull were advertising for an IT position, with the added bonus of moonlighting as the rear jack man as well.

[KittenHuffer]

If he was 'moonlighting' would he have to use a psueonym for <cough> tax purposes <cough>?

If he was a big guy then he could always go by Huge Jack-Man!

[Caver_Dave]

F1 Jobs

I produced the first F1 in-car telemetry system many years ago as well as selling lap-timing equipment to the F1 teams (and many other formulas). Everybody had two or three jobs. For all the teams I can remember, the guy who worked my lap-timing equipment, was also the gearbox man and a truck driver.

[werdsmith]

Re: F1 Jobs

I worked on telemetry for two of the teams quite a few years back. It was actually one team but it got a new owner and name. I got to go to testing sessions, but no races. By then, telemetry was well established, but doubtless very basic compared to today. It stored a lap of data and then attempted to transfer it in a burst past the pits. It didn't always work.

[I ain't Spartacus]

I think the F1 teams do trials for the pit crew. You don't need to be a skilled mechanic after all, just quick, dextrous and precise. So they select from the people who could be in the travelling team anyway - and have another job - and then they do the pit crewing as an added bit of work. Which requires lots of extra practice throughout the year. Redbull nailed 3 stops that were under 2 seconds this season! You don't get that without days and days of running drills - critiquing them at a ludicrously detailed level of precision - and then doing it all over again.

[druck]

I was lucky enough to be at the front of the FIA garage watching Redbull do practice pit stops next door during winter testing in 2018. You really can't believe your eyes how quick 14 people can move in unison.

[The Quiet One]

This has always intrigued me, would love to see more content on the IT Side of a travelling motorsport team and the F1 grid clearly has the budget to make it interesting.

The lack of A/C is an odd one, I'm surprised they can't get efficient portable systems that can be rigged on on-site but i guess working to the outdoor temps they don't normally struggle.

At Silverstone they probably need a 2 bar electric fire to keep things running properly ;)

[big_D]

HP kit is also fairly tough.

We had a "computer room" (old office on the top floor, south facing, floor to ceiling windows and no AC). The first thing I did when I got there was recommend AC, the price was too high and the CEO told me that they had never had any problems. I also suggested putting the servers in the cellar, but that was nixed as the mirrored SQL server was down there...

I put a thermometer in the middle of the rack. In the summer, we were getting temperatures in excess of 60°C and most of the kit kept running. I think the highest temperature I saw was around 68°C, the room temperature was over 40°C most days. The first person into the office opened the windows to allow some fresh air in.

One of the servers crapped out, a 7 year old HP ProLiant server, it was so full of black dust that the fans had to work overtime in the winter... I did manage to get budget after that to get a man with a compressor in and we blew out the dust from all of the servers in the rack. They ran reliably after that, despite 60+°C temperatures. I left shortly thereafter, my successor did manage to get the replacement CEO to re-locate the servers to the cellar.

[Anonymous Coward]

I worked in the cellar

Ambient went up to 28C

The most crucial bit of kit - and INTEL 8088 ICE stopped working.

It was full of ECL and Schottky TTL and timing was critical.

We asked for cooling. Refused.

Then I read te manual

"Equipment operation not guaranteed above 27°C ambient"

Showed it to boss, who grabbed it and raced upstairs...

Aircon arrived the next week.

[SImon Hobson]

We had a "computer room" (old office on the top floor, south facing, floor to ceiling windows and no AC).

You've almost described our server room at my last job - the one I rebuilt, wired properly, but wasn't given any budget for aircon (and we probably wouldn't have got permission from the landlord). We did have forced air cooling - one fan sucking "cool" air in, another blowing hot air out. Except that the inlet and outlet were above a tin roof - so in calm summer conditions, the combination of heating from the roof, and pooling of the hot air we'd blown oout meant that the intake air wasn't all that cool. I also estimated that we needed at least twice the airflow - which meant that as the temperatures went up, the servers sucked more air than we were supplying and so that sucked hot air from the back of the racks round to the front - increasing the temperatures further, so the servers ran up the fans more, so ...

One summer it got really hot, and we took a punt on a second hand "swamp cooler" (we call them evaporative coolers here in teh UK, but the merkin version sounds better in some ways). It sat in the front of our server room, and we filled it twice a day - IIRC it could get through something like 40-50 gallons some days !

It certainly did the trick, though the front inch or two of all the server metalwork went a little rusty.

And the kit (mostly) had to fend for itself over the weekends.

[Anonymous Coward]

"We had a "computer room" (old office on the top floor, south facing, floor to ceiling windows and no AC).

In 1970 we had a network of comms nodes round the UK based on M2140 industrial computers. The one in Manchester would randomly go down in the daytime - and then come back before the engineer arrived.

One day the root cause was finally identified.

It was housed in a small room with large windows. The cabinet doors were often left open after scheduled maintenance. If the sun shone long enough at a particular time then it illuminated an exposed rack of cards. There was one faulty small IC that distorted slightly in the extra heat - breaking one of its pin contacts inside the package. When it cooled - the contact was re-established ok.

[Cederic]

The lack of aircon is farcicial.

"What if we depended on it and it broke?" Then stick dry ice into a bucket of water. That shouldn't be Plan Bloody A.

[big_D]

Yes and no. If you can design the system to work with ambient temperatures with direct cooling from, at most, dry ice through blowers, then you don't have to "waste" space on additional AC units, which is extra weight and extra cost - although AC units seem to be fine for the drivers, sitting in the pits or the paddock...

It also means it is one less thing that can fail during the race. You don't want half your pit crew disappearing, because they double up as IT support, because the AC in the data container is on the fritz and you aren't getting telemetry and can't communicate with the driver.

[Cederic]

So the retain the 'we can run hot with airflow' design. That's good design, I like that.

But if you've got a container with three racks of servers, two racks of UPS and a rack of networking kit, a spot of aircon isn't going to make much difference to space and weight. Shit, don't use a container, just use a refridgerated lorry.

How do I spell refridgerated?

[Anonymous Coward]

A big truck might work for the European races where the teams travel by road, but not the fly away races that make up half of the calendar.

[NeilPost]

A standard Cargo-plane sized/shaped container fitted out seems idea.

[I ain't Spartacus]

although AC units seem to be fine for the drivers, sitting in the pits or the paddock...

The driver is important. They've got to do 10-20 corners a lap, for 50-70 laps without making a major misjudgement in changing conditions and while dodging (or better passing) the other drivers on the track. Sometimes at 200mph and at some considerable risk to their lives. In an ideal world they won't even make many minor misjudgements - because that means they go faster and put less stress on their tyres.

Compare in-car footage of Lewis Hamilton to most of the other drivers, and notice how many fewer inputs he makes on the steering wheel mid-corner. This is because he is one of the top 3 F1 drivers of all time and is extraordinarly good at this. He's turning the wheel once at the beginning of many corners knowing where the car will be at the end of the corner, so he doesn't have to adjust the steering later to correct it. And he's consistently better than most other drivers at doing this. This means his tyres wear out less quickly, so he can either use that saved rubber to go faster, or to go longer without changing them. To concentrate like that for 2 hours, in a hot uncomfortable car while being distracted by all the stuff that goes on around him means that he needs to be delivered to the car fresh at the start of the race. Calm, cool and collected. And it's easier for your driver to do that mental preparation if he's relaxed and not hot and sweaty. Or in the case of James Hunt - if he's just had a few bottles of champagne and page 3 girls in his bed all night... In an ideal world you'd control everything and have it all perfect - but where that's difficult or impossible you have to prioritise. Which means drivers and cars first.

Also some of that air-con is provided in the racetrack's buildings - which you don't have to ship everywhere with you. Plus every power requirement that you add, is more power you've got to get trackside (and might not be able to) - or run generators for.

[NeilPost]

I’m a little surprised the narrative about days to set it up and days to pack away. Surely having a fitted out Data Centre in a container with appropriate cooling that could be put on the back of a container lorry would have been path of least resistance.

Sounds like an opportunity for AWS Outposts ???

[Starace]

Overselling it

That 'mini dstacentre' would be what, half a racks worth? Maybe a whole rack if you really wanted to push it and have lots of spares. Hardly extreme.

And you don't treat the possibility of no aircon by having no aircon; you have aircon, you have backup aircon, then you have contingency cooling. You don't just skip to the contingency straight away.

Not a big shock though, often the team budget for these things is actually derisory. And no one actually wants the jobs as the pay and conditions are rubbish as the expectation is that 'working in F1' is a reward in itself. Hint; after the first week it isn't.

[Martin an gof]

Re: Overselling it

you don't treat the possibility of no aircon by having no aircon

Why not? If you can build an appropriate solution without resorting to forced cooling, that's an awful lot of saved cost, weight and potential problems. A bit like insulating your house. Do it properly and you can get by in a typical British year with ambient heating (incidental gains from sunlight, body heat, appliances) in the winter and ambient cooling in the summer.

A typical British house however needs dozens of kilowatts of heat for six or nine months of the year.

Think about graphics cards without fans, underclocked processors, SSDs instead of spinning rust, 'industrial spec' (or even military spec) components instead of 'commercial'.

No idea if any of these techniques are used (loss of computing power comes to mind) but if you can get by passively, why not do it?

M.

[A Non e-mouse]

Re: Overselling it

Was it Google who buy specially rated processors from Intel that can run at much higher temperatures so they can run their data centers hotter and hence pay less for cooling?

[Pascal Monett]

Yeah, but they still have cooling.

[Jo_seph_B]

Weight, its all about the weight!

I've worked on a bid for a current F1 team for factory and track solution. We lost out partially as they wanted the kit for free, or near enough with random kick backs and corporate deals. Honestly it was basically selling IT to a marketing team with the solution being the secondary concern.

The second reason we lost out was weight. Turns out adding only a few U of extra hardware adds quite a lot of cost in shipping weight over the whole season. It was crazy money for even an extra 30-50KG of kit. Our solution had some options no one else's did but as such weighed more.

The design was simple enough. 6 Node solution in the garage and a backup 4 node solution in the engineers truck.

Regardless of the outcome I loved working with them. Would still love to work in F1 but Its a young mans game these days and It'd be bloody hard to be away from home for such long periods through the year.

[Roland6]

Re: Weight, its all about the weight!

Clearly, it is about something other than real portability/mobility...

"We take about 36 hours to put it together, we run for five days, and we expect it to be absolutely perfect.

...

The bit barn in a box takes six hours to pack away before it is shipped to another country:"

Don't know about the experience of others here, but the building of drop-ship (ie. portable) data centres was largely solved in the mid 1990's. So clearly from these assembly and teardown times, they aren't using a solution actually designed for their environment.

[druck]

Re: Weight, its all about the weight!

It isn't just a case of plugging a containerised systems in to mains and network, the environment is different at every one of the 21 races. The building and tear down involves running all the wiring from the servers in a trailer in the paddock in to the racks of laptops and monitors in the pit garages, and hooking in to all the FIA network which provides the telemetry. The pits are grotty bare concrete boxes, all the nice shiny white plastic interiors are you see on race weekends, are constructed by the teams.

[Roland6]

Re: Weight, its all about the weight!

>The building and tear down involves running all the wiring from the servers in a trailer in the paddock in to the racks of laptops and monitors in the pit garages, and hooking in to all the FIA network which provides the telemetry.

Agree that is all part of the job and will take time, however the article context was clearly referring to just the datacenter component of this.

[I ain't Spartacus]

Re: Weight, its all about the weight!

Roland6,

All the F1 teams are based in Europe. So they freight their stuff by road to the European races - but by air when they go further afield. Which makes things conisderably more expensive of course - and makes weight and size much more of an issue.

I believe they usually arrive at the track on Tuesday or Wednesday and have to be fully up-and-running for Friday and Saturday practice - and then race on Sunday. But in reality many services will need to be working on Thursday as that's when the marketing circus arrives - and entertaining their corporate sponsors' guests is what funds the teams. Then they pack up Sunday and Monday - and on back-to-back race weekends it's off to the next track and do it all again.

Anybody who thinks this is easy hasn't thought about it much.

[Ian K]

Overly Server-tastic?

"...we take a data centre that supports a hundred or so users as domain controllers, SQL servers, file servers, Wi-Fi, Ethernet gigabit to all clients, around 200 devices, 100 or so virtualized servers."

That's more or less one server per user/2 devices - compartmentalising functionality is all well and good, but shurely a bit OTT?

[I ain't Spartacus]

Re: Overly Server-tastic?

I suspect that the team limit of 60 does not include the marketing people. It will come as no surprise to you that there are a lot of marketing people involved in F1.

Bear in mind that each team is going to have at least 10 sponsors - and probably much more than that. And all of them have paid the big money so they can take clients to races and wine and dine them. And seeing as they're paying all this cash, they want access to the team, hand-holding, explanations of all the shiny stuff (especially if they're the kind of sponsor whose kit the team actually use) - so that takes a big marketing team. In the case of Mercedes of course, Mercedes themselves will be using the team in their sales drives.

My mate worked for CA in their McLaren sponsoring days - when McLaren built the 2-seater F1 car for taking sponsors out for a joyride. Getting taken round Silverstone at 200mph by a top racing driver is the sort of jolly that even directors of top companies just can't do themselves and so it's a salesman's dream as a pressie to dangle in front of them. It doesn't matter how rich you were, as only two teams ever built those cars (if I remember correctly), and all the rides went to the big sponsors. So it's the kind of experience that only a few hundred people ever got.

[werdsmith]

Re: Overly Server-tastic?

"The first person to experience McLaren’s two-seater? The great Murray Walker, who in 1998 was given the ride of his life by TV co-commentator Martin Brundle. He first needed a two-hour medical.

Treadmills, X-rays, a seat fitting at Woking: then unleashed at Silverstone. “So I’m sitting in the back trussed up like a chicken,” remembers Murray. “Watching the apex at Abbey get closer and closer at an alarming speed with no reduction of revs from the screaming V10 which was massaging my back and I think, ‘Ah well, I’ve had a good life!"

--from Mclaren website.

[I ain't Spartacus]

Re: Overly Server-tastic?

As I recall McLaren did that car as a one-off, to be sent to the museum after one season. And the sponsors begged them to keep it going, because it was such a brilliant sales tool. Like everything else in F1 - it cost a bloody fortune.

The memory banks kicked in and tell me the other team to do it were Minardi. But it was a century ago, so I could well be wrong...

[tea junkie]

Re: Overly Server-tastic?

Your right, it was a minardi.

Theres video of Nigel Mansell binning it on youtube.

https://youtu.be/Og7v6mouFBE - mostly onboard with a very lucky person

https://youtu.be/wM4bIKf62j4 - the highlights of mansell hitting alonso

And of course brundle giving murray a drive in the mclaren

https://youtu.be/wU0gq7gUmWM