Masked thieves storm into Chicago colocation (again!) The recent armed robbery of a Chicago-based co-location facility has customers hopping mad after learning it was at least the fourth forced intrusion in two years. They want to know how C I Host, an operator that vaunts the security of its data centers, could allow the same one to be penetrated so many times. "I can't believe …
This is one of the best examples of how IT is continually loosing touch with the "non-virtual" world. Techies get hung up on the technical merits of a particular product and forget that any technical aspect can be overcome/undone by a single person applying "real" physical force.
Once general IT security issues are resolved the focus will return to real security, not some technical crap. Security is about guns and ammo, not firewalls and OS. Mark my words, the IT world is not prepared for "real" attacks. Too many pizzas and nachos have rendered IT professionals worthless in regards to practical security.
http://img81.imageshack.us/img81/4881/report1part1vx1.jpg
http://img81.imageshack.us/img81/8002/report1part2kw8.jpg
http://img222.imageshack.us/img222/4590/report2part1bm4.jpg
http://img86.imageshack.us/img86/8008/report2part2jo8.jpg
http://img86.imageshack.us/img86/6286/report3part1zx1.jpg
http://img81.imageshack.us/img81/3007/report3part2wo8.jpg
Hell the solution after the first robbery should of led to shotgun wielding rent-a-pigs. Those of us who have grown up brooding on cyberpunk-esk ideals know the only "safe" data or hardware is that which is guarded both physically and electronically.
After the fourth robbery and no armed guards... the company deserves to flop for their incompetence.
Those Police Reports make interesting reading. Like the way the alarms are rarely triggered. Or the way the burglars usually don't seem armed. Just stroll in and grab some kit. Those reports make it sound like this "secured location" is abandoned in the evening, with maybe someone living in a flat above. LOL!!!
It looks like it is more convenient to nip down to this "secure host" to help yourself to new hardware than it is going to the shops or EBay. Maybe I should put in an order... could do with a new server. :)
"at least two masked intruders entered the suite after cutting into the reinforced walls with a power saw... ...During the robbery, C I Host's night manager was repeatedly tazered and struck with a blunt instrument."
How long does it take to cut through a 'reinforced wall' with a power saw? Wouldn't that be noisy? Why didn't the soon-to-be-tazered-and-battered night manager wander over to the nearest telephone and call the police? 'Crikey, there's someone cutting through my reinforced wall with a power saw. Fourth time. Send the plods.' Was he asleep AND wearing a walkman with noise cancelling headphones?
Doesn't make sense.
Firstly, to those that doubt tasers are good, I am quite sure the guard and any family members are happy he was tasered and not SHOT. Those of you in the UK just don't seem to understand that in the US, that is the frequent alternative. With handguns available cheaply and easily on the streets to any 14 year old, I am rather glad to see that these perps at least bothered to avoid serious jail time if caught and spring for the less-lethal tasers. I'll bet the guard is too.
LESSON #1: PUT YOUR CO-LO IN A SAFE AREA. Deserted downtown city centers? Bad neighborhoods? Hmmm, maybe not...
LESSON #2: WTF have only 1 guard who has to leave the freaking secure room to use the toilet??? DUH!
LESSON #3: Underground is a great place for a co-lo...XYZ Airlines (a major US airline who I will not piss off by naming) has theirs under their main headquarters car park, under several feet of re-inforced concrete to stop truck bombs. With one secure freight elevator that is totally non-descript and VERY sucure, and walkway entrances that are difficult to remove equipment by..oh, and 3-4 staff in the secure ops room all day and night. They are not armed, but I am fairly sure the glass cage is bulletproofed...
LESSON #4: After three previous break-ins, realize that there is probably an insider giving away information...
and those pictures must be really old when they first builty out the space.
When i went to the center
I had to be buzzed in at the street level and buzzed in at the suite doors.
Once inside the guy wrote my name down (never looked at my ID)
showed me where my servers were and left me be.
there were times i was left alone for well over 30 min free to roam the datacenter all alone. I did see one smart customer who rented a cage and had a rack or 2 in there and had multiple cameras of their own... he must have been aware of the past breakins... I wish we were aware prior to putting our boxes there.
and a sledgehammer or two, it would take you 5-10 minutes to make a hole big enough for a person to fit through, depending on the density of the concrete.
The white noise "hum" of all the servers would actually do a lot to dampen the sound of the centre being broken into, from the inside at least.
I live within walking distance of where the break in occurred...
First, the city of Chicago has the toughest anti-gun laws currently on the books of any major city. (Washington DC's laws were overturned) So it would be illegal for a citizen to have a fire arm. Since this is a "hosting center" and not a "7-11", I doubt that they could get a permit to allow an employee to have access to a gun. Not to mention the insurance premiums would be sky high.
Second, if you look on google maps, the street is where Franklin dead ends and border's Moody's Bible institute. (Its the athletic center, with Walter Payton High just north...)
With Cabrini Green going down, not a real high crime area. There are Chicago Police cameras in the area, but I don't think there is one in that block. (You can't miss them. they all have flashing blue lights to warn you that they are there....)
Third, this was on the third floor of a building. I don't believe that there is a fire escape on the building which means that in order to gain access to the building, you'd either need a key or would have to be buzzed in. (Can you say "food delivery"?)
Fourth, what is meant by "re-enforced wall" ? That could mean a lot of things. But it clearly isn't a bank vault wall. So you can cut through the wall pretty quickly. Sure its noisy, but it could be that the "night manager" was outside of the server "vault" and was already incapacitated before they tried to gain access.
Fifth, you can't easily move a hosting center. So once you put one in, you're pretty much stuck with it. As to why Chicago? There are too many reasons to list, but its a great place to be. ;-)
Having said all that, this was clearly a planned and targeted attack.
The hosting company definitely cut a lot of corners, but if a crook is determined enough to get something, they will get it.
You could have a cage around your equipment, but if they're going to cut through walls, do you really think that they couldn't take out a cage quickly?
IMHO, they could have bought a building in/near that neighborhood, do a rehab and have a much more secure environment....
I was totally shocked when delivering some equipment to Easynet's colocation in Reading last weekend to find the rear exit propped open with a bin.
I strolled right into the datacenter area and delivered the equipment. Fair enough but had there been others around at the time, they could have easily have forced entry and taken whatever they wanted.
There appeared to be no security at that exit at that time. I'd think they might have learnt from the thefts at their Brick Lane datacenter last year.
As apparently only the second person who's actually BEEN to that data center to comment (and I also live about five blocks from it), and maybe the only one who's been there in the past month (as a customer, not as the criminals :-) ), here are my comments:
1) The area is quiet, but not deserted. The building has a restaurant in the first floor that's open until at least 11. There are also two late-night restaurants (4am I think) that are popular with cabbies within a block, and within two blocks south, east and north are densely populated residential blocks.
2) I'm not sure what is meant by a "reinforced wall," but it's definitely not reinforced concrete. It may have steel mesh, but the wall between the public hall and the cages is not concrete. If it was just steel mesh, like is used as cage dividers, it would take a circular saw with a "cuts anything" blade no more than 2 minutes to create a hole big enough for a man to get through. I'll admit that, as creepy as this coincidence is, I actually wondered if people could come through the wall the last time I was there.
3) Have any of you been to ANY datacenter? They're LOUD, like the ultimate white noise generator, and this one is no exception. The main office is near the entrance and along the east exterior wall, but the wall between the datacenter and the public hallway is at least 50 feet long, with multiple (insecure) walls, rows of computers and power equipment between the office and the wall they likely came in through (I don't know exactly which wall they came in through for sure, but some seem more obvious than others).
4) I don't know which night manager was on duty, but the guy I've dealt with the last three times I was there always checked my ID while in the foyer between two locked doors (it wasn't a man-trap, but it was two separate locked doors). He also had a prosthetic leg, which could very well have caused him to be subjected to more antagonism by the intruders if he was the one on duty.
5) All that said, the last time I was there, I was removing my equipment (for reasons unrelated to the hosting company), so I'm feeling pretty damn lucky right now.
6) I agree the worst thing is that the company denied and failed to notify for days afterwards. Yeah, it sucks to have to tell your customers something like that, but it has to be done and delaying is just gonna make things worse.
Those cables aren't so bad, at least they're labeled... I did some work for the Board of Education recently, not in some school but their main site. Hundreds of cables terminating in seemingly random ports WITHOUT labels, or site documentation. I had to spend hours just figuring out what was plugged in where...
How crap is their network, or rather, how drunk is their sysadmin?
http://www.cihost.com/images/about/virtual-tour/cdc01/network/DSC_0279.JPG
And most of their practices make it nice and easy to nick kit or screw up their network.
- no rackmounting (the ole' stack it in the rack ploy)
- what kit that can be loosely referred to as being rackmounted is a joke
- zero cable management (not withstanding the big ball of fibre)
- nice "radial reciprocating air conditioner" (um, the black office fan)
What a shit-hole, to be polite.
http://www.cihost.com/images/about/virtual-tour/cdc01/staff/DSC_0601.JPG
A good tazering is what this lot needs:
http://www.cihost.com/images/about/virtual-tour/cdc01/staff/DSC_0601.JPG
And finally:
http://www.cihost.com/images/about/virtual-tour/cdc01/servers/DSC_0220.JPG
Nice "servers".
If my facilities looked like that, I'd kill the sysadmin responsible. That's of course if I hadn't already wired my nipples to a tazer for taking photos of said facilities and posting them on the corporate website.
On the page that describes the "Chicago Data Center, Building Spec, Physical Structure" is listed as 10 000 sq ft.
If I rent one of the 100' x 100' Private Data Vaults per the available colocation spaces listed, that is also 10 000 sq ft, so they would not have enough space left over for a 1U machine. Much less any of the other spaces that are mentioned for rent. Or space for an office, a 24x7 guard, 24x7 on-site technicians, another 100' x 100' Data Vault, or anything along that line.
One of the features they mention is "Constant n+1 ambient air temperature of 68° (+/- 5°)", huh ? They also have "Internal and external cooling units", so they must cool Chicago, Wow!
100% Service Level Agreement - impressive, very impressive!
Or under building structure they have - "Line of Sight: Clearance of the roof is to the East, all other directions are blocked by surrounding buildings", but if you are looking for roof space they have: " Available for tenant's mechanical and technological communication needs; building has a clear line of sight in all directions", nice trick. (and are smoke signals a mechanical communication need?)
But if you need it they do offer, as a "Business Convenience ... Internet connectivity" always a good feature in a data center.
I also note that the Manhattan, NY, facility, which is not in Manhattan, or New York, for that matter,(notes below) is "our newest state-of-the-art data center", for ease of access they offer, not some claustrophobic elevator, but instead, a "single flight of stairs with no turns". And when you talk about any facility in "New York", you simply CAN NOT say, that it is " built to ensure ... continuous operations - even in the event of severe... man-made catastrophes", it is just bad for your credibility.
Other than that and a number of similar examples, the place looks good. But if you post things like the above of your website, your credibility for any other claim, just can't carry much weight.
Manhattan Notes:
The Manhattan data center is listed as being in New Jersey,but it is touted as being located 8 miles from the heart of Manhattan. Manhattan consists of at least 6 islands of 'habitable' size, and a hunk of 'mainland'.
The Statue of Liberty is on an island in Manhattan.
Part of Ellis Island is in Manhattan also, it is also part of the Census location of, New York, New York. The 'natural' part of Ellis Island is in Manhattan in in New York. The landfilled part of Ellis Island is in New Jersey. The fill was done on all sides of the natural island, so, at this place, New Jersey completely surrounds New York. Sorry I digress. This was intended to highlight the lack of knowledge that C I Host shows about their location.
Chicago specs from: http://www.cihost.com/about/data-centers/specifications/cdc03.php
New York specs from:
http://www.cihost.com/about/data-centers/specifications/cdc04.php
*******
Many, many years ago in my stoodie days, I went out-of-hours to a supermarket in Reading to do some work on their servers. The security was tight! Vibration alarms in the walls, armoured shutters on all entrances, cameras, infra-red beams, time-locks on doors not supposed to be opened before the morning, retractable steel pillars in the loading bay entrance to stop ram-raiders. And this was in a relatively quiet area of town. I was amazed to find out the place had been successfully robbed twice in a year. Each time, all the alarms had worked, but the Police had taken too long getting there and the thieves had made off with the safe on a flatbed! After the second time, the Police changed their local patrol pattern, started parking a patrol car in the carpark between patrols, and they weren't touched again. Maybe the Chicago Police need to do more....
Whilst we're naming and shaming, I had the misfortune of being given a 'tour' (loosely described as,) of the then Redbus Sovereign (now Telecity). Before I went, I was told security was tight - I'd need a token number, photo ID and business ID. Fine, 45mins of calls, and I have a token number, so I trek off to the centre.
I stand in reception for 20mins, I tell the desk chappy that I was there for a tour, and before I could mention anything about ID's and tokens, he buzzed me through. Didn't even take my name...
So, he takes me round, although I'd mentally switched off to using them already, and shows me one of the suites. No shit, he takes me into a private suite. I ask about the rack dimensions and clearance/cable questions, so back on the colo floor, he finds the nearest cabinet, uses his master key and pops it open pushing cables out the way to point at the rack brackets.
On the way out - still in a "secure" corridor (so now only a door away from the colo floor) I ask if I can use the toilet. He points the way, then tells me to head out when I'm finished. Rrrrrright. Anyway, I get stuck in the secure corridor for 10mins, until a suite chappy from a floor above asks if I'm okay. I mumble something to the meaning of "the dear fellow at reception was kind enough to give me an excellent tour of the centre, and the dear chap forgot I didn't have a swipe card." Slightly phased by my abundant use of the English language, he let me out using his card.
To let armed robbers saw through your wall, you're either incompetent, or bent. Or both. Or work for Redbus.
To all those carping at the poor quality of CI Host's infrastructure, cabling, servers, and more... well they are by no means unique in this respect. I've seen a lot of "datacentres" and a lot of hosting companies are as bad as CI Host, or worse. Even the ones that aren't stupid enough to put shitty pictures on their web site but put up photos of other people's racks in other people's datacentres can be just as bad.
For some reason the yanks seem to be much more likely to run their operations like this. Brit web hosting companies seem to go for good-quality kit in a half-decent datacentre like Telehouse or Rebus (ok maybe Redbus isn't half-decent but it's better than some geek's bedroom). I think it's because in the UK the local loops for high-bandwidth links were always obscenely expensive, leading the hosting companies to go to the Docklands datacentres, whereas the Merkins could get OC3s into any old office for not too much money.
Monkeys, peanuts, etc etc.
http://www.cihost.com/images/about/virtual-tour/cdc01/facility/DSC_0299.JPG
There seems to be a number of wall mart type fans throughout their photos which wouldn't fill me with confidence of hosting here! That along with the appauling cable management down all the rows that are prone to being kicked and pulled by the 'Crash Trolley' and people wandering connected to nothing less than extension blocks purchased from your local radio shack.
Are we sure this isn't just someones shed in the back garden?
... Doesn't look like a data center to me. Looks like some warehouse space with PCs shoved on shelves. But, that said, I used to have my server housed in a space like this, but at least the space was secure. It was in a secure business park with security guards (armed). The building also was secure, with armed guards, and finally, it would take ages to get into the damn room because only certain people had access. And the cabling looked better too.
Eventually I pulled my server out of there, not because of the service I got, but because of circumstance.
I'd probably prefer one of those white-painted white-everything rackmounted rooms next time round.
That was the sound of the company shooting itself in the foot with those photographs of its business. Crumbs. In some of the off photographs you can read their paperwork - that must be bad form, surely? The row of servers looks like one of those recycling depots where old PCs go to be "reborn", like in Logan's Run.
When I read this...I had to shake my head in disbelief...How do you get broken into that many times with that much security....if you ask me....way too many thing's here do not add up...this in my opinion has all the hallmarks of an inside job or someone on the inside supplying the info for the people doing this heist...all that security and they never heard or saw anything??? c'mon ...sorry but power saws to cut thru meatl, concrete, stucco etc...make a lot of noise...and smoke...surely this should have been heard...if not...it should have been picked up by their secuirty or firedection system ( as most have smoke detectors that have a photelectric element that pick up smoke wafting in the air ) if this place has motion detectors it should have tripped as would any glass sensors due to the amount of noise they possibly would have advertanatly tripped due to some of the sheer noise being put out...and how can an employee not see this with all the surveillance cameras there...most comapnies will have several monitors around so employess can see who is around their building ...before anyone can get in...and with all the card access and magentic door locks...this makes getting into certain areas within the premise tough as well....in my mind this has all the hallmarks of an inside job...I have been in the security industry for close to 20 years and this screams inside job to me...too many coincedences to ignore in my mind
Someone mentioned that security is about firearms, not OS/Firewalls. Well, actually its both. Working in a bank will open both things up for you; they usually have the best example to set on physical security. Even IT Security folks like me don't have physical access to the datacentre! Even the access door is bulletproof, about 3" thick, and armed guards. Take in mind also that this is *inside the building*. Even getting into the building itself would be a feat requiring James Bond-esque skills.
So basically, physical security ain't a problem for us. We got to worry more about IT, because that is something that can't be protected by twelve-gauges. ;)
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2025
