Xmas software update knackered US Customs computer systems

If you were trying to enter the US on Monday, queues were much longer than usual. That's because a Christmas software update borked the main computer systems used by US Customs and Border Protection (CBP). On Monday, the main passenger checking system used by CBP to check for undesirables trying to get into the Land of the …

[Anonymous Coward]

Unfortunately...

Because the U.S. is so large, a lot of people get processed at major arrival airports then get domestic flights on to their final destination. Long delays getting through customs mean a lot of people missed their connections :(

Probably want to run updates like this through the test & dev environment a little longer before going live in production.

[Yet Another Anonymous coward]

Re: Unfortunately...

Plus a lot more who don't want to ever go to the US but have to because the US don't understand in-transit

[wsm]

Re: Unfortunately...

Must have taken a lesson from Microsoft. Production, what's that? Users are our property and we know what's best for them.

Seriously, roll-outs of anything new, system-wide, during a holiday rush?

[Pascal Monett]

@wsm

Agree with you on principle, but honestly I don't know if there is any down-time at major airports anywhere in the world, so one day is technically probably just as bad as any other.

For me, if I were one of those developers, I would have postponed the upgrade to after XMas on the basis that I would have wanted to avoid a chance of being called in during my own vacation.

[anothercynic]

Re: @wsm

Oh, there *is* downtime... it's just not when CBP staff would be around to do the updates! You don't expect federal employees to be present between 1am and 5am, do you?

[Anonymous Coward]

Re: one day is technically probably just as bad as any other

All the companies I have been working for lately are familiar with the concept of the "change freeze". No big changes around the end of the year or other business critical time periods. No changes to live production environment unless there is a failure with business impact. All smaller failures remain frozen, and are to be fixed in January. The type of the business does not matter much - telco, banking, retail, government, utilities, etc. All services facing big portions of the populace.

</Captain Obvious>

[Anonymous Coward]

Re: @wsm

yeah agreed that's why we have a "read only" Friday we never do critical stuff on a Friday as we don't want to be working at a weekend if breaks!

[Herby]

To err is human...

To really foul up things, you need a computer!!

Old saying that applies here. Then there is "testing, yeah, we heard of that (somewhere)".

This reminds me of the hours long wait for passport control at the Moscow airport. They had one guy (that took his time) for a plane load of us nice tourists (who want to spend good money). As the line got pretty small, (a dozen or so out of a plane load or two), they added more passport inspectors. Real understanding of queueing theory folks!!

[redpawn]

Welcome to Amerika

Just a taste of delays to come. If you want important documents delivered securely, use a courier not a computer.

[Adrian 4]

Re: Welcome to Amerika

Won't the courier be stuck in Customs ?

[Anonymous Coward]

Re: Welcome to Amerika

No, he means a US Robotics Courier Modem, as in; "Let's ship these files like we did back in 1992! With a couple of slow-ass modems and a POTS phone line!"

https://en.wikipedia.org/wiki/USRobotics

[Anonymous Coward]

Re: Welcome to Amerika

Just the courier has good chances of being an underpaid Afghan or Syrian... and delivering a bomb in a email is quite harder.

Anyway, once most will be forbidden to enter USA (and many will stay away as much as they can once "hate the foreigners" will reach its apex), custom queues won't be an issue even without computers.

[Daniel B.]

Re: Welcome to Amerika

Some commenters probably didn't get the courier reference you're talking about. ;)

[redpawn]

Re: Welcome to Amerika

Trump- "You know, if you have something really important, write it out and have it delivered by courier, the old-fashioned way. Because I'll tell you what: No computer is safe,"

[elDog]

I thought MS had stopped pushing updates for the ICE Windows XP systems

Maybe the customs systems are still on Win97 - in which case pardon my gaff.

Or perhaps Drump has already started building is wall - virtually for now. Keep them outsiders out ---- unless their (they're) cute little puddies.

[kain preacher]

Re: I thought MS had stopped pushing updates for the ICE Windows XP systems

A little birdie told me it was the software not the os.

[Mikel]

Re: I thought MS had stopped pushing updates for the ICE Windows XP systems

Everybody knows the OS isn't software.

[x 7]

So Trump's Mexican-detecting software got uploaded early? All the stories about a borked update are a smokescreen. This is the future reality in the United States of Trumpton. If you don't fit Dickhead Donalds vision of a New America then you're going to get barred/banned/sent to Guantanamo

[Dr Scrum Master]

People are Goods

I was wondering why a computer problem with Customs would affect so many travellers, but it seems that people are goods so they don't go through Immigration. (Are the visa fees now called Duty?)

[Daniel B.]

Re: People are Goods

In the US, the part where people go through is also called Customs.

[Malcolm Weir]

Nope, your terminology is wrong

The update was to the systems used by the Customs and Border Protection (CBP) staff. This is one of the post 9/11 changes that actually makes sense: there is just one organization handling both the Border Protection (i.e. Immigration) stuff and the Customs (i.e. stuff) stuff.

20 years ago I came across one airport that actually made sense in the way they handled the border formalities: you got off the plane and went to baggage reclaim for your bags; once you have your bags, you went to immigration, and once they passed you, you went out via the customs inspection. If they decided to hold you back at immigration, you had your baggage with you, and the random pace of baggage claim metered the flow of people to immigration.

This was at SFO, incidentally (which has since reverted to the usual immigration/baggage claim/customs routine).

[Yet Another Anonymous coward]

Re: Nope, your terminology is wrong

But with that system how were people able to just walk in off the street and take your luggage while you waited at immigration ?

[Dan 55]

Re: Nope, your terminology is wrong

Why would baggage claim be open to people walking in off the street?

[IglooDude]

Re: Nope, your terminology is wrong

That's a great question, to which many US airport designers apparently do not have a coherent answer. My local airport is a good example, where the baggage claim belts are less than a hundred meters from the doors to the "arrivals road" sidewalk, with no apparent access controls and no one even occasionally spot-checking luggage to retrieving-person.

[qwertyuiop]

Re: Nope, your terminology is wrong

I haven't been through Atlanta in a few years now, but it used to be that baggage reclaim was a publicly accessible area due to the weird layout of the gates/immigration/customs/etc.

[Daniel B.]

Re: Nope, your terminology is wrong

I can attest that most US airports have baggage claim in publicly accessible areas. It's weird for obvious reasons you've noticed, like any J Random dude being able to nick someone else's baggage with impunity.

The one place where this doesn't happen is when going through CBP. Thus your baggage is actually safer when you're arriving into the US than during domestic or connecting flights.

[a_yank_lurker]

Timing?

Who was the sub 1 watt bulb that thought a major systems upgrade during a major US travel season was a good idea? The period from Christmas to New Years is a major holiday period with many traveling. Any major screw up would be noticed immediately.

[Yet Another Anonymous coward]

Re: Timing?

Best time to roll out a new system, when everybody is out of the office - otherwise it can interrupt vital meetings

[theblackhand]

Re: Timing?

I thought (i.e. based on passenger numbers through airports) that travelling between Christmas Day and the New Year was a quiet period compared to a normal working week?

Heathrow stats show December as the second quietest month of the year and traditionally long haul flights in December/January are cheapest between Christmas and New Year.

[Paul Stimpson]

Change freeze

My customers all have change freezes in place from the middle of December until at least the end of the first week in January. Similar, shorter change freezes cover the Easter break. No system or network changes, other than emergency fault remediation, are allowed during these freezes.

Who on earth does major change work on a mission-critical production system during a period when the availability of your own people may be restricted and it's likely that your vendor's A-team (or possibly anyone) won't be available in their support department to help you pick up the pieces if it all goes horribly wrong?

[doug_bostrom]

""Travelers at some ports of entry experienced longer than usual wait times, and CBP officers worked to process travelers as quickly as possible while maintaining the highest levels of security."

Was there a noticeable difference? Arriving at (major west coast airport) during high summer to find a sphincter consisting of exactly 4 officers handling entering international passengers leaves one to wonder how much more of an incompetent face we can present. Arriving in the US consisently conveys the impression we'd just opened borders yesterday, with visitors being an unfamiliar novelty. Visiting Russian in 1971 was less of a CF.

What's also puzzling is why the system includes so many steps -before- a dead-eyed officer glances at the passport. What is the point of the machine passport scanning and biometric antics before one is granted a sneer? Private-sector contractor parasite efficiencies dictate the theatrics in part, but are those scanning terminals actually connected to anything?

[Richard 12]

The terminals are supposed to be Immigration

Then the person is supposed to be Customs.

Except that several brands of those the terminals have a 90-99% failure-to-decide rate.

This isn't surprising.

[Andrew Moore]

I had a friend who regularly did business in East German back in the 70s/80s- he said Checkpoint Charlie was a doddle compared to US Border and Immigration.

[Death_Ninja]

Go on, lets out the culprit contractors

And your winner is... Unisys

http://federalnewsradio.com/technology/2013/07/dhs-awards-460-million-it-contract-to-enhance-border-management/

I'd forgotten those dinosaurs still existed until now.

[Anonymous Coward]

This is another example why you should join Global Entry if you could

During the past 15 years, the general immigration queues at all major US international airports have gradually became such a nightmare that I won't even consider transiting through the US any more, and am extremely leery of visiting. You simply never know how long it will take you to get through the customs: it could be half an hour or it could be half a day, with absolutely no facilities (not even water or toilets in many places) available while waiting in the standing queue.

If you are eligible at all, I strongly advise joining Global Entry or, if you are a Canadian, NEXUS. Even if you cross that border just once a year, it is still worth the cost of the application fee: you get to bypass the general immigration queue completely; in many airports, you also get to bypass the general security queue (which is yet another nightmarish thing about flying in/to the US).

Unfortunately for myself, I am no longer able to maintain my NEXUS membership: even though as a Canadian expat I am technically eligible to join NEXUS, I would likely have to show up at a Canadian port of entry for an inteview/fingerprinting - and that would be a little too expensive to do from where I am based now. Perhaps if I manage to time it to coincide with my next visit back ... that's worth thinking about.

[anothercynic]

Re: This is another example why you should join Global Entry if you could

You *do* realise that Global Entry is only available to a *very* limited list of countries and none but five of the VWP countries, right?

And the kicker for UK citizens is that you must apply via GOV.UK (and pay £42 for the privilege *on top* of the $100 fee to the US)... No refunds for *any* failed applications either. Nice.

[Anonymous Coward]

Re: This is another example why you should join Global Entry if you could

Yes, I do realize it is an option not open to many people - this is exactly why I did include the link to the eligibility criteria. And yes, it is relatively expensive for a casual traveler.

However, if you need to travel to the US while maintaining relative sanity, it is the only game in town. Unless, of course, you are a part of the 0.1% which goes through the general aviation terminal after arriving on their private charter jet.

[Anonymous Coward]

Normally updating over the holidays makes sense

Like if you were updating your company's SAP system - few users at the time, so there is more time to work on problems before they become critical (of course it sucks for the IT people working over the holidays, but that's another story)

But there is MORE travel over the holidays, so this is probably the worst time they could have chosen for this!

[Captain Badmouth]

this was not a malicious attack

Are they sure it wasn't Santa's elves trying to smuggle toys from Mexico?