Vendor's secret 'fix' made critical app unusable during business hours

Welcome to another installment of On Call, The Register's Friday column that tries to improve the health of the tech support ecosystem by sharing readers' sickening stories of bringing broken tech back from the brink. This week, meet a reader we'll Regomize as "Raoul" who told us about his former life as a commercial Linux …

[SVD_NL]

Lost for words

The production database stored medical data, personal information, and handled payments had no access controls," he told On Call. "It was configured 'ALL ALL ALL', so any user on any system could access any database as any user.

I had to stop and reflect on that one for a second.

But seeing how the vendor is behaving, it wouldn't surprise me if they suddenly scrambled and implemented access control, in the process of that they break the app, and silently patch their errors without having anyone know.

[John Riddoch]

Re: Lost for words

I've seen numerous apps over the years where the "fix" for a permission denied error was "chmod -R 777 /app/directory". It doesn't overly surprise me, but it's awful, awful practice.

[b0llchit]

Re: Lost for words

Besides awful practice, it is a sign that they have no clue what they are doing and they should be avoided to touch your systems at all cost.

^{_{Only /tmp [and /var/tmp] require[s] 1777 (== 777 with the T-bit set).}}

[Christoph]

Re: Lost for words

Only /tmp [and /var/tmp] require[s] 1777 (== 777 with the T-bit set).

Do applications ever store confidential information in temporary files? I wonder if anyone's ever written something to scan such files?

[Roo]

Re: Lost for words

It's SOP at some sites to scan places like /tmp for sensitive data. There is a cost for doing so and it's going to miss stuff (and report false positives). Hackers do it too. :)

[logicalextreme]

Re: Lost for words

It irks me no end that I've now worked for (and been made redundant from) two healthcare software providers that thought they were doing God's own work WRT data security in scraping through the paltry audits that came their way, but at a previous (easy-going) employer we could regularly be in lockdown-investigative-battlestations mode because something that might have looked like a credit card number had popped up in a temp file or memory somewhere, overwhelmingly false positives.

PCI/DSS is, or was at the time, paranoid enough to get infosec at least partially correct. Healthcare in its current state from what I've seen is a free-for-all with a lot of sound and fury slapped on the top of it and labelled "secure".

[rskurat]

Re: Lost for words

well, lives or health might kinda-sorta matter, but if you really want action you have to risk money

[Claptrap314]

Re: Lost for words

In my experience, I'm not even sure it was labeled "secure".

Well, I guess UHC would probably be up to that level. Now. It certainly wasn't 2 1/2 years ago...

[jake]

Re: Lost for words

"Do applications ever store confidential information in temporary files?"

All the time, alas.

[xyz123]

Re: Lost for words

Both Oracle AND Servicenow store plaintext stuff in temporary files for "caching speed". basically if I got access to your PC, I can see every URL you visited AND the contents of that page using nothing but Notepad.

[doublelayer]

Re: Lost for words

At least you can create subdirectories which have a more limited scope, which is what I have done when software needs to store temporary data that something else shouldn't read. I doubt everyone has done the same.

[Alistair]

Re: Lost for words

I've configured environment TMP directory pointers on SO MANY MF@#$%@#$% third party "enterprise" apps in my career I have considered just filing security bugs on the vendors anonymously. Wether or NOT the data is confidential or not, it is a habit any *nix admin should engage. <And in many cases one needs to actually create a tmp folder for the app, in its own disk space>

[Yes Me]

Re: Lost for words

"Do applications ever store confidential information in temporary files?"

I notice that nobody answered that question, presumably because the answer is rather obvious.

[Anonymous Coward]

Re: Lost for words

Many years ago I was doing a migration when abroad. Sites had control of their kit, but there were some global standards for workstations (they ignored the OS upgrade to W8). Servers all had to be in English.

Doing the migration which for the servers involved a lot fo scripting to reconfigure from the sites old standard / settings to our global ones. A well tried and tested script

The workstations which were now in W8 and in a non English language were a bit confusing at first, but commands are commands and established the scripts that ran on the clients to move domains, sort users, remap etc were all good too.

This all started on a Friday night - was a bit slow, but nothing too unusual

Sunday night comes and at this stage we are pissed off for a couple of reasons and then one of the local team who is checking all the users devices and configs says "I cannot get this users home drive to come up". I have a look and sure enough, no mapping to their homedrive. i check the server, its there, all set to go.

I spend some time and am a bit bemused as I cannot see anything that would l cause this and is working for my account (admin) then I ask "Is anyone else affected?". His response "Oh yes, everyone" I am now opened mouthed and ask "When did you notice this?" and their response was "Since the first machines we checked on Friday night..."

I had not choice at that stage to grant them all local admin on this box to get access to their data - just because they had to work on the Monday.

Took a few days back and investigations to find out that the local IT had set some setting on the server (I cannot recall what, but it was obscure) that would cause this. Reset the key and then removed local admin and it was all good.

The easy win would to have just left it - as it is a local box so not as critical as they being admins for that city/country, but it pressed that it needed to be dealt with and could not be left as it was

[Mike Pellatt]

Re: Lost for words

Yeah, me too of course.

I find accounts package developers are the worst, going back to the early 1980s (in-house devs on an Olivetti S6000)

The most recent one was, of course, a Windows one who told me it absolutely had to run with Admin privs. Asked 'em why - "because it has to write to the registry".

Turned out to be just one key, so created an account for the app, changed the privs for the one key to that account, job done.

Unbelievable yet unsurprising. The usual cavalier attitude to security.

[GlenP]

Re: Lost for words

I find the worst are financial; institutions' security apps, some of which will simply not run without local admin rights and short of disassembling the code I've not found any way round that. Sadly it's impossible to refuse to allow them to be used.

[MnK72]

Re: Lost for words

I have seen enough applications where the DB username and password is the same (i.e. if app name is app001 then the username and password is both set to app001). Sad thing is that the password cannot be changed as it will break the application.

[parlei]

Re: Lost for words

There are almost certainly rules with the force of law that would make this a serious matter. Medical records are (a) sacrosanct and (b) not to be open to outsiders to read.

[Claptrap314]

Re: Lost for words

I'm reminded of the wisdom of Boss Hogg. "It only illegal if you get caught." Especially with health care, I would say that this is the dominant view--to the point of it being "industry standard".

[J.G.Harston]

Re: Lost for words

Microsoft's in-Windows CD writer software would decide that because a CDROM was a read-only media, that it should also set all the *file* attributes to read-only, destroying the metadata. When copy things back off the CDROM to their destinatin, to get things to work you'd often have to do a recursive attrib -r *.* on everything without knowing what really should really be read-only. I tried to distribute data and applications via ZIP files on CDROM, but go too many users who couldn't comprehend the concept of archives. So, I wrote a simple "installer" than unzipped the data to the correct destination, and then got fails from a couple of users who had something installed that made ZIP files look like directories to all filesystem calls, so it bombed out with "Cannot do file actions on a directory" sort of errors.

[Michael H.F. Wilkinson]

Re: Lost for words

A totally, gobsmackingly bad "design decision" (phrase used without prejudice) by the application vendors, and monumentally stupid decision by management to accept the vendor's "explanation".

The vendor had better not try this on any BOFH, or they might have an ... accident in the elevator, or a database normalisation warning (the latter seems more appropriate)

[disgruntled yank]

Re: Lost for words

Without telling tales out of school, I will just say that the notion of "least privilege" is lost on a lot of people. There are commercial, widely used platforms out there that appear to regard as standard the use of root-level credentials, of course saved in clear text.

[NetMage]

Re: Lost for words

Especially galling since least privilege is baked into HIPAA and any unauthorized access would undoubtedly be a violation of Federal Law - for the clinic.

[JKROBERTSON]

Re: Lost for words

And for those of us bad, sad, and dangerous enough to actually deploy such tools the quality of programming leaves a lot to be desired in terms of their function, stability, resiliency, and the expectation that they can be used in a virtualised desktop environment…

User profile specific configuration file encryption, so non-persistent VDIs can’t hold common approved settings.

Security prompts with no centralised/secure way to approve them resulting in repetitive user security warning click-through habits.

Screen recording failures through web browser add-ons regularly crash.

Session recording stored in a 300GB database alongside user/permissions/target host lists because they didn’t consider storing session capture in a different database.

No formal support for Microsoft AOAG, even if you have multiple data centres that use a shared Microsoft AD Domain, password changes are not unique to a specific site but must be known to users regardless of which site they log into so you have to able to reach the application data base even if one site drops offline.

The vendor thinks using a whole thread of a CPU, 100% of the time, is fine when you list available credentials.

Same vendor believes using an additional thread 100% of the time when selecting target hostname to login to is also acceptable.

Management you sign up.

SME’s don’t know the product enough to set it up correctly (“We’re the first to do this…. Uh-huh.)

Users hate it because the know you’ve taken their direct permission away, and put another tool in the path to doing their job.

The vendor just says “It’s not designed to work that way, we’ll have to raise a development note. No guarantees it will be changed/improved.”

The fallback is I have to deal with all of these levels on a daily basis. Sigh.

[J.G.Harston]

Re: Lost for words

Yes, I can't remember which of the UK GP clinical applications it was; but after installing I couldn't configure the printer settings, it had to be done by one of the users logged in with their keycard inserted.

[Timop]

Re: Lost for words

What a way to reduce the scope of a project, just skip implementation such not that critical parts like access control in system handling personal information.

I'd like to know how big the bonuses for PM were...

[GlenP]

Vendors...

the developers said that was the required config

We recently had an issue with a 3rd party integration vendor. They were involved in us uploading open receivables data from our SQL based finance system but their default requirement was a user with full access to the entire database - that wasn't going to happen! They then stated as an absolute fact that access had to be to the tables, not views, again not going to happen as there are multiple companies running on the same database and they were only involved with one of those.

Eventually they corrected themselves and said views would be fine but neither they nor the primary vendor could say what data they actually needed so we killed the project. It was frustrating as I could have sorted the whole thing out in a few hours (create views with the correct data subsets, provide a user with restricted access to those views, install the integration software) which would have taken far less time than the back-and-forth discussions.

[Phil O'Sophical]

Re: Vendors...

This makes me think of the far-too-many vendors whose installation instructions contain a note to the effect "if it doesn't work, turn off your security software and try again". They never seem to learn.

[Mishak]

Real common when SELinux is active

And no, I will not disable it globally.

[GeekyOldFart]

Re: Real common when SELinux is active

SELinux is a royal pain in the arse and is an ever bigger pain to manage, but even so there is but one excuse for disabling it - temporarily to prove that it's where you need to tweak $STUFF to make $THING work...

[l8gravely]

Re: Real common when SELinux is active

I hate SELinux with an undying passion because it makes a tough job impossible. Completely impossible. The docs suck. The tools suck. The examples suck. The mentality sucks.

I'd post examples, but I've burned them out of my brain.

[anothercynic]

Re: Real common when SELinux is active

I find it ridiculous that some Linux packages categorically don't set up SELinux contexts and rules for their application.

I have not had any issues with SELinux in the sense that I set up contexts correctly. When I have issues, yes, I have to sit there for absolute hours trying to find the cause, but SELinux stays *on* on our servers and we make sure that the applications run in user contexts and appropriate SELinux policies.

Any app developer (looking at you, Shibboleth) not bothering to provide SELinux contexts should be taken out back and... taught to do it right

[ecofeco]

Re: Vendors...

I've seen that far too many times.

[mw_foot]

Re: Vendors...

I can’t believe the amount of times on windows servers a vendor has slapped an Everyone, Full Control on the entire OS drive.

It baffles me that there isn’t an understanding of the service account being used and apply least privilege.

Don’t get me started on the “Disable the Windows Firewall” instructions.

[parlei]

Re: Vendors...

Currently I have one such vendor on my List. No, there is no reason your install suite needs to trigger our malware scans. Except incompetence on your part, but I hesitate to label that as a *valid* reason.

[Anonymous Coward]

Re: Vendors...

We had an integration from one product to an MS one.

When trying to get the config, the documentaiton says "admin". the customer of course was "no bloody way".

A call was arranged with the product vendor who went through why and showed their refrence - some MS Docs.

Next call was the same people and added MS. They still inisted it needed admin

The customer engineer mumbled and grumbled and a few days later came back with the "right" permissions for us to use and that worked.

Documentation now updated.

[Jou (Mxyzptlk)]

Re: Vendors...

This is why I gave up on "one big database server cluster for all". They get their own SQL, in their cell, and then i don't have to care about that.

[Mike Pellatt]

Re: Vendors...

And of course they now all need their own DB admin.

Or every DB is running sub-optimally.

Still, once it's all in the Datalake, who cares about the access controls?

[rnturn]

Re: Vendors...

Two examples of vendor boneheadedness:

A hospital management system upgrade that was performed by the vendor. I was required to be present in case they ran into a snag. The only interaction I had with them was when they asked which disk was the OS (VMS) on? I answered but wondered "Huh?". After they left I checked the system out and found that their upgrade had blown away all the account password parameters we'd set up for HIPAA compliance. Password lifetimes changed to 9999 days and crap like that. Nothing in the release notes mentioned any changes like that were to be made. While it was a simple fix via some DCL magic, I had a lengthy phone call about the user account changes with the vendor. Seems those changes were requested by another customer. We got 'em by accident I guess.

When we audited a back office application (again, VMS) for a bank trading system, we found that the users had been granted BYPASS (i.e., God) privilege and it was enabled by default. When we tried to remove that, users were unable to log out of the application. Another "feature": the application opened dozens of files during operation. There were logical names created for each file... all pointing to the same disk. You can imagine how that affected performance. (I/O queues were through the roof.) My boss let me change that by determining which files were the "hottest" and tweaking the logical names and distributing them across multiple disks on our test environment. The application ran like a bat outta hell. Faster than the production system despite the test system running on lesser hardware. Proposing the change to the vendor generated a response of "We won't support you if you do that". We came to the conclusion that they'd developed their application on a workstation with a single disk and because they, apparently, didn't understand VMS very well, set themselves with BYPASS privilege to avoid having to deal with the unpleasantness of proper file permissions. We had to get an official statement from the vendor that the BYPASS priv was mandatory in order to satisfy the bank's auditors.

[Claptrap314]

Re: Vendors...

If I were your auditor, I would make replacing that app mandatory.

[Lee D]

There is/was a major school MIS provider who provide a feature within their software which, when used, executes the given command as a plain SQL statement against the underlying main school database as a full administrative user.

I discovered this one day at random by being sat in someone's office while they were troubleshooting a minor pupil-data error in the database and they were instructed by the software support line to enter this menu, and I heard "DELETE * FROM" as it was read over the phone to them and they repeated it back as they typed.

I was horrified that their support staff were instructing otherwise ordinary users to type plain SQL direct into a dialog that was then EXECUTING... including, I found out when I dived over the table to take the phone from, things like dropping tables, changing schemas and removing rows.

And there were no logs of this, no controls on it, the support staff reading out the commands were oblivious to their actual danger (just reading off a script for a given problem), they were then getting users who knew nothing of SQL syntax to type this in over the phone, and there was no attempt to ensure backups had been taken (this was on-prem!) and no attempt to even enclose the statements in a transaction.

The next year we went cloud because... they can sort that out when it all goes horribly wrong, I'm not taking responsibility for it!

[ecofeco]

Sweet jumping Jesus.

How do those people even have jobs?

[Excused Boots]

Because, probably 99 times out of 100, just blindly following a script; works. Of course, when eventually it doesn't and their 'advice' takes out your production DB and possibly your company as well......

Sorry about that!

[rskurat]

nothin bayesian about it

lotta gamblers out there these days; the problem is they don't know it. "Trust, but verify" seems like a good idea but I don't know if IT audits are really a thing anymore.

[Jou (Mxyzptlk)]

Re: nothin bayesian about it

don't know which reality you live, but audits are more a thing than ever, and they will get more and more. You have no choice, 'cause no matter how much you put into creating a "procedure to follow", there is always a chance it isn't followed or the automatisation failed for some reason.

[AVR]

Because people who actually know SQL cost money, whereas people who can read off a script cost peanuts.

[Anonymous Coward]

Medical systems are a nightmare

I put in an electronic patient record for a hospital group about a decade ago. Vendor selection was "best of a bad lot" but the real problem was getting the decision makers to understand the issues.

"Why can't we use the same password for everyone, we always did before" (for a prescription system with hardwired terminals for 5 users kept in a locked room). And medical devices are worse.

I did eventually come to understand their logic. If we spend the money on increasing security it may prevent a breach sometime in the future. If we spend the money on extra clinical staff hours it will prevent a death in the next few days.

[Korev]

Re: Medical systems are a nightmare

I had an appointment with an anaesthetist a few years ago, whilst talking to him I noticed there was a username and (guessable) password for a computer system on the wall.

[Admiral Grace Hopper]

Re: Medical systems are a nightmare

I once went on a visit to a customer office to diagnose a problem in the wild. To fill the time I provided onsite support and training for the poor suckers lucky punters using our desktop software. It was the first time saw someone lift and invert their keyboard after being asked. "What is your password?".

[midgepad]

Re: Medical systems are a nightmare

Reduces keyboard problems from paperclips and fluff.

[Jou (Mxyzptlk)]

Re: Medical systems are a nightmare

Better a good password below the keyboard than a bad one. Just has to be out of sight.

[Caver_Dave]

Re: Medical systems are a nightmare

My company tried to get us to use a company wide password vault.

One on the early items in the training was how a wonderful feature in the vault allowed you to share passwords.

At which point anyone with sense dropped off the call.

Even the HR droid monitoring the call noticed this drop in attendees and everything has gone quiet on the compulsory use of the tool.

[Anonymous Coward]

Re: Medical systems are a nightmare

not sure that was understood correctly

2nd time I've had to use a password vault.

some times you have passwords of last resort that can't be easily changed.

guess what, you often have to share that password amongst people.

many jobs ago, we had to change that password quarterly or when ever someone left. a bit of a pain when you have thousands of systems where that password needed to be changed.

Some places reuse that same password across multiple systems. With a password vault that can also change that password then each system can have a unique password of last resort & that password can be shared on demand to all your team.

perhaps the purpose of the vault wasn't understood?

some password vaults also can do PAM so you may never need to know your admin password as the vault can rotate it & just log you in to your kit from the gui.

permits having a username & password & an admin-username & automatically rotated random password to login to kit & do work.

[Anonymous Coward]

Re: Medical systems are a nightmare

There aren't very many times where "my direct supervisor can use my password without telling me" is a good idea. In fact, I can hardly think of any good reasons for someone to have direct control of an employee's account.

[Giles C]

Re: Medical systems are a nightmare

A properly configured password vault will not have this problem. For a start there should be permissions and restrictions on how the passwords can be accessed. If a password is just for yourself to use then don’t give anyone else permission to access it, however If the system allows other users to override that then it must be logged and justified.

If a password is for a group of users who need it because of their role then the role gains access.

Regardless of this there should be a full audit trail to determine who and when has accessed the passwords. Using a password vault I.e. hashicorp or similar that makes it a lot easier to manage the password and only those tasked with updating the password need to be able to view it. Bearing in mind tha a vault is not the same as a manager.

[Prst. V.Jeltz]

Re: Medical systems are a nightmare

Dont service accounts need to have their passwords shared amongst a few people?

[jake]

Re: Medical systems are a nightmare

"Don't service accounts need to have their passwords shared amongst a few people?"

No. That's what groups are for.

[Jou (Mxyzptlk)]

Re: Medical systems are a nightmare

Microsoft developed several methods for not requiring a password for a service account (which, on top, does not rely on starting as superuser and then drop rights):

"Standalone Managed Service Account" (2008 R2). An account, bound to one server and one service.

"group Managed Service Account" (2012). An account which can be used among many servers for a service (lets the service account on those servers talk to each other).

"Delegated Managed Service Accounts" (2025). An account which can be used among specified, changeable, group servers for a service.

All hose accounts are a kind of hybrid between a user and computer account. Behaves like a user account, but things like "automatic password change every 30 days" is done automatically, AND you cannot set and see the password. And you cannot type the password since it would requires characters you cannot enter, like ESC, Enter, backspace and so on. Technically it is a binary password.

While these are all good, one thing is annoying (again): If you create a new AD with level 2008 R2, 2012 or 2025, why are the first steps to use those features done during creation? Similar to "Time based group membership", with AD Level 2016: Why not activate the feature right away? AD-trashcan with 2012: Why not activated right away? ***sigh***

Technically this is a bit off topic to the "share password" thing here, and these accounts have to be set up and managed by someone with an admin account and therefore a password.

EDIT: And why is the spooler still run as BUILTIN\SYSTEM?

[Claptrap314]

Re: Medical systems are a nightmare

In the Unix world, no. In part because you generally cannot log in as a service account at all. You log in as a user who has permission to either become the service user or to invoke apps/scripts which do.

[Anonymous Coward]

Re: Medical systems are a nightmare

Fully agree, have to use "shared ones" sometimes for work, creds autofilled in GUI so I do not know/ see the password* but can login to appropriate web applications with appropriate permissions (creds frequently change, e.g. when people leave, move role, in addition to short time based change so a given password never "live" for too long, so password manager essential)

* Yes, I obviously have the technical expertise to find out the password that gets pasted in (& someone with malicious intent could do that & have a "live" password for a day or so before it gets replaced if they so wished as no system is perfect). We are not supposed to know the password though (& trusted not to obtain it surreptitiously as that's a disciplinary)

[Giles C]

Re: Medical systems are a nightmare

We use a password vault all the time, mostly for things such as radius keys, snmp passwords, service account passwords. All those things you only need occasionally and it is better than writing them down.

Of course we share passwords for systems with over 2000 network devices just one person knowing the emergency password would be dangerous…

[Jou (Mxyzptlk)]

Re: Medical systems are a nightmare

A vault to "share passwords" is not wrong. You have a number of people, which then with their accounts can access the password they need for their role. Another group can access another set of passwords for their role. or in other IT words: Those 15 people are responsible for that customer, and NEED those passwords. The other 3000 of the company don't. Access is logged of course.

But I deduct from your words something was amiss? Please share more details.

[doublelayer]

Re: Medical systems are a nightmare

It is dangerous because anyone with access can copy that password and continue to use it and you can't lock them but not everyone else out by changing it. You can try to lock them out by changing the password and locking them out of the vault, and as long as that's what your planning, it's probably not too dangerous. If you ever locked someone out of a vault without changing every common password they have accessed in the past, then a step was missed which can lead to yet another of those "IT employee sabotages employer after termination" articles. Read the comments on one of the articles. See everyone calling the entirety of IT idiots? That's you in this scenario, which is why shared passwords are a danger that needs to have some plan for how to prevent it going wrong.

[Jou (Mxyzptlk)]

Re: Medical systems are a nightmare

That is a problem which cannot be solved. In other words: This is a human problem, not a technical.

There is always an instance where password(s) have to be shared to someone else, which requires a level of trust (and singed how-to-handle-passwords-according-to-government-standards). You can try to mitigate, but in the end there is always someone with enough access to cause damage. The password managers allow for a central level of control to mitigate unwanted spread of passwords, but those cannot prevent someone sharing them manually too, which boils again down to my first line of this answer.

[doublelayer]

Re: Medical systems are a nightmare

But there is often a way to create user-specific passwords for something which you can then revoke individually rather than needing to revoke the common password every time a single user isn't supposed to have access anymore. In my experience, most shared passwords don't get revoked when someone leaves, so unless you're already doing that, you've failed, and if you are doing that, there's likely an easier and better option. If you have individual accounts, there are many situations where you never have to have a shared password for anything.

[Claptrap314]

Re: Medical systems are a nightmare

I agree that this is a human problem. The technical solution is to grant the user access to what he needs to get the job done, not to create a flock of system accounts which can be logged in to.

Of course, with high enough privilege, and, as you say, someone must have this, it is possible to re-enable access once it has been lost. But now we're talking go to jail kind of behavior, and that's really pretty rare.

[Anonymous Coward]

Re: Medical systems are a nightmare

@caver_dave

My company tried to get us to use a company wide password vault.

One on the early items in the training was how a wonderful feature in the vault allowed you to share passwords.

At which point anyone with sense dropped off the call.

Even the HR droid monitoring the call noticed this drop in attendees and everything has gone quiet on the compulsory use of the tool.

https://www.theregister.com/2025/12/11/docker_hub_secrets_leak/?td=rt-3a

this is a pertinent reason why a vault should be used instead of a password.

instead of lots of passwords needing revoking, could just revoke the vault access if that was shared and have vault rotate passwords in its store if need be & not already rotated on a frequent basis.

sharing passwords is more about disseminating unique passwords for systems amongst automated tooling rather than sharing peoples individual passwords amongst others.

looks like your company should take another look & perhaps ask as many seemingly stupid questions as possible until it makes sense. Stupid questions are really just a failure of the answerer to communicate their message effectively which prompts what some deem as stupid questions. In reality they are not stupid.

[An_Old_Dog]

Re: Medical systems are a nightmare

If we spend the money on increasing security it may prevent a breach sometime in the future. If we spend the money on extra clinical staff hours it will prevent a death in the next few days.

1. If a business or organization is so underfunded that that choice -- security vs near-immediate patient death -- is truly binary, they should not be in business/operation.

2. A security breach of a medical organization's computers can lead to many deaths.

[John Sager]

Re: Medical systems are a nightmare

a business or organization is so underfunded that that choice -- security vs near-immediate patient death -- is truly binary

I can think of at least one organisation where this is true. But that's almost certainly down to misapplication of funds rather than lack of funding per se.

[midgepad]

Re: Medical systems are a nightmare

[citation desired]

[Anonymous Coward]

Re: Medical systems are a nightmare

I'm too far removed from the problem to give a current state view but look around the next time you have a medical appointment.

You will see devices from multiple vendors. They will be of different ages. Many vendors have gone out of business or stopped supporting older devices. Many devices require drivers that only ran on the version of Windows they tested on before putting it on the market. Most were selected by people with a lot of knowledge of medicine and little of IT security or support. Most were designed by the same kind of people. Some of them need to be used really quickly so don't have any logon process. Some need to keep a record and do that via a serial port to a printer.

[Anonymous Coward]

Re: Medical systems are a nightmare

Medical STAFF are the issue. We bought a large medical practice and 90% of our issues come from them.

Lack of security controls, "I'm a doctor, I need this to care for a patient." Admin rights given.

No security / privacy training, "I'm a doctor, I don't have time for this." Got a $100K EU fine months later.

No security / privacy training, "I clicked the MFA request because I was with a patient and the phone was making too much noise!" Access for one hacker. Didn't get anywhere, but they were in the system.

Lack of common sense, "I needed to make an important phone call and couldn't get through." He was standing on the 14th hole of a Scottish golf course without any mobile coverage.

Was once asked by the reception manager to find a mono (single one ear) headset with boom mic, that had noise cancellation. Just think about that for ten seconds.

[Excused Boots]

Re: Medical systems are a nightmare

"Was once asked by the reception manager to find a mono (single one ear) headset with boom mic, that had noise cancellation. Just think about that for ten seconds."

It really didn't take me ten seconds - 'Christ on a crutch'!

[Not Yb]

Re: Medical systems are a nightmare

That sort of thing used to exist, but the noise cancellation was not very good, and only reduced noise from around the person speaking into it. AKA "Call center headset", They used to be fairly commonly available before earbuds with internal microphones got so cheap. One ear would get the speaker and boom microphone.

Now that MEMS has made microphones so small and cheap ($0.10-$2 depending on size of order/quality of component)), it's better to put them on the internal side of the earbud and process the speech from there instead of getting it from just outside the mouth.

[Anonymous Coward]

Re: Medical systems are a nightmare

Yes but, if you have the power to upgrade, replace or remove the medical staff then you're operating on a whole different level to the one I was.

[Anonymous Coward]

Re: Medical systems are a nightmare

1) It's all shades of grey, you spend money for _more_ cyber security or more clinical cover (or more training for them). Maybe the computer system doesn't get breached, maybe the patient doesn't die. These are largely publicly funded organisations with people from clinical backgrounds in charge and public sector unions making the case for more and better paid or trained staff.

2) Ireland's HSE breach by Russia being a case in point.

[tip pc]

Re: Medical systems are a nightmare

1. If a business or organization is so underfunded that that choice -- security vs near-immediate patient death -- is truly binary, they should not be in business/operation.

2. A security breach of a medical organization's computers can lead to many deaths.

sounds like the NHS, same story regurgitated constantly yet nothing done to stop any waste.

[Anonymous Coward]

Re: Medical systems are a nightmare

Partly because the definition of waste depends on which political group is claiming to find it, and they're all biased one way or another. Reducing waste by one group's definition, frequently increases it by another. If one group is considering patient outcomes much more important than money, their definition of waste is "people aren't as healthy as they could be", and much less "we spent money on something that only helps a few people".

[Claptrap314]

Re: Medical systems are a nightmare

Regards item 1, in the US at least, due to the commercialization of medicine, this is this case in almost every business. The bean counters essentially demand it.

As for item 2, are you familiar with the term "overinsured"? If we want security to be viewed as insurance (as opposed to a cost), then we need to learn the lesson, which was espoused by L0ft before congress: security does not have infinite value. You must balance your security spending. For the case in question, you're talking about physical security being the primary guard--and it sounds like the only access was for doctors, who are REALLY motivated not to behave badly in this situation. Not that I agree with the business in this specific case--it would still be a violation of HIPPA, and as such, I would need to send an email including legal to that effect.

[BartyFartsLast]

Arghhh

Yes, I've seen similar, recently, and I'm still under NDA.

[Excused Boots]

Re: Arghhh

Hence the AC designation for posting.....

[Anonymous Coward]

Similar Story with FTP

About 20 years ago, my company outsourced marketing mailshots to a third party and we had to regularly upload the file from us to them for them to process. My colleagues thought nothing of it - they were Windows-based and just did what the procedures said - but as I was Solaris-orientated, I decided to cd to the parent directory. Just for a bit of a laugh, because, well, it won't work, will it?

Oh... I didn't expect to be able to do that. And look at all those company names, some big hitters there. I wonder...

And yes, I was able to access a couple of directories, list the files to view the permissions, but I didn't do anything else except show my boss, and I especially didn't view any files.

[Lee D]

Re: Similar Story with FTP

Working for a school which was in "special measures" for student behaviour and it was in the process of being taken over by one of these superheads (the scam behind them is too big for a single Reg post).

Anyway, one of their first decrees - you will merge the IT department with 3 other sites, you'll buy what we tell you and you'll buy this "new" remote virtual workstation system.

After we saw them off on the merger bull (by being able to outperform their "experts" in literally every metric they could supply us), they made us trial this workstation thing.

Basically, an early Linux-based cloud virtual desktop. Unfortunately for them, I knew Linux inside out.

They were claiming that it would "run everything" (via Wine) and it would all work... INCLUDING our workstation security software. Yeah, that AD-integrated, Windows-specific, lock-down-your-logon-screen, impose permissions on Windows applications, etc. software is just going to run unmodified on Linux and push the same settings... sure... that's gonna happen.

Anyway, because I knew Linux far better than anyone else in the department I was asked to trial it and come up with a list of issues. Which I did. Including things like "You're illegally using the Microsoft Office icons to load OpenOffice applications" and all kinds of other problems.

But the biggest problem came to light when the company said that I hadn't supplied them with a list yet.

"Yes I have."

"No, we haven't got it."

"Okay, log into your Linux server that's running all this system. Go to your root home folder. Look in there".

Yep... the permissions on the LIVE SYSTEM (it wasn't a demo system, they'd created a user for us already on their live system that they were making a bunch of other schools use and pay for) were so lax that you could just get a terminal, cd up a few folders, see ALL their users at ALL their sites, and even pop into other school's folders, play with configurations, and... yes... put files into the root home folder.

I'd left a text file in there with a laundry list of about 120 items, including the above, that they'd need to fix before we'd even consider touching them.

Whoops.

The ultimate irony came when all that was shoved to one side and we were NEVER asked to deploy that software (gosh, the owner was... a golf buddy of the superhead... strange that), but they stole our kids for the day to take the system to BETT (a UK educational conference) and have them "demo" it by just sitting on it and playing it. Not only had the kids never seen it, but we were never going to deploy it at that school anyway and they didn't use it, but they were made to pretend to. Instead, the kids compromised it within an hour, and then they were all telling the people at the conference how crap it was.

Strangely, that company is dead now but I bet they made a lot of money from those kinds of deals before they went bust, and I bet the owners and shareholders (including the superhead) pulled out before it collapsed.

[Antron Argaiv]

Re: Similar Story with FTP

Instead, the kids compromised it within an hour, and then they were all telling the people at the conference how crap it was.

Hands up anyone who did not anticipate this sentence.

[ecofeco]

Re: Similar Story with FTP

Good old boys network fails strikes again!

[Excused Boots]

Re: Similar Story with FTP

"I'd left a text file in there with a laundry list of about 120 items, including the above, that they'd need to fix before we'd even consider touching them."

So you had write access to the home folder root? And you weren't tempted to run a rm command, just to see what would happen?

[Doctor Syntax]

Re: Similar Story with FTP

Some people are professional and value their jobs.

[Excused Boots]

Re: Similar Story with FTP

True, but note that I didn’t suggest that they actually start deleting files and folders, but maybe try it with the -i parameter and abort if it looked like it was going to.

My bad, I should have included the ‘joke’ icon.

[doublelayer]

Re: Similar Story with FTP

That won't work. rm -i will just list files until you try to delete them, and only then will you find out whether you can. You can do manual stat calls to find out whether you can if you need to know that information. However, this sounds like it was from FTP, meaning your access is slightly different and normal terminal commands aren't what you're using.

[Cessquill]

In the UK...

...the vendor would be likened to Fujitsu. Because of the Post Office scandal.

[Filippo]

All access to everyone is, sadly, SOP for a lot of industrial systems I've worked with. The reason is that a bunch of these systems run on ancient versions of Windows and communicate through equally ancient protocols and once setup are very reliable until anything at all changes, at which point they become extremely brittle. Getting this to work alongside modern systems while maintaning security is a nightmare at best, and outright impossible at worst.

Worse, the IT folks that are responsible for the networks very rarely know what they're doing (or they'd be working elsewhere) and very frequently are overworked to hell, so even when all vendors are providing reasonably updated software, they still can't get things like automatically logging on a remote machine to work reliably.

If any of the vendors involved appears to actually have a clue, they'll pester him/her to hell and back to fix things that aren't their responsibility. Which of course they'll resist viciously, knowing that if they cave, then it does become their responsibility.

The fundamental problem is that the least competent vendor is also the most likely one to "solve" its problems by pointing fingers and demanding that everyone else proves it's not their fault, before they even attempt looking into a problem.

Basically, it's an environment where you'll typically have at least three factions, but usually more, in a Game-of-Thrones-like struggle to get everyone else to configure their fucking firewall properly this time and to stop changing it in production without testing anything.

I assure you that, after the sixth time you get a call because someone else has fucked up but the burden of proof is on you, the temptation to just tell them "sorry, the application requires that no firewall is present between it and the database" is very, very strong indeed. I mean, they have bought the thing already anyway, and (sadly) nobody will replace you because of something like that, so who gives a fuck? Substitute firewall for domain controller, database permission, encryption certificate, antivirus, whatever.

[Andy Taylor]

Nothing changes

We use a bulk email system with a name similar to bloke auger from a company which has a name similar to letter simian.

In order to view the bloke auger interface, a user has to be a letter simian administrator, no lesser permissions will work.

[Doctor Syntax]

"the vendor of the application had found a bug in their wares, and was patching database errors on the live system, during business hours, without telling their customer."

The system was flat-lining. You might even say it was Horizontal.

[ecofeco]

I wish to return this parrot.

[Excused Boots]

It's dead Jim!

[Anonymous Coward]

Sneaky remote access

I once worked for a vendor that had to demonstrate an automatic failover of a system that they'd installed for a very demanding customer on the other side of the globe, or risk being hit for $millions of liquidated damages.

On the night of the failover test (having only worked there for a few months and with bills to pay, don't judge me) I was told to join the conference bridge on mute, and when our project manager announced he was pulling the power, I was to manually trigger the failover - because the way the system had been designed and implemented meant there was in fact no automated fail over and we hadn't been able to fix it with no time or budget.

It worked, but the system was so badly designed that the manual failover process took two hours to run and the customer refused to sign it off anyway - which is exactly what I suspected was going to happen.

Difference being it now looked as though we'd hit the letter of the contract even if not the spirit of it - so we were given time to remediate the failover timing at our expense but with the threat of liquidated damages off the table. Everyone ended up happy-ish in the end, and most importantly, lawyers didn't get a cent.

[Excused Boots]

Re: Sneaky remote access

"and most importantly, lawyers didn't get a cent."

Which is always a win!

[A.P. Veening]

Re: Sneaky remote access

and most importantly, lawyers didn't get a cent.

That in and of itself is a positive point (the only one though).

[samsungfreud]

Vendor gone wild

I did a consulting job by special request.

A health organization received a custom C.R.U.D. database for tracking clients, payments, treatments, etc...

The vendor insisted there was nothing wrong with their software and it was the organizations fault for not understanding how to use their software.

I asked for and received the executables, databases and source (Alpha IV - early 1990's relational database).

I immediately noticed that the source did not match the screens or functions of the delivered application and had to ask the vendor several times for

the correct source sode.

Numerous features on the delivered software simply did not work: client info didn't always post to the database unless you entered a case first.

In the case of a new client, a phony treatment needed to be started before the client info was written to the database, only specific fields were written

during an update of patient info, fields in the insurance database were incorrectly labeled, the backup routine always ran in less than 2 seconds and

yielded an empty set of tables, new users could not be added using the "add user" screen and numerous other errors.

In short the package as delivered was a huge disaster and one user who deemed responsible for the use and training of others using this app

suffered a nervous breakdown. I was told by others that the vendor treated her very harshly and blamed her for all the problems in the software.

I later found out that the president of the health organization was best friends with the owner of the vendor that delivered this mess.

After two weeks I typed up all my findings and did a live demonstration of the app and showed how the source code was lacking in the features

to perform all the tasks as outlined in the poorly documented "manual" delivered with the software.

The president was visibly upset and actually turned red, her friend the vendor, kept trying to interject while I was speaking only to be stared down

by the president of the health organization.

I didn't get to finish my presentation.

The president told me she'd heard enough and thanked me profusely.

I left and received a bonus for my work.

Later I heard that after I left there was a *huge* shouting match between the vendor and the president.

Their friendship was no more.

Another vendor was tapped for a new software package.

The organization changed hands and names several times before being disbanded.

My only thoughts were of the poor woman who suffered the breakdown from this mess.

[ecofeco]

Re: Vendor gone wild

I call shennanigans.

Thanked by president? Given a bonus? Fired the best friend's company?

Go on, pull my other one!

Jokes aside, I'm glad it worked out for you. That is truly a unique set of events. Many of us here can tell you our experiences are often the opposite at every step of those events. I've lost more than one job for documenting issues. And never in my life received a bonus for solving a problem that no one else could.

Again, jokes aside, glad it worked out for you.

[samsungfreud]

Re: Vendor gone wild

It worked out very well, as a result of this another organization hired me to work on numerous databases for required reports under foxpro 2.0, 2.5 and visual foxpro.

That was fun until microsoft bought foxpro.

Shenanigans?

You wouldn't believe some of my experiences.

[PRR]

Re: Vendor gone wild

> You wouldn't believe some of my experiences.

Try us. Email to oncall. [e-mail]

[Doctor Syntax]

Re: Vendor gone wild

"Thanked by president? Given a bonus? Fired the best friend's company?"

The president had her job to protect. At that point it must have been looking very dicey indeed.

[trindflo]

Fixed IP addresses

I recall a vendor wanting to drop a client/server tool into our networks then announcing at some point that the machines had to have specific fixed IP addresses; every installation must have that specific IP address. As this was going into hundreds of distinct networks, I flagged it as a no-go. The vendor was baffled that I would have a problem with that.

[Herby]

Databases, permissions, and all that...

Obligatory XKCD: https://xkcd.com/327

[JulieM]

It's because it's proprietary software

All these things are only problems because the software is multiple levels of proprietary, and no-one has access to the complete Source Code.

And all this is only made worse, when back-bedroom "developers" use pirate copies of programming languages and don't read the dire warnings included in the printed manuals about how properly to interact with other software. Which means they end up (perhaps unwittingly) using exactly the same techniques used by viruses and malware, and run into security software.

Open Source software just works -- or else it gets fixed, or just not used.

[Kev99]

Where I once we worked purchased an electronic sign for placing near the street. One of the selling points was the sign used wi-fi so we could change the messages from inside our offices. What they didn't tell us was the wi-fi had a range of only a hundred feet or so. When we tried to access the sign using the pc based controller we got repeated error messages. Only much griping and grousing were we told of the very short range. Our treasurer, who was an engineer at his full time job, and I had to run CAT-5 cable from the office PC through a suspended ceiling and, thankfully, pre-existing holes in the concrete wall to a room with large windows that had a clear line of sight to the sign. As an aside, neither of us had any experience running cable.

[J.G.Harston]

I was on a Windows 7 roll-out project 12 years ago, and about a third of the applications we migrated refused to work unless they had ALLUSERS access to the whole harddrive.

Seriously, users only having full access to their user data has been normal programming since before the '80s, why are commercial developers still shooting holes in their head 30 years later?