It's completely unsupportable. Yes, we mean your brand new system

The concept of "shadow IT" is a familiar one. One of my favourite descriptions of it comes from security vendor Forcepoint, which says shadow IT is "the use of information technology systems, devices, software, applications, and services without explicit IT department approval." It has grown exponentially in recent years with …

[b0llchit]

Small steps

The article points out the very truth of all systems, IT and non-IT. Innovative at small scale is a maintenance problem if done wrong. Innovative at large scale is a maintenance problem if done wrong. Often, only when the innovation becomes general mainstream, then you have established procedures that keep things running and cost down.

The real kicker of problematic is people at a company dictating a specific system and or sub-system. How often I've seen the management level interfering with development and innovation by forcing the use of marginally correct or incompatible pieces. That is the nightmare of maintenance doubled down (and the techies get the blamed for all the problems).

The lesson all should learn is small steps are extremely important in making big systems. Planning not only the functionality of a system but the longevity and maintenance are all integral to the system and take time to make/build and describe/document. And lets not forget planing to handle failures and problems while running.

[Anonymous Coward]

Re: Small steps

" take time to make/build and describe/document."

Aye, there's the rub. Time is money, and management wants things yesterday. To get it done as close to management demands as possible, IT drops the describe/document for a 'later' release.

There is also management's desire for more than a small step. And, as you mention, their 'input' as to what that bigger step should be.

There are people, in and out of IT, who know how to do things right. It's just that nobody's listening to them.

[Phil O'Sophical]

Re: Small steps

The real kicker of problematic is people at a company dictating a specific system and or sub-system.

In my experience this is the biggest driver of shadow IT. To take a recent example, our corporate IT group decided to replace the in-house wiki & document-sharing platform that we had been using for years with a different, more modern one. The argument was that it was newer and more supportable. This was simply imposed on the company without consultation (beyond the C-suite).

For many of the basic sales/marketing users who used it to upload plain info, HowTos, etc. it was just the usual pain of learning a new system. No big deal - "they'll get over it". The development organization, however, was a big user of in-house tools, specific to our products and processes, and they produced HTML output. The new wiki system would not render HTML without new plugins which were not part of the deployment, and were clunky & unfriendly to use in any case.

After a few months of trying to adapt, various parts of the development org opted for the obvious solutions, they started to re-purpose lab systems as standalone web servers. They served the HTML and the wiki system just became a front-end to them.

Is this maintainable? Sort-of. In most cases the systems were stood up by the team that needed them, so when someone leaves they rot until someone else figures out how they work, which depends on how well the initial developer of the system documented it. Does it get patched in time? Sometimes, when people remember. Does it respect the in-house security rules? Best-effort, but obviously IT don't audit it. Does it allow the development teams to get on with their core work, and make money for the company? Absolutely.

That's what causes shadow IT.

[Pascal Monett]

We in IT know the golden rule : if it works, don't fix it.

[Claptrap314]

There are TWO golden rules:

#1 if it works, don't fix it.

#2 always keep up to date on patches.

[Strahd Ivarius]

#2 is not always compatible with #1

Try to update the ESA Mars orbiter with the latest Windows patches for example...

[martinusher]

Re: Small steps

> interfering with development and innovation by forcing the use of marginally correct...

For much of the corporate world 'IT' is really Office365 which implies a Windows environment. This is irritating, but more or less tolerable because there's always workarounds. Then IT decides to try to integrate development into the Windows Way Of Doing Things. The result is a sort of paralysis as developers focus their attention on the bits that could be moulded into thar environment and put of coping with the bits that don't. Eventually the activity folds because nothing got done -- there's plenty of activity but no closure, nobody actually finishes a project.

(I can recommend retirement. Its ducking the problem but who cares?)

[Anonymous Coward]

Re: Small steps

(I can recommend retirement. Its ducking the problem but who cares?)

7 weeks and counting...

[Mark 65]

Are we confident that the technology has the longevity we need?

This is interesting itself in the cloud context where you control nothing and features/aspects you may be relying on get upgraded in an incompatible manner or receive a swift dispatch as Google tends to do with things it has lost interest in. You don't have to patch it yourself but then you may get a breaking patch installed for you (the greater good) or a feature removed.

[amanfromMars 1]

YMMV SNAFUBAR MRDA

The majority of organisations — particularly the IT and security teams — are conscious of the potential threats from shadow IT and are on the lookout for it so it can be stamped on. ... Dave Cartwright

"I'm sorry Dave, I'm afraid I can't let you do that" is an appropriate misquote here, methinks, to give you an accurate picture of the present situation in current circumstances racing ahead to and fro into the future and out of the past.

[KalF]

organisations should specialise on what is core to their business. That part should never be outsourced, but everything else either should be completely outsourced or operated in a manner that prioritises a stable and efficient service to the broader organisation. Effectively replicating the benefits of outsourcing, while mitigating the downsides we've all suffered.

Therefore if you innovate within your core business, but prioritise the capabilities within the support arm of the organisation, as appears to be the moral of the story, then you are holding the stick wrong. I think many CTOs and CIOs feel connected to their enterprise backoffice support backgrounds, to the detriment of business value.

There's clearly a balance that must be found, skills and control within your supporting services are important, but its the core business that should drive innovation. In the story in the article, I would have refocused the team on the product platform (assuming those skills werent there to begin with) and thought long and hard about my dependence on skills for 'mainstream' systems that didnt make me any money. They had effectively invested in skills and capabilities that didnt offer a competitive advantage. The good staff will adapt and thrive.

[Anonymous Coward]

everything else either should be completely outsourced

That can create a different sort of shadow IT problem.

Company decides to outsource the internal helpdesk. Instead of being operated by employees who have a commitment to the business, it's now run by people who are paid to close calls as quickly as possible and have script-driven responses ("Step 1: have you turned it off and on again?").

Doesn't take long for people to find that logging support calls isn't very helpful any more. Then someone says "Bob had that problem, ask him". Bob explains his solution, it works. His solution may not be correct, it might even make things worse, but soon Bob is the unofficial go-to person for that group. After a while every group has their "Bob", who get frustrated because their real work is suffering, but they try to do their best for their colleagues, not realising that they may actually be making things worse in the long run with kluges & workarounds (ever tried to update a Word document from someone who thinks they know how to format stuff correctly, and shared their tips with everyone?)

Meanwhile the calls to the official support desk fall, which the IT group uses as proof that the systems are working well but which in reality shows that things are totally broken.

[yoganmahew]

And then you have the "what is core? Baby don't code it" crowd. Those accountants and program managers that effectively run a large organisation. To them, they are core, everything else can be outsourced. So soon your core product is being built by outsourced teams. You will never know what it does or how it does it or how to fix it. It will always be broken in one way or another. Every fix will require a refactoring, change control, and a large project team.

I'm close enough to the end of my career that it's sad, but I no longer care. The battle's lost and the game is too.

[Anonymous Coward]

"Company decides to outsource the internal helpdesk. Instead of being operated by employees who have a commitment to the business, it's now run by people who are paid to close calls as quickly as possible and have script-driven responses ("Step 1: have you turned it off and on again?")"

Year 1: heating fails. diagnosed as faulty pump. got working on secondary pump only.

Year 2: heating fails. diagnosed as another faulty pump. unable to resolve immediately as both pumps are now kaput!

Yes, the outsourced help desk passed the problem to an outsourced maintenance company and all the tickets were closed promptly as 'resolved'

(don't get me started on getting them to replace dodgy fluorescent tubes... "we can't get that type anymore", so it's not in the maintenance contract, it's someone else's budget)

[Mike 16]

Tickets were closed

Ah, yes. Like when I worked for a major manufacturer of networking equipment and the access point that (allegedly) served the area including my cube lost its back-haul. I lost track of how many times I filed that, only to have it closed because the WiFi crew had verified that they could connect to the AP. It didn't matter how I phrased "test by connecting to something other than the AP".

I developed a strategy of carrying my laptop close enough to a nearby AP to connect, and _carefully_ walking back to my desk, where I could continue to work via the "wrong" AP until something hiccuped and the laptop sees that wonderfully strong signal from "my" AP and re-connects to the bridge to nowhere.

After a couple months of doing this a couple times a week, I took the obviously less hassle route by moving to another company. This of course is easier in an area like Si Valley.

[doublelayer]

"In the story in the article, I would have refocused the team on the product platform (assuming those skills werent there to begin with) and thought long and hard about my dependence on skills for 'mainstream' systems that didnt make me any money. They had effectively invested in skills and capabilities that didnt offer a competitive advantage."

This is the dangerous part of your recommendation. The focus only on the things that make you money. Most of a business doesn't make you money, but it does permit the existence of the part which does. You have to look at each part that doesn't make you money, decide whether it is needed, and for the most of them that are, how you're going to pay for it. Lots of those things can be outsourced safely, but each outsourcing carries costs as well as benefits.

It's the responsibility of people like CIOs to do that kind of analysis. They have to be connected to their "back office" work because nobody else is going to do it. They need to know what outsourced IT is like, and whether the company can handle the costs involved. Outsourced support is sometimes doable but carries risks as described in the first reply to you. Outsourced administration is usually not because it involves a lot of information you need if ever something goes wrong. Outsourced hardware (AKA cloud) is reasonable but you have to spend some time checking out the financials because the cloud providers won't protect you. Outsourced design can work if you have someone internal who can take designs from multiple people and deploy them, but if the outsourced design people are doing that, then it's hard to replace them when needed. Outsourced development will work in the short-term but you could end up with a dependence on the provider if you're not careful right now. Those are things that somebody has to know and decide, and the operations people who never worked with it won't be familiar with the details.

[KalF]

I may not have made the 'focus on what is core' part clear. The skills you prioritise should be for your core money making systems. If you allow your back office tech stack to dictate tech choices for your core products you'll have a company that can't compete or evolve.

Your back office is not unimportant and ppl shouldnt draw that conclusion simply because I say it shouldnt drive your tech choices. But efficiency gains should drive the cost and focus down over time for those support systems.

As for deciding what your core systems are, if the CIO and CTO can't decide that, then tech stack choices are the least of your challenges.

[doublelayer]

"If you allow your back office tech stack to dictate tech choices for your core products you'll have a company that can't compete or evolve."

I disagree. If you're like a lot of businesses, your back office tech staff know a lot more about the options than the people managing them. In turn, those technical managers know a lot more than any other department. Ask a financial person whether cloud is suitable for this deployment and all they can do is look at numbers provided by someone else. The tech staff and management should be able to tell you what cloud services they would use, what non-cloud alternatives exist, and eventually generate the numbers for the finance department. The people who generate those numbers are usually making the decision by the numbers they send, so you really need them to know what they're doing.

"Your back office is not unimportant and ppl shouldnt draw that conclusion simply because I say it shouldnt drive your tech choices. But efficiency gains should drive the cost and focus down over time for those support systems."

And depending on what "efficiency" means, that could kill your core service. Business people understand about long-term survival needs, where they will have to take lower profits for a while in order that they continue to have a revenue stream. They don't always go for that option, especially when they don't care about the long-term viability of the business, but they at least understand what it is. Efficiency which reduces quality is dangerous, and it is not difficult to explain this to someone else.

[Terry 6]

Most of a business doesn't make you money, but it does permit the existence of the part which does.

This is the bit that fascinates me when I've heard these kind of stories. To me it sounds like they're saying "It's only the wheels that make the car move, so lets use the cheapest possible engine and do away with the doors and seats".

[Anonymous Coward]

Controls?

How often have you been to a lessons learnt meeting and bitten your tongue? The lessons are very rarely new because the mistakes are not new!

However when the lessons have been learnt and are applied as part of a process delivery is slowed down, costs go up and then the business is frustrated by the perception that the benefits it wants realised are diluted!

Change the methodology; bring in new terms such as Minimum Viable Product, bypass the process and get everyone onboard and off we go onto make the same mistakes and learn once again how quickly experience is lost!

At the end of the day it is these repeated mistakes by many very clever people that keep a great many of us employed . . .

[ThatOne]

Re: Controls?

> off we go onto make the same mistakes

Lessons are never learnt, if humans were able to learn from their errors we would be really superior beings by now, wouldn't we...

Errors are mostly created by peoples' ways of thinking and acting, and thus will be readily repeated after a short cooling-off period allowing them to forget that their obvious genius might not be that infallible after all. You forget those things very fast, it takes mere days before any self-doubt is thoroughly erased and failure is put down to bad luck or somebody else's fault.

So yes, same people make same mistakes. Nothing surprising there. Only (some) young and still inexperienced persons are still willing to challenge their preconceptions, and that ends rather quickly when they start realizing their "teachers" don't know that much either.

[Anonymous Coward]

A wise man once said . . .

Common sense is not common

[Nifty]

Don't overlook the human factor with the techies themselves though. Senior tech individuals would favour certain platforms and languages because they have the most experience/have an ideological affinity with them. Makes themselves more valuable to the detriment of eventual supportability.

[Eric Kimminau TREG]

Death by Innovation

We have an internal team whose stated goal is to drive innovation.

Their current "drive" is to demand that IT support Docker on Windows Server 2019.

First, Microsoft has flatly stated that they do not support Docker but they do support a single vendor with a single stack and all support for Docker would have to come through that one vendor. Otherwise, "not our problem.

Second, who the hell wants to build a virtualization stack on top of a Windows Server OS? Really?

Third, and the major nail in the coffin, is that our entire Windows Server environment is already virtualized on VMWare. Microsoft clearly states with no ambiguity that they will not support ANY virtualization stack on any OS that is already virtualized. Forget Docker. Microsoft wont even support HyperV on VMWare (all you VDI wankers remember that Windows 10 now includes Hyper-V turned on in base build, right? Ever wondered why the hell Microsoft would enable virtualization on a standard desktop?). Microsoft clearly states that if your OS is already virtualized and you add Docker to the mix, your OS is entirely unsupported.

We laid these issues out for the Innovation team. Their response? "So as long as we are OK with not getting any support for Docker, we are OK to move forward?"

Our response was that we had no intention of deploying an environment that would result in an unsupported OS on our enterprise shared vmware environment. They would need to deploy their own vmware infrastructure to host their unsupported OS.

Their response?

"IT is being a roadblock to Innovation. We will just buy our own beefy desktops and run Docker on Windows Server 2019 on bare metal."

When asked "why do you even need docker on Windows Server 2019?" their response was purely brillance. "Because we are innovating."

[BinkyTheMagicPaperclip]

Re: Death by Innovation

Innovation=CV padding so they can get another job

[Claptrap314]

Required to understand

"We were defeated by one thing only - by the inferior science of our enemies."