Stick to the script, kiddies: Some dos and don'ts for the workplace

Scripting is now the first choice for clued-up administrators who want to get things done quickly and in an automated fashion. However, scripting does bring its own set of issues. One only need look here or here to feel the pain. Software-defined data centres exist purely in code (obviously with hardware underneath) so to help …

[Bubba Von Braun]

Great Advice

Good common sense advice, after all its code of a form, and should be subject to reasonable change controls.

The same goes for SysAdmins who Google/Technet/Spiceworks etc solutions. I always have my admins verify a solution before they try and implement it. How many times have we had a user or a Help Desk guy grab a solution from a Google search try it and make a situation a whole lot worse.

Key thing is peer review/support. Its a whole lot easier to get a colleague to cross-check what your planning to do, rather than tie up the entire department cleaning up the mess created by that "quick fix" you found.

[Steve Button]

Really useful article.

If you've just left school and want to get into coding.

For the rest of us, I suspect this is bread and butter stuff. Are there really still SysAdmins around who don't know this stuff (apart from at the local council)

[Anonymous Coward]

Re: Really useful article.

Oh yes, I personally saw a three day outage caused by an AD/DNS mess at a big 4 audit company because some lad in Argentina tried to be smart and ran his own scripts against production.

Anon because I'm technically under a NDA

[Anonymous Coward]

Re: I'm technically under a NDA

Naked Database Administrator?

[DuchessofDukeStreet]

Re: I'm technically under a NDA

"Naked Database Administrator"

It's not Friday....I don't need mental images like that in my head!

[Doctor Syntax]

Re: I'm technically under a NDA

"It's not Friday....I don't need mental images like that in my head!"

You do on Fridays?

[Bill M]

Re: I'm technically under a NDA

I just hope he doesn't tell us about Saturdays.

[veti]

Re: I'm technically under a NDA

"Naked Database Administrator"

It's not Friday....I don't need mental images like that in my head!

It is Friday, and I still don't need that mental image thankyewsoverymuch.

[jake]

Re: I'm technically under a NDA

Regardless of the day of the week, beauty is in the eye of the beholder.

[kventin]

Re: I'm technically under a NDA

beauty is in the eye of the beerholder

FTFY

[jake]

Re: I'm technically under a NDA

Only at closing time, kventin.

It's not quite 8AM, but it's Friday, so cheers anyway.

[RyokuMas]

Re: Really useful article.

Yes.

I have just been involved with a project in which the client required me to get their website up on a number of new environments, each comprising of a back-end edit server and a number of content delivery servers. So far, I have encountered the following problems:

- Wrong .NET version

- Missing IIS HTTP modules

- Ports not open

- Missing Windows features/components

- Servers turning out to be completely dead

- Missing access rights

... across various servers. In fact, I would go so far as to say that no two servers are exactly alike. Not exactly confidence-inspiring when it comes to deployment and testing...

[Anonymous Bullard]

Re: Really useful article.

So far, I have encountered the following problems

...

Windows or not, there's no excuse for this type of thing. If you haven't got an reproducible and automated deployment then you don't know what you're doing, just hacking and tweaking until it happens to work.

(I don't mean "you")

Too many self-taught "power users" who think they're experts.

[BinkyTheMagicPaperclip]

Re: Really useful article.

If you said understood, reproducible and documented deployment I'd agree. Deployment does not have to be automated. Certainly it's useful, particularly if the number of deployments is non trivial, but it isn't essential.

There are also 'automated' and documented deployments that are not understood. They work fine until moved to a very slightly different server environment, and then *crunch*.

Documentation is the most necessary item, so that when unusual configurations are encountered, or say the OS upgrades and implements new restrictions not yet supported by the automated script, it can still be implemented.

[John Smith 19]

Too many self-taught "power users" who think they're experts.

True.

But there is a worse problem.

They don't know what they don't know. Hence they don't know what they have to learn, or why things can be made a lot better.

[HmmmYes]

Re: Really useful article.

I've given up on the whole Windows thing - Powershell, .net, c#.

Just decided its no longer worth the investment og my time it as, by the looks of it, 90% of the stuff Ill be doing in the future will be Linux based.

[Anonymous Coward]

Re: Really useful article.

Are there really still SysAdmins around who don't know this stuff

Yes, plenty (in the Windows area). They don't even script things - excusing their lack of ability with things like "scripts get it wrong" and "I don't trust it to work repeatedly", "it's too advanced to be scriped" - click-click-click

I've worked at places where I've been told not to produce scripts for these reasons, and follow/write numbered bullet points.

If anything, I script out of pure laziness.

[Anonymous Coward]

Re: Really useful article.

In Windows you have some things unheard of on other systems - like installer and wizards that can create reply scripts to be run in quite mode... just some sysadmin coming from other systems stubbornly refuse to learn...

For the matter, to deploy software on Windows I prefer to write an installer with a proper setup creation tool - which usually ensure you can also rollback changes quite easily - than a bunch of scripts utterly incapable of doing anything usefull when a command line call goes wrong, and leave the system in a mess.

Even on Linux writing a proper rpm/deb is better than a bunch of ugly scripts.

You can still orchestrate them better from Ansible or something alike, than trying to mess with the whole system yourself.

[Anonymous Coward]

Re: Really useful article.

"In Windows ..."

Perhaps you're the type who should take heed of this article.

[Ken Hagan]

Re: Really useful article.

"installer and wizards that can create reply scripts to be run in quite mode"

...like a deb, then?

"prefer to write an installer with a proper setup creation tool"

...like a deb, then?

"Even on Linux writing a proper rpm/deb is better than a bunch of ugly scripts."

...so you *have* heard of them, even if you haven't yet grasped the fact that they package, amongst other things, scripts.

[Doctor Syntax]

Re: Really useful article.

"For the rest of us, I suspect this is bread and butter stuff."

That's what I thought until I realised it was aimed at Windows admins. However I suspect Linux could be going that way; how else does one explain init scripts as being too difficult?

[Teiwaz]

Re: Really useful article.

However I suspect Linux could be going that way; how else does one explain init scripts as being too difficult?

Init scripts are certainly easier than the systemd unit setups. Bash and JCL I can pump out in my sleep (oo-er!) but Systemd I keep having to refer to the docs.

Other than that, I don't have much issue with it, but then I use Arch at home.

[I Am Spartacus]

JCl

Score an upvote for being able to write JCL in your sleep.

For me, it was always a nightmare.

But then again, I got a DD statement wrong, my job crashed and produced a JES2/HASP dump. And nicely printed out the whole of the companies shareholder list, with their names, addresses, holding etc. Gotta love a core dump!

[Nick Kew]

Re: Really useful article.

You merely suspect? Haven't you been following the whole systemd debate?

[Doctor Syntax]

Re: Really useful article.

"Haven't you been following the whole systemd debate?"

What did you think my comment was about?

[John Smith 19]

Re: Really useful article. If you've just left school and want to get into coding.

Funny you should say that.

I'd just started at a very large company and had no phone service for 3 days.

Phone system run by SQL server application, which was updated by a script.

Running under a long term employee's login.

Who'd just left, killing any running processes.

So yes, sometimes reminding people of the obvious is quite a good idea.

[Anonymous Coward]

Powershell is absolutely shite, but at least it's given use the ability to script.

Sure, we had WSH and batch, the languages where "adequate" but they couldn't do anything useful due to the lack of bindings

.

[Ken Hagan]

"Sure, we had WSH and batch, the languages where "adequate" but they couldn't do anything useful due to the lack of bindings"

Ummm, the lack of bindings is a problem with the thing you are trying to script, not the language you are trying to script from. However, I will agree that the adoption of Powershell *did* give Microsoft developers a strong kick up the arse on the matter of providing such bindings in a more systematic fashion across the system.

[Anonymous Coward]

"Powershell is absolutely shite, but at least it's given use the ability to script"

Translation:

I don't understand it, therefore it's shite.

[HmmmYes]

Hmm, not sure.

Sure, Powershell is better than DOs - it would struggle to be worse. And its means I dont have to piss around with GUI.

But the syntax is way too wordy for being a script - PS needed a powerful editor to bang heads o gethe to keep the core short + sprightly. As it is, is a spawling mess. Its seems to be stuck in a nether rworld between a shell and a system language.

And its slooooooooooooooow. I use PS as my Window shell for admin stuff. Its very laggy. Yes, Ive ried all the recommendations.

And the worse thing - the versions and the bloat.

Oh, I ned to do something on (old) box. OK Ill run this script - Oh fuck it does not work. OK Ill update Power shell - .net download kicks off ...

[Old Coot]

No, I don't understand it; enlighten me.

Does it do something that the Unix shell doesn't, or do it better?

If yes, please enumerate

If no, then why make up a new shell system? Is it just for the sake of doing it, or is there a 3rd possibility that I've missed?

[HmmmYes]

It has an object pipeline, thats the main differencre

Rather than a fifo of characyers, you have fifo of objects.

And, like anything involving objects, everything expands like mad.

And they did not have a No man when it came to adding new commands and options.

Powershell is open source now. Maybe somone wil fix it...

[Dan 55]

Blocking e.g. github fixes the symptom, not the cause

In the absence of documentation, toolchains, process management, and central repositories, people will try anything they can find.

[Pete 2]

Something missing?

the important aspects of the script:

Who is running it?

Is the script version controlled?

Is the script approved for implementation?

Pardon my ignorance, but isn't the single most important aspect of a script that it actually works?

I would also suggest that the differences between scripting an operation and performing it manually is little different from performing it "manually" (on a computer) and actually doing the operations in person.

Insert data into a table, or place it in a filing cabinet

Delete a file or shred a document

All equally prone to error - filing "B" before "A", failure (the shredder jams), incorrect instruction - which files to destroy. The only difference being that the supervisor feels more confident by having a person doing the work, rather than a machine. Either that, or they know they can blame the individual for any mistakes rather than taking the blame themselves.

[Doctor Syntax]

Re: Something missing?

I would also suggest that the differences between scripting an operation and performing it manually is little different from performing it "manually" (on a computer) and actually doing the operations in person.

1. It can be reviewed by yourself and others. That way it can be checked for errors, including typos such as rm -rf ~fred /*

2. It's repeatable. If you have a saved script that was successfully yesterday you know it will perform exactly the same operations today and tomorrow which you might not do yourself if sitting there thinking "Now what did I do next?".

[Anonymous Coward]

Re: Something missing?

> Pardon my ignorance, but isn't the single most important aspect of a script that it actually works?

No, that's number two. Number one is that it does something sensible and well-defined if it doesn't work.

(Ideally, it leaves things as they were, but that's not always possible. For example; a script updating 20 machines might fail on one. It should be possible to fix whatever is wrong with the one machine and run the script again and it will know that 19 succeeded and don't need to be re-visited. Or it should be easy for you to remove those 19 from the list to process. And so on.)

[veti]

Re: Something missing?

Pardon my ignorance, but isn't the single most important aspect of a script that it actually works?

No, the most important aspect is whether or not you can tell that it's working.

And without the abovementioned change control aspects, the answer is: no, you can't.

It could be running exactly as designed - but without any way of knowing what it's designed to do, that's no help to anyone.

[Tom Paine]

Re: Something missing?

Alright. Look. They're ALL the most important. Alright?

[David Woodhead]

Re: Something missing?

@Pete 2:

Pardon my ignorance, but isn't the single most important aspect of a script that it actually works?

This times 1,000,000. When I did my fairly simple end of first year project at UMIST (under Jeff Rohl, one of the creators of Atlas Autocode), he taught me what was possibly the most useful thing I ever learned about software development. To quote very loosely: 'You can write the most elegantly structured software, with reams of helpful comments, but if it crashes or doesn't do what it's required to do then it's worthless.'

I owe him a great deal just for this.

[Colin McKinnon]

Am I missing something?

"Scripting is now the first choice for clued-up administrators who want to get things done quickly and in an automated fashion"

- what was it before now?

[Daggerchild]

Re: Am I missing something?

A binder of printed procedures that you read and manually type in the commands. 1960's stylee

And I say that in full knowledge that this is alive and well, and there are people out there doing that now in response to a computerised alert from expensive computers.

Yes, humans are using computers to run automated scripts on humans using computers. Meatscript.

[HieronymusBloggs]

Re: Am I missing something?

"what was it before now?"

Knock out a quick Cobol or PL/1 program?

[Lee D]

I see scripts as one-off sticking-plaster patches.

You should not be relying on them for everyday tasks, or even "every year" tasks, because you're just opening yourself up to problems.

If you use a script on any kind of regular (even if infrequent) basis, you need to get that logic put into whatever system you're performing it against, and get it ratified, and test-suited, and checked, and put into lists of things to keep up to date. Effectively you are then doing software development, whether you're a tiny one-man operation or a huge multi-national, and the same standards as you'd expect a software developer to use should apply - testing, verification, dummy-runs, early bail-outs, stop on every error, etc.

If you're using a script for a one-off action, on your own head be it. At the very least, you test against copies of the production data first, but even if you spot nothing it doesn't mean you haven't broken anything.

Any idiot who just codes up a script and executes it should be fired. At the very least, even as a network admin coding up a "replace all job titles" AD Powershell script or similar, you replace the dangerous parts with a print statement along the lines of "This would remove user X" rather than actually modify the data.

I think of this every time I'm forced back to Powershell / Exchange Management Shell to do simple actions like search for a particular email, etc. Check you have backups, do a dummy run, be careful of any command that actually performs an action until you've tested the action itself on one test user, and got a list of users that it would try to apply it to, etc.

And if you have an ounce of common sense, any action that "takes out the entire AD" shouldn't even be executed as a user with permissions to do anything like that... you test as a delegated admin with almost no rights except to the fields you're modified because then if the script "goes rogue" it can't shoot off any bits that you weren't going to be playing with anyway.

Playing fast and loose with your business / customer's data is a sure way to end up on the dole queue or in court.

[Daggerchild]

"You should not be relying on them for everyday tasks"

Most of my estate is scripts. A script is a computer language just like any other, unless you want to view it instead as an application configuration, which is equally valid. If you treat it differently based on what it's called instead of what it does, you're holding the wrong end.

I've got scripts in my head I cough up on demand. I've got pages of scripts I wrote yesterday. I've also got scripts doing high availability failover, which go through change control, with peer review.

And yet business processes, which are basically scripts that run on humans, do they go through equivalent change control and peer review? It's data process configuration all the way down...

[veti]

And yet business processes, which are basically scripts that run on humans, do they go through equivalent change control and peer review?

As a matter of fact - in the companies I've worked in, at least - they generally go through much more rigorous change control. Because they affect whole teams/lots of people, rather than just you and the one guy who's looking at your reports. So there will be real thought put into exactly what the change is supposed to achieve, and how to tell whether it's working.

[Doctor Syntax]

You should not be relying on them for everyday tasks, or even "every year" tasks, because you're just opening yourself up to problems.

This is just what you should be using them for. As you say, you can get them ratified, check into the source code revision system of your choice or whatever in order to have a repeatable set of operations on which you can rely. For rarely performed operations this is even more important than daily ones.

Effectively you are then doing software development, whether you're a tiny one-man operation or a huge multi-national, and the same standards as you'd expect a software developer to use should apply - testing, verification, dummy-runs, early bail-outs, stop on every error, etc.

Yes. Why would you fly by the seat of your pants doing operations manually when you have this option?

[HieronymusBloggs]

"You should not be relying on them for everyday tasks"

I take it you're not a Unix/Linux sysadmin.

[handleoclast]

Do not run with scissors

Do not point gun at foot and pull trigger.

Do not stand directly behind somebody about to fire an RPG.

Scripts can be useful, but be careful with them.

[Doctor Syntax]

Re: Do not run with scissors

"Scripts can be useful, but be careful with them."

Be even more careful just going in there and hacking it by hand.

[Daggerchild]

With Great Power, comes Great Change Control.

Great. If the suits latch onto the word 'Scripts' and associate that with 'the power to damage stuff' they will demand all scripts go through The Change Process, and, just like that, I'll only be able to talk to my machines once a week...

[Ken Hagan]

Re: With Great Power, comes Great Change Control.

Welcome to my world. I'm an application developer and my machines actually belong to my customers and, yes, I'm only allowed to talk to them after the script (my application) has been tested on mock-up systems that don't matter and then signed off by quality control once they demonstrably work.

Why should you have lower standards when the target system belongs to your own company?

[Daggerchild]

Re: With Great Power, comes Great Change Control.

"Welcome to my world. I'm an application developer"

Hi, I'm the whole stack. Did you know not everything looks like a nail?

"Why should you have lower standards when the target system belongs to your own company?"

'Lower'. Gee thanks. Try 'appropriate'. I use version control and operational acceptance testing too you know.

If you ever want me to find out why your app works on test, but not production, I'll need to talk to the machines. With bespoke scripts, written on the fly. They don't all have the power to blow up the world.

[Captain Badmouth]

Re: With Great Power, comes Great Change Control.

"I'll only be able to talk to my machines once a week..."

Don't go telling them you talk to your machines, either.

[jake]

Re: With Great Power, comes Great Change Control.

I don't talk to my machines. I'll cop to swearing at them occasionally, though.

[Tilde]

Re: With Great Power, comes Great Change Control.

Unless it's malfunctioning why would you ever need to talk to your machine. The machine is created by scripts and managed by scripts. An admin never has to touch a good machine in it's finite life. All systems scripts should be controlled at the same level as software.

[Daggerchild]

Re: With Great Power, comes Great Change Control.

"Unless it's malfunctioning why would you ever need to talk to your machine"

Do you only exchange information with an entity when something is wrong?

[a_yank_lurker]

Issue with Scripts

The real issue is not scripts but a lack of a proper life cycle including a rudimentary spec (why are you doing this), peer review, testing, version control, and documentation. Some of the spec and documentation can be in the script itself but should exist. Also, testing as much as possible should be on development system not production. Yes this could add a few hours or days to the development time but scripts are actually small programs and should be treated as a program.

[Doctor Syntax]

Re: Issue with Scripts

"The real issue is not scripts but a lack of a proper life cycle including a rudimentary spec (why are you doing this), peer review, testing, version control, and documentation."

You say that as if they're good things - which they are, of course, as is repeatability. Because, as you say, scripts are small programs these can be applied and more easily than into manual operations; even if the latter are written down in your ops manual as an - errrm - script you still rely on the operator following them.

[Herbert Meyer]

read old warnings, change word

Re-read all your old warnings about macros, change the word "macro" to "script".

[Amorous Cowherder]

The number of times I've had to step in an deal with morons who've just downloaded some chunk of code, didn't bother reading it and turned a minor problem into a phone call home to the missus to ask her to shove our evening meal in the oven for whenever I finally get home!!

[clouseau_]

This is why we are moving to Ansible Tower.

Version controlled scripts, standard way of doing things (ansible), and tower restricts permissions to just those who should have them.

[Alistair]

scripts! OMG scripts!

I've built out automation processes for hosts using version control and an automation tool to ensure that the hosts were built, managed, monitored and had (a whole mess of different) applications installed on them correctly, consistently and repeatably. The backbone of this was scripting. Some in python, some in perl, some in bash, some in rshell, some in sh. I've got core application scripts in ksh that were deployed to an app over 11 years ago that are *still* in use today. I've written a *very* few powershell scripts to monitor and maintain a non windows based application on windows hosts (that I very much despise having to deploy to windows boxes, hopefully never again since MS has ssh for windows). Most of my scripts were written to do one thing and one thing only, and to bail if the environment they were pointed at did not match what they were meant to do. One of the reasons I'm still kept around is because I can go through the mental process of (from box on loading dock to plugged in, built, deployed, connected, made operational) and pick out the hundreds of steps that go into each step of that flow.

The biggest issue anyone ever runs into in scripting is *not* comprehending that things can go *wrong*. Error cases are *hard* when you haven't had time to think them through or test them out.

[OzBob]

I use scripts all the time

But I test them like they could possibly delete the environment (and I insist on a separate playpen area to develop in, that replicates prod).

Part of the problem I have with DevOps is that there is no "preview" mode and the recovery method should something go wrong is long and convoluted. Surely the ability to revert a change should be paramount in a dynamic environment? (data as well as config?)

And I don't use scripts to text my girlfriend when I am working late, like this guy. https://github.com/NARKOZ/hacker-scripts

[Tom Paine]

Re: I use scripts all the time

Part of the problem I have with DevOps is that there is no "preview" mode

Say what? CI should give you a preview of what to expect. If it doesn't, you're doing it wrong.

[jake]

Re: I use scripts all the time

"Customer intelligence"

I knew DevOps was a tool of the Devil^W^WMarketing!

[Herby]

Check for "errors" and make sure!

Yes, they are wonderful, but scripts should be through. Many a time I have looked at scripts only to find out that the writer is lazy n the third degree. Some examples:

Routing stderr to /dev/null

chmod 777 <file> just because it solved the "problem".

Leaving temp files floating around

Such things as "cat file | command"

No explanatory comments on what is going on.

Scripts that say "Doing xxx" when they are actually "Doing yyy".

scripts that mimic standard commands in weird ways with errors themselves (see above)

right

Unfortunately this happens with all to much regularity, and the "clean-up" usually find an error undiscovered that will really much things up if executed in the wrong/right way.

So we all fix them drawing from experience, and life goes on. (*SIGH*)