Closing the gap between development and operations

A few years ago, when I still had what could loosely be classed in this industry as a ‘proper’ job, I happened to be sitting in a meeting – the purpose of which was to plan the deployment of a new application. And things were not going well. It didn’t help that the application had already been written, and the stated goal was …

[Bassey]

Tried working closer

We tried working closer with Ops. It went well. For a short time. Then the auditors came in and insisted that the closeness was a potential risk. Now we work on different floors with weekly meetings that happen once a month.

Us developers aren't allowed to touch anything under the Ops umbrella, even when Ops would like our help because we know more about it. It makes deployment a complete nightmare. We have the knowledge, they have the access. I read that story about the guy amputating someone's arm whilst being given instructions via text and thought it sounded terribly familiar.

[Anonymous Coward]

Not uncommon

That's a classic problem. The development team has no idea how ops work. And ops have no idea what the dev team needs to be told in order to produce a serviceable solution. On the project I work on at the moment, we've been asking ops for requirements and they have been unable to provide any for several months. So we're going with our best judgement (which may or may not be adequate).

You typically end up with a discussion like this:

ops: "we need the solution to be serviceable"

dev: "ok, so what do you need the solution to do to achieve that?"

ops: "tell us what error conditions you have and how they can be detected"

dev: "what types of errors do you want to see?"

ops: "well, we can't tell you until we see what types of errors you have"

dev: "well we can't give you the list of errors until you tel us what type of errors you need to see"

So how do you solve it? In the same way as you solve requirements: ops requirements should be given the same importance as end user requirements and the applications created by the dev team should deliver those requirements. Of course, this assumes that ops actually know what they want and are able to come up with a set of sensible requirements, which is not always the case as illustrated above. Maybe the solution is to have some people in the ops team help design the system? Or have some of the developers do real 2nd level support for the apps they write to that they understand what is needed?

Anyway, every single project I've worked on, this has been a problem and, although I've seen better and worse ways to address it, I've never yet seen a procedure that was really working.

[Beelzeebub]

Architects

I thought this type of 'smoothing' was part of the architect's role.

Oh, and doing standards, n roadmaps, n strategies for ops and dev.

[Justabloke]

interestingly...

the company I work for have solved this problem by employing me... my background is in DEV but I have experience of Tech OPs from a couple of the places I worked at previously and so here I sit with a foot in each camp.

I give the dev team an understanding of how the tech ops stuff work and I give the tech ops team an understanding of how the dev environment works.

I handle deploymenst and the more technical aspects of application support, leaving the tech ops people to look after the day to day ops stuff.

so far its all working very well.

Perhaps its the way forward?

[Anonymous Coward]

Difficult to work together

In my experience (having been on both sides of the fence over the years), the aims of operations and development aren't really compatible. Operations like software which is proven rather than cutting edge as they are measured on availability and reliability. Developers on the other hand (in my experience) are measured on productivity which generally goes hand-in-hand with new and often slightly unstable technology.

For example, I worked at a software house that adopted VS 2005 a year before its release in order to build our next generation product. We also adopted the beta of SQL Server 2005. All of this stuff crashed a lot and as we got closer to go live, those responsible for deploying and managing the software grew increasingly concerned about the stability of the product and platform (ASP.Net 2 Beta kept killing sessions early when it encountered an error somewhere in the framework).

The problem being that it was significantly harder in our geographical area to hire developers than it was to hire operations staff, so in order for the development team to get the product to market in time, we had to use unproven technology and hope it would be stabilised in time for release (it was, luckily). However for the people responsible for deploying this, all they could see was a nightmare scenario coming and fought to try to stop the deveopment team from using beta tech ever again.

[David Grierson]

Aren't you talking about systems integration?

What you're talking about used to be called systems integration - a role which, for many companies, no longer actually exists.

In fact I was involved in an exceptionally similar conversation just 3 weeks ago:

Dev Mgr: "Our dev is in our private SIT environment and we want it in prod before the Christmas change freeze in 3 weeks."

Me: "Don't think that's going to happen - we haven't even got it in UAT."

Dev Mgr: "This needs to be the top priority then to get it UAT'ed and in prod before the Christmas change freeze in 3 weeks."

Me: "Don't think that's going to happen - we've got end-user support priorities. What's involved in your deployment plan?"

Dev Mgr: "We don't have one - we need to define it so that it can go into prod before the Christmas change freeze in 3 weeks."

We're now 3 weeks later and we have only just defined who will be responsible for the roll-out (me) and you can be damn sure it's going to have a fully defined deployment plan before it even gets near the UAT environment.

[Anonymous Coward]

My experience

I think separation of development and operations is essential. Operations need software that works. That's the task of the development team, to construct the software and to test it, debug it. That can not be done by operations.

I've worked in an investment bank, which in theory had clear boundaries between development and operations but I found myself doing both. When dealing with large sums of money, operations comes first and always did, consequently, application development was stifled.

Fortunately, the development projects I was engaged in were quite small and no great dependency on them, deadlines for them weren't business critical, if I was a bit late, then so be it, there wasn't much training to be given so no real hardship.

Now, there needs to be close communication between operations and development, right up at the start of the project definition, requirements analysis phase.

Operations will be using the system so they should be involved in establishing the requirements, and they should be involved during development, particularly so in factory acceptance testing of the system.

Yes, there needs to be a line of demarquation between the two, but the development should not continue without the input and the involvement of the operations.

For one very large military project I worked on, when the decision was made to involve operations staff from the client in the development team's testing, this made a huge difference to the success of the system. Requirements problems, software bugs were caught much earlier on in the life cycle by the operations staff, it also acted as a form of training for them because they were able to get some hands-on on the system before it was thrown over the fence to them and expected to use it operationally.

Having operations involved in that way, helps win them over to using a new system and does help improve morale.

[Anonymous Coward]

let us know if your opinions differ

I work in ops for a large UK based investment bank and I'd say a large majority of new applications here would fall into the category of ops hearing about them only once development is complete.

This is partly the fault of operations as standards are not well defined but also in the business aligned development teams who sometimes have a mindset of "we will do whatever the hell we like and you will support it".

I've worked in other banks that have similar problems though not quite as extreme as this fine organisation.

[Dave]

Size matters?

My own experiences are from much smaller departments where in most cases Ops WAS the development team but with a couple of the devs more in the ops side than dev. This worked well for one place but it also helped that the Ops Director did a bloody good job of handling everything and buffered the whole of Ops from the rest of the company leaving the devs and techies to get on with things. This worked for a small team (about 11 or 12 tops) though I dare say with larger teams there would need to be a bigger separation.

Another place I worked did have stricter divisions between Ops and the devs to the point where as a dev I had no DB access at all even to dev ("what do you need that for?"). The dedicated DBAs would optimise the database for the devs; not sure quite how that would work when the DBAs were not involved at all with the application development so had no knowledge of HOW the DB was being used. It was a nightmare (for me!) and I didn't stay long.

Without seeing how much larger companies operate I don't feel capable of answering for those situations, but certainly for small(ish) teams (as I mentioned already) it is more beneficial to have Ops and Devs together with a single Operations Director (IMHO). Even with this though, it is no substitute for regular meetings, road-mapping and planning sessions.

[Steve Davies]

plus ca change - just chuck it over the fence

So (it appears) there was no consideration of the non-functional requirements at all during design and code ? Who signed off the design ? Smack them. Any Production Acceptance Criteria ? If so are they used to make judgements/decisions or just treated as 'bumph' ? As for configuration management spanning the groups of course it does IF it is planned and you have defined level 1, 2 and 3 activities and control of your baselines. Instead of just incenting the dev folks on meeting dates incent them on reported problems at every stage after UAT (that includes performance testing) and things WILL change. And keep the operations implentations diary filled out as far into the future as necessary and visible to all. For example at a weekly delivery meetings that involves delivery and development - it is develpment that resolve the application problems after all isn't it ? Its not rocket science to do it properly nor is it difficult. FFS these stories are enough to make one weep at the organisational ineptitude. A favourite phrase of mine to ponder on "Development is vital for the business - but Production pays the bills".

[Anonymous Coward]

Operations

tends to be day to day running.

The best solution is to hire a development team for operations, operations remit extends beyond IT,. That development team becomes IT Operations, and they manage the admin and hell desk, supporting with fast development solutions, and also interfaces with IT development in the company, problem solved.

What you want to do is move those with knowledge to the forefront of any business, and push those with limited knowledge out or into ancillary roles, quite simple.

No one wants a buffer, people want to be in the loop and be paid appropriately for that, only an idiot wants a buffer or a nanny, and only idiots are buffers or nannies, well you do the logic there.

[Richard Cottrill]

Big companies are the worst

I tend to think that development-side people are cowboys, and operations-side people are librarians. Their various neuroses are absolutely vital to a big IT organisation.

My 2c is that every architect (<spit!>) I've met has come exclusively through development, and so they're usually the biggest cowboys of all. In this case everyone suffers.

Mine's a drizabone with pockets stuffed with index cards.

[Anonymous Coward]

No, ops are customs officers

Poorly paid, bored, bitter and finding some meagre self worth in subjecting the development team to full-cavity go-live checklist reviews

[Anonymous Coward]

You build it, you run it

Unconventional as it is today, I have seen "You build it, you run it" work in practice at large enterprise scale.

By having the development team own the operation of their software, you get more resilience built into the software (developers don't want paged in the middle of the night). By retaining ownership rather than "throwing it over the wall", the application benefits from programmatic, ongoing improvements. I've heard the argument that developers will be too caught up with operational activities to get on with new development but in practice I've found the development team will quickly fix the operational problems at root cause.

I still think it's worth having dedicated operations teams to look after the hardware and network infrastructure but not the applications.

[Anonymous Coward]

Not uncommon, but the solution is simple

A developers job is to automate the storing and processing of information, digitally.

The users job is to provide this information to the software and to instantiate the software processes in a particular order (whatever that might be). Its worth noting the 99% of users need to be trained before they can do this.

Developers know how to automate stuff digitally, but they don't (or probably don't) know what information or processing needs to take place.

Take a banking system for example.

The tellers out front probably shouldn't be defining the processes or information. They are the users of the system, and more often than not that means they are just a dependant-monkey of the automated system once its done, they don't have knowledge of the magic that happens underneath...

The devs need a qualified bank-accountant to define the processes of things like currency convertion, tax formulates, balancing credits and debts, interest loop holes etc.

At the end of the day as you can see... the devs basically must become a qualified (insert domain expert label here) themselves to make the system.

If nobody in the company understands the processes, or how to define and communicate them, or you don't have devs that can learn them; then you have a BIG problem that will result in failure everytime.

[n]

ywned

That's nothing....this one time,at band camp....i took a bear into the woods....and you'll never guess what he did.