Conjecture at the moment, but the technical explanation im hearing is that an amended cics program went live last night, failed, and crashed cics. Batch probably couldnt start or had started and things needed forward recovery. I may well be off the mark, but hopefully this will get the ball rolling. They are fucking hopeless.
Posts by Cokezero
5 publicly visible posts • joined 27 Jun 2012
IT MELTDOWN ruins Cyber Monday for RBS, Natwest customers
Half the team at the heart of the RBS disaster WERE in India
Friday 29th June 2012 00:51 GMT
Re: Whether the staff were Indian (onshore or offshore) is irrelevant
Sounds like you work in IT. Sometimes people just F* up and there comes a point in Problem Management where the analysis just becomes a witch hunt. The trigger (if you pardon the CA7 pun) for this was backing out a 11.3 upgrade to CA7(performed at the weekend) on the Tuesday. Presumably there was little choice in the matter after it had been identified Monday. Either the backout is the problem , or a batch administrator cancelled a job they should have force completed in CA7, completely unrelated to the backout with ensuing chaos . One or the other, and it sounds like the former. Its very difficult for an Op or Batch Admin to wipe the queue by mistake. I repeat, sometimes people just f* up. Solution ? Two sysprogs on each software change ? You cant automate version upgrades. And i didnt even mention offshoring. Oops.
Thursday 28th June 2012 16:22 GMT
I dont think theres any question that the CA7 Upgrade was performed in the UK. Of course it was. with 20 years ca7 experience I'm still scratching my head trying to work out how an Operator or Batch Administrator could clear the entire CA7 jobqueue by mistake , assuming thats what happened. Offhosring has done me no favours, but lets be honest here. Lets assume an offshore member of staff was responsible for the mess. Are RBS and all the other banks going to stop offshoring ? Nope. It's just a witch hunt with ulterior motives.
'Inexperienced' RBS tech operative's blunder led to banking meltdown
Wednesday 27th June 2012 16:39 GMT
Fascinating that anything typed here concerning the word offshore can and will be picked up by half the worlds press. Probably covering old ground here but woul a "lowly operative in Hyderabad" really be responsible for a CA7 software upgrade? Those roles are batch admin, not sysprog. Of course ive no access to the real incident report but it takes real concerted effort by a batch administrator to remove all jobs from a CA7 queue, assuming thats what happened. If the schedule definitions are missing the batch cant run. I'd love to see that incident report but the whole offshore thing just doesnt seem to ring true. I can envisage a scenario where an Op Cancelled a failed job in Ca7 instead of doing a force complete. A cancel doesnt satisfy job requirements, it removes the whole thing and subsequent actions from a schedule. Christ knows.
Biting the hand that feeds IT
