Jump to content
House Price Crash Forum

'inexperienced' Rbs Tech Operative's Blunder Led To Banking Meltdown


Recommended Posts

http://www.theregister.co.uk/2012/06/26/rbs_natwest_ca_technologies_outsourcing/

A serious error committed by an "inexperienced operative" caused the IT meltdown which crippled the RBS banks last week, a source familiar with the matter has told The Register. Job adverts show that at least some of the team responsible for the blunder were recruited earlier this year in India following IT job cuts at RBS in the UK.

Following our revelation yesterday that a bungled update to CA-7 batch processing software used by RBS lay behind the collapse, further details have emerged. According to a Register source who worked at RBS for several years, an inexperienced operative made a major error while performing the relatively routine task of backing out of an upgrade to the CA-7 tool. It is normal to find that a software update has caused a problem; IT staff expect to back out in such cases.

But in the process of backing out a major blunder was committed, according to our source. It was this error which made the task of restoring services so prolonged:

When they did the back-out, a major error was made. An inexperienced person cleared the whole queue ... they erased all the scheduling.

That created a large backlog as all the wiped information had to be re-inputted to the system and reprocessed. A complicated legacy mainframe system at RBS and a team inexperienced in its quirks made the problem harder to fix, our source adds.

So was this someone inexperienced in the UK or India? Combination of both?

Will Hesters claim that this wasn't an outsourcing / cost cutting problem stand up? If it was cost cutting that is to blame will the govt fire him?

Link to post
Share on other sites
  • Replies 62
  • Created
  • Last Reply

Top Posters In This Topic

Top Posters In This Topic

Guess what?

RBS computer failure 'caused by inexperienced operative in India'

The computer glitch at the Royal Bank of Scotland which left millions of customers unable to access their accounts could have been caused by just one junior technician in India, it was suggested last night.

Stephen Hester, chief executive of RBS, said that there was “no evidence” that outsourcing had caused the problems

By Hannah Furness6:45AM BST 27 Jun 2012

The “inexperienced operative” accidentally wiped information during a routine software upgrade, it has been claimed.

The member of staff, who was working on the programme for the Royal Bank of Scotland, NatWest and Ulster Bank, is believed to have been based in Hyderabad, India.

According to technology website The Register, at least some of the team responsible for the error were recruited in India following redundancies in the department in the UK.

Unions have already blamed the fiasco on the decision to outsource much of the company’s IT jobs, as Indian staff are paid as little as £9,000, compared with £50,000 their British counterparts were paid.

Stephen Hester, chief executive of RBS, said that there was “no evidence” that outsourcing had caused the problems.

RELATED ARTICLES

RBS cancels Wimbledon corporate hospitality 27 Jun 2012

Goodwin: we weren't at fault in RBS failure 26 Jun 2012

NatWest computer glitch could drag on for weeks 26 Jun 2012

NatWest glitch 'like motorway accident' 25 Jun 2012

RBS computer glitch condemns man to cells 25 Jun 2012

RBS glitch 'well on the way' to being fixed, says chief 25 Jun 2012

He said: “The IT centre in Edinburgh is our main centre, it is nothing to do with overseas. Things go wrong. Things go wrong in technology.

“We have to learn the lessons from what went wrong here and try to make then less likely to happen in the future.”

The error is understood to have occurred after a software update froze part of the banks’ computer systems last Wednesday, affecting 17 million customers.

Although the problem was resolved on Friday, it created a backlog of more than 100 million transactions that were not paid in or out of bank accounts as they should have been.

Deleted information then had to be painstakingly re-entered into the bank group’s computer system.

A source, who worked for RBS for “several years”, told the Register an “inexperienced operative” had made an error while performing the relatively routine task of backing out of an upgrade.

He said: “When they did the back-out, a major error was made. An inexperienced person cleared the whole queue ... they erased all the scheduling.”

RBS have not yet commented on the claims.

The bank has already promised customers will be reimbursed for the cost of fines or late payment fees incurred as a result of the delays. Banking experts said that the cost to RBS of dealing with the IT problems, including extra staff costs as well as the money to reimburse customers, is already likely to be between £50 million and £100 million.

Mr Hester has also admitted that the bonuses of senior members of RBS staff are likely to be reduced because of the incident.

He said: “All of us are judged in part by customer service, from me on downwards. And we should be.”

Earlier this year Mr Hester was forced to waive a share bonus worth £963,000 after a public outcry and political pressure.

The Financial Services Authority said: “We will expect RBS-NatWest to provide us with a complete account of the issues once this is fully resolved and to take any necessary steps to ensure that the risks of these problems occurring again are addressed.”

http://www.telegraph.co.uk/finance/personalfinance/consumertips/banking/9358252/RBS-computer-failure-caused-by-inexperienced-operative-in-India.html

Link to post
Share on other sites

not sure what a queue has to do with a reverse of a system, which was stopped.

and there was no backup of the queue?

methinks not.

sounds like the days scheduling was deleted (the queue) when they backed out the scheduler change. You could re-apply the schedule, but they probably didn't notice until the next day. It's all automated these days you see!

Link to post
Share on other sites

This has been such a problem that they have to blame a person, not the process/system. Same happens when a plane/chopper goes down - it's less problematic to blame the pilot (even posthumously) than admit there's a generic problem with the airframe.

If someone came out after this debacle and said "our systems are antiquated and not up to the job" there'd be real problems.

it is someones fault though. CA7 doesn't make mistakes. It does what you tell it :)

Link to post
Share on other sites

it is someones fault though. CA7 doesn't make mistakes. It does what you tell it :)

Yep computers don't make errors, somebody somewhere has screwed up and either pressed the wrong button or coded the script wrong.

However I like how it's some junior who did it, if they are inexperienced wtf was where the senior managers doing at the time?

Link to post
Share on other sites

Yep computers don't make errors, somebody somewhere has screwed up and either pressed the wrong button or coded the script wrong.

However I like how it's some junior who did it, if they are inexperienced wtf was where the senior managers doing at the time?

I wouldn't for one second think a senior manager would know anything about a batch scheduler.

What amazes me is that the Operations dept didn't realise there was no jobs on the queue!

Link to post
Share on other sites

sounds like the days scheduling was deleted (the queue) when they backed out the scheduler change. You could re-apply the schedule, but they probably didn't notice until the next day. It's all automated these days you see!

And they put the guy just in from the job centre to do mission critical button pressing....

Look, if you try to delete a file in Windows, you are hit with a pile of "are you sures".

Are we saying these £1000 per day programmers never put an "are you sure...get supervisor code before transacting" warning in?

Link to post
Share on other sites

This story is evidence that job=slavery, if a company are willing to risk their entire database on cheap wage monkeys then there is a pathalogical obsession with cutting the workforce/ labour cost meaning absolutely zero loyalty to staff.

You are better off earning less than minimum wage and maintaining your DIGINITY than working for almost ANY company out there. Unless you are the kind of person that is so skilled/ saught after that you can walk straight from one job to the next on the same salary then your job is a disservice and amounts to opting in for slavery.

I don't think in the old days people worked for JUST the money, they worked for the STABLE life that most jobs used to give people.

Link to post
Share on other sites

And they put the guy just in from the job centre to do mission critical button pressing....

Look, if you try to delete a file in Windows, you are hit with a pile of "are you sures".

Are we saying these £1000 per day programmers never put an "are you sure...get supervisor code before transacting" warning in?

If you are given authority to use these functions in CA-7 then you can do what you want.

Someone who didn't know what they were doing were given such access.

Link to post
Share on other sites

Perhaps they just assumed it was uber efficient? :lol: They had just eliminated queues?

hmmmm a source tells me that they had upgraded ca7 without applying tolerances patches. when it fails it causes a bit of a problem especailly as no backups of the initial system had been made :ph34r:

Link to post
Share on other sites

It stinks of a logical error to me.

They would have a backup of the batch schedule, and even if they didn't, it could be re-created reasonably easily. A the very least they would have the schedule at the DR site. OK, they could have been so incompetent that they didn't have a backup, they didn't have any documentation and they had replication running for the schedule database/file system. That would have been stupid.

I think they deleted some jobs, then ran the rest. Batch is built on dependencies: job 2 does not happen till job 1 is done. Take away Job 1, and Job 2 just gets going. So a zillion database updates probably were made in error, a load of real transactions layered on top of it, and then you are in the "unpick job from hell".

I wouldn't for one second think a senior manager would know anything about a batch scheduler.

Your average senior manager cut their teeth on batch schedulers, and sat up late in the early 90s testing overnight batch. It is the younger guys who wouldn't know what a scheduler is - hell, most of them don't even know what batch is....

Edited by rxe
Link to post
Share on other sites

This has been such a problem that they have to blame a person, not the process/system. Same happens when a plane/chopper goes down - it's less problematic to blame the pilot (even posthumously) than admit there's a generic problem with the airframe.

If someone came out after this debacle and said "our systems are antiquated and not up to the job" there'd be real problems.

+ 1

A system can't be vulnerable to one person, backups must assure that - and backups located in different locations. And in the case of international banks, these backups should even be located in a different country, for extra security (all eggs in one basket and all that).

Link to post
Share on other sites

And they put the guy just in from the job centre to do mission critical button pressing....

Look, if you try to delete a file in Windows, you are hit with a pile of "are you sures".

Are we saying these £1000 per day programmers never put an "are you sure...get supervisor code before transacting" warning in?

Clearly you think too much. The majority of people when presented with yes/no just hit yes and don't bother reading.

One of our managers had an infected PC the AV was telling them it had found a virus and they just hit OK to clear the message, never once did they actually read the message even though it was frequently popping up for several weeks. I only discovered this by chance as I was talking to them I saw the message pop up on screen and they immediately pressed OK, at which point I said what was that and they replied it had constantly been appearing but had no idea what it was saying and they just pressed OK every time and the message went away.

Link to post
Share on other sites

It stinks of a logical error to me.

They would have a backup of the batch schedule, and even if they didn't, it could be re-created reasonably easily. A the very least they would have the schedule at the DR site. OK, they could have been so incompetent that they didn't have a backup, they didn't have any documentation and they had replication running for the schedule database/file system. That would have been stupid.

I think they deleted some jobs, then ran the rest. Batch is built on dependencies: job 2 does not happen till job 1 is done. Take away Job 1, and Job 2 just gets going. So a zillion database updates probably were made in error, a load of real transactions layered on top of it, and then you are in the "unpick job from hell".

Your average senior manager cut their teeth on batch schedulers, and sat up late in the early 90s testing overnight batch. It is the younger guys who wouldn't know what a scheduler is - hell, most of them don't even know what batch is....

not in my company! all the senior managers have been brought in from external companies and it would appear none of them have a m/frame background!

Link to post
Share on other sites

As long as they can blame some minion Hester's bonus is safe, no doubt this shouldn't affect his inevitable knighthood/peerage.

They'll only start to sweat if the MSM is allowed to ask the obvious, why were inexperienced staff allowed to do this in the first place. Truth is the media dept will have earned their bonuses if they can isolate the blame to this small team and defend the image of the higher management.

Link to post
Share on other sites

If you are given authority to use these functions in CA-7 then you can do what you want.

Someone who didn't know what they were doing were given such access.

then the fault lies with the access giver.

That would be Board level responsibility.

Of course, the buck will stop with the tea lady.

Link to post
Share on other sites

One operative can bring down a huge organisation. Something is badly wrong there.

+ 1

As long as they can blame some minion Hester's bonus is safe, no doubt this shouldn't affect his inevitable knighthood/peerage.

They'll only start to sweat if the MSM is allowed to ask the obvious, why were inexperienced staff allowed to do this in the first place. Truth is the media dept will have earned their bonuses if they can isolate the blame to this small team and defend the image of the higher management.

+ 1

Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
  • Recently Browsing   0 members

    No registered users viewing this page.

  • 433 Brexit, House prices and Summer 2020

    1. 1. Including the effects Brexit, where do you think average UK house prices will be relative to now in June 2020?


      • down 5% +
      • down 2.5%
      • Even
      • up 2.5%
      • up 5%



×
×
  • Create New...

Important Information

We have placed cookies on your device to help make this website better. You can adjust your cookie settings, otherwise we'll assume you're okay to continue.