[lassie]

Does anyone know the background to the recent computer "glitch" that caused three UK bank computer systems to go down (Royal Bank of Scotland, NatWest Bank, Ulster Bank)?

It sounds like a single point of failure occurred in this case, which is scary stuff. I wonder where would such a point exist in a system like this?

[Necrosis]

From everywhere I have read, the bank refuses to release the exact details on what happened. Apparently it occurred during a simple patch update in the UK. They do not say where the patch was written, ie, UK or their offshore IT in India.

Needless to say, a "simple glitch" is of course the easy way for them to say "we don't know what happened". It sounds like the real problem is a faulty system in combination with the recent layoffs of experienced staff. Someone "Oked" an untested upgrade and it failed with no one around to fix it.

[lassie]

I know someone who worked for a different banking group. She project managed the production/testing of an update and then had it rolled it out worldwide with no problems. Perhaps as you say, they got rid of computing staff and this impacted on the testing which should have been done.

[Necrosis]

Yea they probably also had terrible (if at all) backup servers. Something of this magnitude has to be a dramatic fault to last this long.

Being that it's a public traded banked, I'm pretty sure the information will be released at some point. Will be very very interesting to see what they say.

[Kricket]

I haven't read anything about this particular failure, but maybe it was a core switch like it was with
To view links or images in this forum your post count must be 2 or greater. You currently have 0 posts.
. It's hard to believe that an institution of that size would leave themselves so vulnerable, but I've been on meetings where the CEOs can't believe IT wants to buy two of every expensive piece of equipment just so they can have an accurate test environment. And you mean you want to spend a whole six months to try out this upgrade on systems that aren't even in production? Just get it done already. Of course it will work with every application and every level of firmware on our multi-vendor equipment.

[lassie]

You were right, Necrosis. This is Computer weekly's interim report on the computer system failure:

To view links or images in this forum your post count must be 2 or greater. You currently have 0 posts.

[Uber10]

	Originally Posted by lassie To view links or images in this forum your post count must be 2 or greater. You currently have 0 posts.
	It sounds like a single point of failure occurred in this case, which is scary stuff. I wonder where would such a point exist in a system like this?

Shared time on a mainframe?

[plotthickens]

	Originally Posted by lassie To view links or images in this forum your post count must be 2 or greater. You currently have 0 posts.
	You were right, Necrosis. This is Computer weekly's interim report on the computer system failure: To view links or images in this forum your post count must be 2 or greater. You currently have 0 posts.

Reports suggest a faulty software patch update to the CA7 batch process scheduler used by the RBS banking group to update customer account balances. Up to 12 million people have been unable to access funds deposited in their accounts from Thursday onwards or use online banking systems. Although CA would not confirm its specific responsibilities when it came to the RBS banking group, it did admit that RBS is one of its customers. A spokeswoman for CA told Computer Weekly: “We are offering all assistance possible to help them resolve their technical issues which are highly unique to their environment.” A source close to the situation told Computer Weekly on Friday he suspected the fault was down to a software upgrade, but was surprised any changes had been made to the system on a weekday.

"Test Environment? We don need no stinkin Test Environment!"

[Photolysis]

	Originally Posted by plotthickens To view links or images in this forum your post count must be 2 or greater. You currently have 0 posts.
	"Test Environment? We don need no stinkin Test Environment!"

However, sometimes the test environment cannot reflect the real world for good reasons.

An anecdote I was given from someone who worked on a billing/payment system a long time ago: the system ran perfectly in the test environment but certain customers would experience a very strange glitch. Eventually it was discovered that a particular rare sequence of numbers in the bank account number led to the bug.

The reason for this wasn't picked up in testing - aside from its rarity - that the programmers weren't allowed to use actual customer data on the system for privacy/security reasons.

Even the very best test system has limitations.

[scorpiomover]

Happens all the time in IT. It's only amazing that it happens so rarely with banks.

Does no-one remember when Sony got all their user accounts hacked? Imagine that was your bank. ALL the customers could have lost their money. Not a few. ALL!

	Originally Posted by lassie To view links or images in this forum your post count must be 2 or greater. You currently have 0 posts.
	I know someone who worked for a different banking group. She project managed the production/testing of an update and then had it rolled it out worldwide with no problems.

In my experience, that situation is a rarity in IT, even with simple stuff. Often, I have heard people make such claims, only to find out later from others, that it was just PR spin, and that the real situation was nothing like that.

Software patch failures now happen all the time, in commercial software. Never used to. But I see it happening now all the time.