Imagine people logging on their bank account and presented not with their own bank accounts but with those of completely different customers. Imagine people seeing their life savings suddenly missing from their account. This is what happened in April 2018, when a series of outages led to the meltdown of TSB’s online banking system, costing the bank £366 million in ‘post-migration charges’ and a big loss of customers. A bill of £153 million was sent to Sabis, the bank’s IT provider, for its role in the crisis. The overall cost of the software failure could increase further if fines are subsequently imposed by regulators.
The reason has been an attempted systems migration of grand proportions but –as it happened on most of IT failures - testers took the blame, although this proved unjust and unfair. The huge pressure within the project to go-live with urgency resulted in lowering test targets which obviously was not a good idea. Bad code likely set off TSB’s initial problems, but the interconnected systems of the global financial network meant that its errors were perpetuated and irreversible. After all, shifting an entire company’s records from one system to another is no mean feat.
IBM report highlights that there was little evidence of test and design documentation, performance, operational and cut-over testing that would prove production readiness and whether capacity could be managed successfully. There also was a lack of realistic test environments. It is highly unlikely that any Test Manager would have agreed that testing is complete and that all risks had been mitigated. If they had not cut corners across the project to keep deadlines, more testing would have been performed.
Slaughter and May law firm report, came also heavily critical of testing. It claims that only one of the two data centres involved in the migration had been tested and that the bank’s IT provider and responsible for the data centre, hided the lack of testing from the TSB board members, who were kept in the dark and led to believe that all was under control. Based on their report, just 12 days before the planned migration the project had a defect backlog of 5,359 defects, 840 of which were classed with severity one or two, which are the highest levels. To TSB’s disagreement, a whopping of 4,424 defects were still open when they went live.
Considering the scale of the project; 70 third-party suppliers, 1,400 people, new a new system that was not ready to support the bank’s customer base, It is obvious that a ‘big bang’ migration approach to be performed in a weekend was not the right choice. To add to the complexity, new applications were introduced, along with use of advanced microservices and active data centres, which added more risks.
Due to time and budget constraints, testing was insufficient and the lack of robust regression testing in particular, prevented the bank to catch bad code before it could be deployed to production.
Understanding how all systems will function when you do a migration project is very difficult without having machine-assisted end-to-end Regression testing. Is migration a risk to your digital transformation project? Bad code is likely to set off initial problems and eventually break the bank. Testers may seem to be the scapegoat , however it is not the testers that cause issues; they are there to help identify them in advance. There is always room for more testing if time, resources and infrastructure allowed.
Bank IT systems have become more complex to keep up with the increasing customer needs and digitization. The business is pushing for more and more changes to stay ahead of the game and optimize customer experience and convenience. The dilemma concludes on how much effort do you put into keeping things running when you have a huge pressure from the business to introduce new stuff?