Never do something in IT that ends up in the newspaper and embarrasses your boss. That’s the second rule of business. The first rule is ‘Never Surprise your Boss.’
It appears that the Bank of America team that manages the on-line banking website system forgot this. They did a feature update AND a platform migration at the same time. This works only when it is done with careful planning and a lot of testing.
I do not know if the server platform migration was a RISC to RISC migration or a RISC to IA migration. Either way, it appears that basic steps were overlooked.
The first basic step is planning. Plan for how the application upgrades intersect with the platform migration. Plan to ensure that the application upgrades will not burden the overall service to the end customer. Plan the day of the release. Conduct a Proof of Concept and test the whole configuration in there before Release To Market (RTM).
Enterprises know the traffic volumes for their web sites. In the case of a bank, they know that traffic increases at the beginning of the month when paychecks, and Social Security comes in and a lot of bills are paid. In the case of a retail firm the web site and back end need to be locked down by Halloween to shake out problems before ‘Black Friday’. I did a retail application (RETEK) RISC to IA migration starting in August and we were hard pressed to get everything done by early November.
Following my blog theme of RISC to IA migration testing is crucial. Test after you complete the PoC to determine if the new hardware really handles the load. Frequently, you may discover that you need new hardware for your production system. This is the hardware you target for the dress rehearsal of the production migration. Test it to ensure that the application meets the firms Service Level Agreements (SLAs) and that the eventual production migration won’t end up in the paper.
A quick perusal of the web shows that there are a wide range of testing tools for web sites and application performance. I don't recommend any particular tool; testing tool selection must take into account your budget and your environment.
Plan the testing to have the application hit by more users than expected in production. With some testing tools, this can get expensive. But what is the cost of an embarrassment where the application doesn’t perform and it causes you to lose customers?
Testing can reveal problems before the application is released. One firm had a large Bill of Materials for their product all in one child table in an Oracle database. The BOM explosion was perfectly fine in the original environment but testing revealed that performance was terrible on the new, more powerful platform. Since this was done prior to the release of the application to the users, root cause analysis could be done and it did not affect the business. (The problem was an instruction in one processor that wasn’t in the other processor.)
Once you complete testing, the team can ensure that the production migration will result in adequate performance to meet the firm’s requirements. Once completed, the team can plan for the production migration. Plan carefully -- eliminate the critical dates like the end of the month for a bank or after Halloween for a retail firm or around a launch date for firms working with NASA. Most enterprises know when the application environment should remain frozen. Give yourself a week or so on either side of the dates for slip-ups and other unforeseen occurrence, and give yourself time to test the final migration release if you can.
The world lost a leader this week. Many of the comments about Steve Jobs point back to this commercial as a key to understanding him. I agree with them.