One of the (then) largest banks in America (let's call it B.A.) had some grand plans. It wanted to leapfrog the competition by building a computer system for its trust fund administration and banking that was far more advanced than anything the other banks had. This might turn into one of the most advanced computer systems in history.
In 1980 all banks had mainframe systems for their banking operations. Written in COBOL and assembly, the systems were fast enough to handle large transaction volume of the day-to-day operation. However, it was very difficult to make any modifications to the systems and nearly impossible to make large-scale enhancements or additions.
Trust Fund Administration was a particularly complicated business. You are buying and selling securities for many client. Each Client has a portfolio consisting of different kinds of securities. There are stocks, bonds, options, calls and certificates of all kinds. Each kind has a different method of calculating its value or processing it. And there are different values, book values, present time value etc. Then there are many different currencies, US$, Deutsche Mark, Yen etc. And many different kinds of reporting. And most large funds have all the above factors in combination to really complicate things. And billions of dollars under administration. You get the picture.
The basic idea was that B.A. wanted to develop a state-of-the art system from scratch that was expandable, flexible and could handle all the complications of Trust Fund accounting as well as its day-to-day banking operations. So it teamed up with a software company in Philadelphia. With the major backing of B.A. this software company started to work right away. Because this new system was supposed to be better than everything existing on the market, it seemed not a problem to abandon many pieces of traditional computer systems thinking but to do everything differently than in every existing system.
Instead of mainframe systems, a minicomputer was at the heart of the system. And where the computing power of that machine might not be enough, they could possibly be networked. Also, newer and faster versions of that minicomputer were coming out at a rapid rate. So computing power should not be a problem.
Who needs the "antiquated" COBOL and Assembly programming languages? The new system was supposed to be state-of-the-art and do complicated Trust Fund calculations. So the programming languages chosen were PL/1 and Fortran.
Who needs IBM or any of the other big-name hardware mainframe vendors? A smaller hardware vendor, PRIME Computers, was known for making some of the fastest minicomputers at the lowest prices. That way the bank got a completely proprietary operating system too, PRIMOS instead of MVS, VM or UNIX.
Who needs centralized systems? Let's spread things out! There would be a back-end processor for databases and storage. In the middle there would be a front-end processor to run the programs that would show information on the screen and handle user interaction. And at the user end there would be intelligent programmable terminals. And if one back-end processor was not enough, one could get a couple of them. The same for the front-end processors, just get a couple of them if needed.
Who needs big single monolithic programs (programs that are really one big program consisting of modules)? A computer can run many programs at once. So let's write different smaller programs for individual smaller tasks and purposes and have them "talk" to each other. There would be a program that deals with a data table and it accepts requests from other programs about what to do with it. Then there is another program that does the financial calculations and talks to it. More programs for other data tables etc. etc. Interestingly enough, these programs were called Phantoms. You could "launch" a phantom. Or you could "spawn" a whole number of phantoms. Or you could kill a phantom. And sometimes the phantoms died by themselves.
A lot of programming needed to be done to get all this stuff working together. Up to 200 programmers were working on this at the same time in Philadelphia and at other locations. And a lot of programming got done. After just 2 or 3 years most of the programming was completed. All the financials. Most of the reports. All input screens. All the interaction of the phantoms between each other. A really advanced system. Very flexible. It could handle many multiple currencies. Many different types of reporting. Many different kinds of transactions for many clients. A very sophisticated software system. It was really amazing to see that all this stuff worked together.
However, it worked very S L O W L Y . If only one or two users were using the system at the same time, it was ok. Response time on most screens was just a few seconds. That would have been tolerable. The bank had more than 2 users however. Ideally they wanted to connect all their branches in one form or another to this system with thousands of concurrent users. But just 20-50 concurrent users could drive the response time for each of them up to minutes for each step, almost paralyzing their work.
The problem seemed still solveable. Newer and faster models of that minicomputer line were coming out all the time and the software could be tweaked and optimized too. So it seemed just a matter of time until the bank could start using the system. Money was no problem either. A lot more energy was pumped into the project. More and more machines were bought and things improved.
However, it was still found that the system would not be able to handle the very high volume of day-to-day over the counter banking transactions for a long time, maybe never. Just too much volume and too slow processing. On the other hand, the most important and complicated business, the trust administration and accounting, was not such a high transaction volume as the chequing and savings accounts. The daily transactions were done at the Securities Stock Exchange and then would have to be entered and tracked in the Trust Accounting System. The bank decided to go ahead with implementing the system for Trust Accounting and Administration.
The bank was working closely with the software company and the implementation proceeded in a closely coordinated fashion. The software company would fix any problems and tweak performance issues. More and faster hardware would be purchased if needed. And the Trust Accounting System grew and grew. After less than a year all the bank's trust accounting was running on the new system. The system was running at full capacity. In the daytime input of the transactions. At night processing the transactions and reporting and backups to tapes. That took many hours and often most of the night. When the system finished processing everything in the morning, users would start entering the data for the next day. So the computers were busy most of the time and there was not much margin for error.
Now that the system was in operation, the real trouble started. Basically, because the system was running on such a tight margin, it was difficult to handle and detect errors in entering the data. Also, whenever something went wrong there were not enough resources to fix the problem. Gradually more and more mistakes entered the system. Also, because there were some links between records missing, it was very difficult to see which transactions went with which transactions. For example, if some money was spent purchasing IBM shares and some Ford shares for a client and the next day or a few days later another transaction came in where that client was paying for these shares, it was difficult to link these transactions together.
Then a number of program screw-ups were really causing havoc with the database. All of a sudden a large batch of transactions was missing but it was difficult to find out what they were. A database restore could have cost two days of downtime. On top of that, it seemed that with every day more problems entered the database than could be solved. In other words, the accounting turned into a mess. There was no way to stop. The system could not be shut down for long enough to clean up the database. Because of the slow response time, it was difficult to even predict how long it would take to attempt such a cleanup. Every business day the system needed to be up in order to follow the daily transactions. And weekends and holidays were never long enough to solve the problem.
By the time it was broadly recognized that there were big problems, the problem was very big indeed. One newspaper quoted a bank employee as saying "They couldn't reconcile anything." That's exactly what it was. There were many tens of thousands of transactions which contained essential financial data but it was impossible to tell which transaction belonged to which transaction. The problem was also that it was not a simple one to one relationship, such as one transaction of +$514.23 matching to one other transaction -$514.23. For example, a number of payment transactions could go together with a number of purchase transactions. One would have to go through billions or trillions of combinations to figure this all out.
This was of course no method of running a trust accounting business. It was impossible to see if there were any fraudulent transactions going on or if funds were missing or any other problems. The bank could not continue to operate this way any longer. It had to put up a financial bond for all the potential liabilities resulting from this accounting disaster.
The only way to stop running into more trouble on a daily basis was to sell all its thrust business to another bank. Because the Trust Accounting and Administration business can be highly profitable it quickly found another bank to buy all its customers. Having lost all its trust clients, now it still needed to clean up the accounting mix-up aftermath. That was not completely possible and the bank had lost a lot of money in these confusing times.
In the end it was estimated that the bank had lost 60 million dollars as a result of the problems with the system and bailing out of it.
So the system that had cost 20 million to get had a cost of 60 million to get rid of! It also resulted in the loss of many billions of dollars of funds in administration and made this bank from one of the biggest ones in America to one of the smaller ones.
(Interestingly enough, that American bank was not be only one suffering from the accounting screw-ups resulting from that trust accounting software. There was a small Trust Co. in Canada which had the same program and the same problems, however on a smaller scale. I happened to be working at that Company at that time. Those were certainly interesting times! See my Computer Systems Horror Story 2 about this.)
Return to For Software Developers.