A major North American bank had a problem with its commercial loan application. The application was used by lenders to look up credit information about businesses, calculate rates, and process loans. At one point in the process, one of the pages would occasionally simply go blank. Rumor was that in some cases, if left open, it might return some of the needed data after a significant delay. In any case, it was damaging the bank’s abilities to service customers.
This had been happening for over a year when I was called in. What stood out was that the overall performance of the application wasn’t terrible. The codebase, or at least what I got to see of it, wasn’t great; about average for being offshored to the cheapest vendor. Lots of copy-paste code, half-working build system, a mis-mash of dependencies. The usual. My primary technical contact was a bright guy who knew the major technical and political inner-workings of the bank. But, he was stumped.
About the only pattern was the number of sloppy errors. For example, in the network captures there were a number of HTTP 404 errors. Generally, these would be considered innocuous; a request was made for a file, but it wasn’t found. The development team had parroted this response for many months, that these were just simple errors with no impact on functionality.
Something so small had cased a year’s worth of angst and incalculable business harm.