Thoughts 25 Sep 2008 11:29 am
London Stock Exchange Failure
The day that the US government announced the Fannie Mae / Freddy Mac take-over the London Stock Exchange went down for over 6 hours.
At first the official line from the LSE was, “We will be investigating this and will do everything we can to make sure this doesn’t reoccur.” Now they say “It was software-related, a coincidence, due to two processes we couldn’t have foreseen,” and not caused by high-volume.
These people handle huge volumes of trade and deal in trillions of dollars. The fate of many a company could hinge on the next failure. Which is why I find the “coincidence” - “couldn’t have foreseen” statement scary.
The problem is simple: Their 100+ servers are running Windows Server 2003 with custom C# and .Net programs connecting to a Microsoft SQL Server 2000 back end.
This is the world’s third largest stock exchange and someone got suckered into running M$ crap. That it ran without crashing for as long as it did is simply amazing.
I just have to wonder what M$ had to pay to get them to use their products or was it some anonymous middle management incompetent who believed the hype and the pretty pictures?
When asked for my opinion about handling large volumes of constantly changing data I automatically suggest Oracle as the primary database. It may be expensive to own and operate, but it’s reliable under load. Put it on high-end Sun or IBM servers with massive raid arrays, institute redundant means of communication and you might not be bulletproof, but you’ll be damn close.
But they’re just dealing with a business that requires nearly instant communication and gambling with the financial lives of a few million individuals so I guess going with the “low bid” vendor is ok.
BTW: The New York Stock Exchange is starting to run Linux on its servers.