The Anatomy of Failure

On May 7th 2010, at around 2:30 p.m. Eastern Time, the stock market went on a wild ride, dropping over 900 points in matter of just minutes. What happened? There’s lot’s of speculation, and some know more then they are willing to say. But what’s clear is that there was just the right confluence of world events, human and computer errors, and system-wide communication breakdown that triggered a mass sell-off of stocks at fantastic prices. In other words, there was a catastrophic failure during product interaction.

I’m not an investment analyst and have limited knowledge in this subject area, but I am interested in product failure. So Thursday’s stock market episode was very interesting. Here’s a little background on the events of that day broken down into steps leading to the failure.

Step 1: When the New York stock market opened on Thursday, bad news was streaming in from Europe—there were fears that Greece would ultimately default on its loans; its people were staging massive demonstrations in Athens; Euro was going down.

Step 2: In our very interconnected world, this kind of news makes investors skittish and the stock market was dropping value all morning.

Step 3: At around 2:45 in the afternoon, about 300 stocks were so off from their normal values that New York Stock Exchange executed a 60 second “slow down” to reduce volatility in the market. In today’s world of high finance, trades are executed in nanoseconds (one billionth of a second). Given the rate of such interactions, 60 seconds is a VERY long time.

Step 4: Our world of finance might be interconnected and interdependent, but the rules and regulations governing the stock exchanges and traders are not. So while New York Stock Exchange executed a “slow down,” Nasdaq and other electronic markets did not. So all of the trades were funneled through them, creating a bottleneck of sell orders, pushing the slide deeper.

Step 5: By a cosmically bad coincidence, some electronic trader (we still don’t know who), accidently typed a sell order of 15 billion dollars (that’s 15 with 9 zeros) instead of 15 million. Since every nanosecond counts, dialogue boxes that would have alerted the poor fellow with something like “Are you sure you mean to sell 40% of Procter & Gamble?” were helpfully turned off to improve system performance. It took minutes before the mistake was noticed, more minutes to figure out what to do and how to alert the market of the mistake.

Step 6: At 2:45, stocks dropped almost 1000 points. Shares of Procter & Gamble fell 10% (it’s a good thing it’s a very large company). New York Stock Exchange stopped all trading for 80 seconds.

Step 7: While naive people like me still fantasize that there are actual human beings involved in every transaction, the reality couldn’t be more different. When trades are executed on scales of nanoseconds, computer programs have to take over–we humans simply can’t think this fast. And so when the stock market paused and orders started to come back at values of $0 (that’s zero dollars, since there was nothing to sell), computer programs added a one cent transaction fee and traded stocks for a penny. For a few moments, stocks that had a value of $50 became worthless. And a few moments is an eternity for a computer.

Step 8: Panic among human traders set in. More stocks lost value. Some traders started to think that either there was World War III and they just hadn’t heard about it yet or there was a gigantic failure. Betting on failure, some tried to cash in.

Step 9: As buy orders started to come in, the stock market turned around sharply and rose for the rest of the afternoon to regain most of the lost value.

So failure was a combination of human error, computer systems, and the structure of the environment that excluded human judgment from the transaction. And at the end of the day, nothing changed. Nasdaq said it will cancel all trades for the hour between 2 and 3 in the afternoon. There will be big losers and big winners. And it could easily happen again. In fact, some are now alleging that this could all have been a maliciously orchestrated event. I don’t believe so, but then what do I know? The events of May 7th obviously create a how-to blueprint for a stock market crash—it’s not that difficult…

Conceptual Design: Most money made on the stock market today is not from the buy-and-hold strategy of the past. The money is made by watching fluctuations in the market and capitalizing on those small differences. The Internet makes the flow of information almost instantaneous, thus opening up day trading opportunities to more players. So conceptual design needs to focus on finding small opportunities in the market.

Interaction Design: Since the shift from long-term investment in a company to day trading stocks, the incentives are all wrapped up in the speed of the transaction. Anything that slows down the purchase or sale of stock results in lost profit. Interaction design needs to focus on increasing the speed of the transaction.

Interface Design: As this incident shows, mistakes are often difficult to spot when speed is of the essence. Interface is then all about error identification.

Product design strategy needs to address speed and error recognition in addition to opportunity alerts.

  2 comments for “The Anatomy of Failure

  1. May 10, 2010 at 4:03 pm

    Some public policy experts are seeking to shift investment strategy back to the “investor” rather than the day trader. To make that shift, they propose taxing each transaction by, say, 1/4 of a percent. Not a huge number if you plan to buy and hold but quite a chunk if you execute many transactions a day.

    The idea to tax trades would be a Conceptual Design decision, intended to change the behavior of traders. The Interaction Design and Interface Design changes would be significant, presumably reducing the importance of speed.

  2. May 10, 2010 at 8:16 pm

    A lot of what you said is completely accurate and you seem to understand the situation pretty well. As you know there are still issues that various people are trying to uncover. I would add one this. This was a situation of unintended consequences of systems design. I am not sure if you know how this mechanism works, but in the stock market there is this order type know as a stop loss. Here is how it works. You tell the computer that if stock A reaches point X, you don’t want to stay in the stock any longer and you want to get out no matter what. When that happens a specific type of order is sent to execute the trade to get you out, that order is called a “market order”. A market order is a type of order that says get me out at whatever the price you can.

    So here is the problem. Like you said the market was already jittery from the Greece debt situation. In your blog the spell checker made Greece “Grease” 🙂 so you should correct that. Next thing a trader make a terrible mistake. In theory proper back office controls should have prevented an order like that from every being executed (kind of like child proofing your house) these controls exist but they are not always on in big institutions that “know” what they are doing.

    Here comes the tipping point. When that “fat finger” mistake was made it made the market go down an unusual amount fast, but what happened right after is the real problem. Stop loss orders, that are all programmatic all handled by computers started the big wave of selling, and as more stop orders were placed, it naturally created a spiral affect as stop loss orders lower and lower were triggered causing the incredible tidal wave of sell orders. This incident is much like those rogue waves that they talk about on the science and discovery channel. Ships were never designed to handle a wave 20 times the size of the worst regular wave. In the market the same thing happened.

    So that is in a nutshell that is what happened. When people realized all of this was glitch based they rushed back to buy and things got stable. The thing that was made to prevent big losses ended up causing the big loss. That is how stuff works 🙂

Comments are closed.