The $12 Trillion Fat Finger: How A "Glitch" Nearly Crashed The Global Financial System - A True Story

Submitted by Tyler Durden on 08/08/2015 17:43 -0400

http://www.zerohedge.com/news/2015-08-08/12-trillion-fat-finger-how-glitch-nearly-crashed-global-financials-system-true-story

 For all the talk of how the financial world nearly ended in the aftermath of first the Lehman bankruptcy, then the money market freeze, and culminating with the AIG bailout, and how bubble after Fed bubble has made the entire financial system as brittle as the weakest counterparty in the collateral chain of some $100 trillion in shadow liabilities, the truth is that despite all the "macroprudential" planning and preparations, all the world's credit, housing, stock, and illiquidity bubbles may be nothing when compared to the oldest "glitch" in the book: a simple cascading error which ends up taking down the entire system.

Like what happened in the great quant blow up August 2007.

For those who may not recall the specific details of how the "quant crash" nearly wiped out all algo and quant trading hedge funds and strats in a matter of hours if not minutes, leading to tens of billions in capital losses, here is a reminder, and a warning that the official goalseeked crisis narrative "after" the fact is merely there to hide the embarrassment of just how close to total collapse the global financial system is at any given moment.

The following is a true story (courtesy of b3ta) from the archives, going all the way back to 2007:

I.T. is a minefield for expensive mistakes

There's so many different ways to screw up. The best you can hope for in a support role is to be invisible. If anyone notices your support team at all, you can rest assured it's because someone has made a mistake. I've worked for three major investment banks, but at the first place I witnessed one of the most impressive mistakes I'm ever likely to see in my career. I was part of the sales and trading production support team, but thankfully it wasn't me who made this grave error of judgement...

(I'll delve into obnoxious levels of detail here to add color and context if you're interested. If not, just skip to the next chunk, you impatient git)

This bank had pioneered a process called straight-through processing (STP) which removes the normal manual processes of placement, checking, settling and clearing of trades. Trades done in the global marketplace typically have a 5-day clearing period to allow for all the paperwork and book-keeping to be done. This elaborate system allowed same-day settlement, something never previously possible. The bank had achieved this over a period of six years by developing a computer system with a degree of complexity that rivalled SkyNet. By 2006 it also probably had enough processing power to become self-aware, and the storage requirements were absolutely colossal. It consisted of hundreds of bleeding edge compute-farm blade servers, several £multi-million top-end database servers and the project had over 300 staff just to keep it running. To put that into perspective, the storage for this one system (one of about 500 major trading systems at the bank) represented over 80% of the total storage used within the company. The equivalent of 100 DVD's worth of raw data entered the databases each day as it handled over a million inter-bank trades, each ranging in value from a few hundred thousand dollars to multi-billion dollar equity deals. This thing was BIG.

You'd think such a critically important and expensive system would run on the finest, fault-tolerant hardware and software. Unfortunately, it had grown somewhat organically over the years, with bits being added here, there and everywhere. There were parts of this system that no-one understood any more, as the original, lazy developers had moved company, emigrated or *died* without documenting their work. I doubt they ever predicted the monster it would eventually become.

A colleague of mine one day decided to perform a change during the day without authorisation, which was foolish, but not uncommon. It was a trivial change to add yet more storage and he'd done it many times before so he was confident about it. The guy was only trying to be helpful to the besieged developers, who were constantly under pressure to keep the wretched thing moving as it got more bloated each day, like an electronic ‘Mr Creosote’.

As my friend applied his change that morning, he triggered a bug in a notoriously crap script responsible for bringing new data disks online. The script had been coded in-house as this saved the bank about £300 per year on licensing fees for the official ‘storage agents’ provided by the vendor. Money that, in hindsight, would perhaps have been better spent instead of pocketed. The homebrew code took one look at the new configuration and immediately spazzed out. This monged scrap of pisspoor geek-scribble had decided the best course of action was to bring down the production end of the system and bring online the disaster recovery (DR) end, which is normal behaviour when it detects a catastrophic 'failure'. It’s designed to bring up the working side of the setup as quickly as possible. Sadly, what with this system being fully-replicated at both sites (to [cough] ensure seamless recovery), the exact same bug was almost instantly triggered on the DR end, so in under a minute, the hateful script had taken offline the entire system in much the same manner as chucking a spanner into a running engine might stop a car. The databases, as always, were flushing their precious data onto many different disks as this happened, so massive, irreversible data corruption occurred. That was it, the biggest computer system in the bank, maybe even the world, was down.

And it wasn't coming back up again quickly.

(OK, detail over. Calm down)

At the time this failure occurred there was more than $12 TRILLION of trades at various stages of the settlement process in the system. This represented around 20% of ALL trades on the global stock market, as other banks had started to plug into this behemoth and use its capabilities themselves. If those trades were not settled within the agreed timeframe, the bank would be liable for penalties on each and every one, the resulting fines would eclipse the market capital of the company, and so it would go out of business. Just like that.

My team dropped everything it was doing and spent 4 solid, brutal hours recovering each component of the system in a desperate effort to coax the stubborn silicon back online. After a short time, the head of the European Central Bank (ECB) was on a crisis call with our company CEO, demanding status updates as to why so many trades were failing that day. Allegedly (as we were later told), the volume of financial goodies contained within this beast was so great that failure to clear the tradeswould have had a significant negative effect on the value of the Euro currency. This one fuckup almost started a global economic crisis on a scale similar to the recent (and ongoing) sub-prime credit crash. With two hours to spare before the ECB would be forced to go public by adjusting the Euro exchange rate to compensate, the system was up and running, but barely. We each manned a critical sub-component and diverted all resources into the clearing engines. The developers set the system to prioritise trades on value. Everything else on those servers was switched off to ensure every available CPU cycle and disk operation could be utilised. It saturated those machines with processing while we watched in silence, unable to influence the outcome at all.

Incredibly, the largest proportion of the high-value transactions had cleared by the close of business deadline, and disaster was averted by the most "wafer-thin" margin. Despite this, the outstanding lower-value trades still cost the bank more than $100m in fines. Amazingly, to this day only a handful of people actually understand the true source of those penalties on the end-of-year shareholder report.Reputation is king in the world of banking and all concerned --including me-- were instructed quite explicitly to keep schtum. Naturally, I *can’t* identify the bank in question, but if you’re still curious, gaz me and I’ll point you in the right direction…

Epilogue… The bank stumped up for proper scripts pretty quickly but the poor sap who started this ball of shit rolling was fired in a pompous ceremony of blame the next day, which was rather unfair as it was dodgy coding which had really caused the problem. The company rationale was that every blaze needs a spark to start it, and he was going to be the one they would scapegoat. That was one of the major reasons I chose to leave the company (but not before giving the global head of technology a dressing down at our Christmas party… that’s another QOTW altogether). Even today my errant mate is one of the only people who properly understands most of that preposterous computer system, so he had his job back within six months -- but at a higher rate than before :-)

Conclusion: most banks are insane and they never do anything to fix problems until *after* it costs them uber-money. Did I hear you mention length? 100 million dollar bills in fines laid end-to-end is about 9,500 miles long according to Google calculator.

* * *

And here is Zero Hedge's conclusion: the next time you think all those paper reps and warranties to claims on billions if not trillions of assets, are safe and sound in some massively redundant hard disk array, think again.

In Dramatic Decision Judge Finds Fed Bailout Of AIG Was "Illegal", Government "Violated Federal Reserve Act"

Submitted by Tyler Durden on 06/15/2015 16:44 -0400
http://www.zerohedge.com/news/2015-06-15/dramatic-decision-judge-finds-fed-bailout-aig-was-illegal-government-violated-federa
Earlier today, former AIG head Hank Greenberg's long-running legal battle of the US government came to a dramatic end when in a 75-page ruling,  U.S. Court of Claims Judge Thomas Wheeler found that Greenberg was indeed correct in claiming the government overstepped its legal boundaries in its "unduly harsh treatment of AIG in comparison to other institutions" which was "misguided and had no legitimate purpose."
But because “the question is not whether this treatment was inequitable or unfair, but whether the government’s actions created a legal right of recovery for AIG’s shareholders" Wheeler found that Greenberg was not owed any money as AIG would have gone bankrupt without the government's forced intervention. Greenberg was seeking at least $25 billion in damages for shareholders.
The reason for the case is that years after the initial $85 billion bailout which eventually ballooned to $182 billion, AIG - with the government's explicit backstop and thus zero credit risk - managed to repay the government bailout funds and the government with a $22.7 billion profit. Greenberg argued that the pre-bailout equity holders deserved a piece of the pie, very much the same way that Fannie and Freddie stakeholders are also arguing they too deserve a piece of the post-government bailout pie.
However, “in the end, the Achilles’ heel of Starr’s case is that, if not for the Government’s intervention, AIG would have filed for bankruptcy. In a bankruptcy proceeding, AIG’s shareholders would most likely have lost 100 percent of their stock value" the judge found, and admitted that the pre-government bailout equity value of financial companies - since all of them were facing bankruptcy without a bailout - was zero.  Whether this opens up the door to a class action lawsuit by all those who were short financials into the bailout and were then squeezed by the Fed's bailout which the court has found to be an "illegal exaction" remains to be seen.
Here are the key sections from the court ruling:

The weight of the evidence demonstrates that the Government treated AIG much more harshly than other institutions in need of financial assistance. In September 2008, AIG’s international insurance subsidiaries were thriving and profitable, but  its Financial Products Division experienced a severe liquidity shortage due to the collapse of the housing market. Other major institutions, such as Morgan Stanley, Goldman Sachs, and Bank of America, encountered similar liquidity shortages. Thus, while the Government publicly singled out AIG as the poster child for causing the September 2008 economic crisis (Paulson, Tr. 1254-55), the evidence supports a conclusion that AIG actually was less responsible for the crisis than other major institutions.

Well, there was Lehman too, whose stock most certainly went to zero and which never got a government bailout but that was to be expected: after all Goldman needed to eliminate its biggest fixed income competitor at the time, and what better way than to wipe it out completely.
Wheeler continues:

The notorious credit default swap transactions were very low risk in a thriving housing market, but they quickly became very high risk when the bottom fell out of this market. Many entities engaged in these transactions, not just AIG. The Government’s justification for taking control of AIG’s ownership and running its business operations appears to have been entirely misplaced. The Government did not demand shareholder equity, high interest rates, or voting control of any entity except AIG. Indeed, with the exception of AIG, the Government has never demanded equity ownership from a borrower in the 75-year history of Section 13(3) of the Federal Reserve Act. Paulson, Tr. 1235-36; Bernanke, Tr. 1989-90.

In other words, there has never been a Fed-mediated nationalization of a private corporation prior to 2008. Which is accurate. It is also illegal according to the court, a ruling that may have dramatic repercussions for all future government/Fed bailouts of banks that Goldman deems relevant.

Starr alleges in its own right and on behalf of other AIG shareholders that the Government’s actions in acquiring control of AIG constituted a taking without just compensation and an illegal exaction, both in violation of the Fifth Amendment to the U.S. Constitution.... Having considered the entire record, the Court finds in Starr’s favor on the illegal exaction claim.

It is not quite clear why the Fed is equivalent to the Government in this case but we'll just let that slide.
Here are the details:

With the approval of the Board of Governors, the Federal Reserve Bank of New York had the authority to serve as a lender of last resort under Section 13(3) of the Federal Reserve Act in a time of “unusual and exigent circumstances,” 12 U.S.C. § 343 (2006), and to establish an interest rate “fixed with a view of accommodating commerce and business,” 12 U.S.C. § 357. However, Section 13(3) did not authorize the Federal Reserve Bank to acquire a borrower’s equity as consideration for the loan. Although the Bank may exercise “all powers specifically granted by the provisions of this chapter and such incidental powers as shall be necessary to carry on the business of banking within the limitations prescribed by this chapter,” 12 U.S.C. § 341, this language does not authorize the taking of equity.

So if they Fed is not authorized to "take equity", does that mean that the NY Fed trading desk at Liberty 33 or its backup desk in Chicago, also known as the "Plunge Protection Team" will have to do a firesale of all its stock, E-mini, and ETF holdings obtained as a result of levitating the market ever higher for the past 7 years? Inquiring minds demand to know.
The good news is that while the Fed's bailout of AIG was illegal, at least it was not unconstitutional, as that particular pathway would have likely led to that Constitutional "Expert", the president of the US, to get involved and opine on the "fairness" of a Fed bailout now and in the future.

A ruling in Starr’s favor on the illegal exaction claim, finding that the Government’s takeover of AIG was unauthorized, means that Starr’s Fifth Amendment taking claim necessarily must fail. If the Government’s actions were not authorized, there can be no Fifth Amendment taking claim.... Thus, a claim cannot be both an illegal exaction (based upon unauthorized action), and a taking (based upon authorized action).

Furthermore, the Court found that like in the BofA negotations over the Merrill rescue, the government effectively strongarmed AIG management into accepting the terms of the bailout it proposed:

The Government defends on the basis that AIG voluntarily accepted the terms of the proposed rescue, which it says would defeat Starr’s claim regardless of whether the challenged actions were authorized or unauthorized. While it is true that AIG’s Board of Directors voted to accept the Government’s proposed terms on September 16, 2008 to avoid bankruptcy, the board’s decision resulted from a complete mismatch of negotiating leverage in which the Government could and did force AIG to accept whatever punitive terms were proposed. No matter how rationally AIG’s Board addressed its alternatives that night, and notwithstanding that AIG had a team of outstanding professional advisers, the fact remains that AIG was at the Government’s mercy.

This would be especially accurate if an armed drone was flying outside of AIG HQ's during the "negotiation."
And yet, despite this clearly favorable to Greenberg ruling, the Court did not award him any damages. Why? For the simple reason that AIG was already effectively broke when the government stepped in, and as such there was be no residual equity value going into Lehman weekend and subsequently.

In the end, the Achilles’ heel of Starr’s case is that, if not for the Government’s intervention, AIG would have filed for bankruptcy. In a bankruptcy proceeding, AIG’s shareholders would most likely have lost 100 percent of their stock value. DX 2615 (chart showing that equity claimants typically have recovered zero in large U.S. bankruptcies). Particularly in the case of a corporate conglomerate largely composed of insurance subsidiaries, the assets of such subsidiaries would have been seized by state or national governmental authorities to preserve value for insurance policyholders. Davis Polk’s lawyer, Mr. Huebner, testified that it would have been a “very hard landing” for AIG, like cascading champagne glasses where secured creditors are at the top with their glasses filled first, then spilling over to the glasses of other creditors, and finally to the glasses of equity shareholders where there would be nothing left. Huebner, Tr. 5926, 5930-31; see also Offit, Tr. 7370 (In a bankruptcy filing, the shareholders are “last in line” and in most cases their interests are “wiped out.”).

A popular phrase coined by financial adviser John Studzinski, in counseling AIG’s Board on September 21, 2008 is that “twenty percent of something [is] better than 100 percent of nothing.”

All of this is absolutely correct. It also applies to Goldman, JPM, BofA, Citi, Wells and so on: all of the banks which accepted a government bailout either in equity, loan, discount window access, and so on, primed their stock to the point where the equity was worthless. As such, the entire equity tranches of the US financial system at the moment Lehman failed was worth precisely nothing.  It is also why the Goldman controlled Fed did everything in Hank Paulson's power to provide the Fed with a blank check to bail out Goldman Sachs the US financial system at any taxpayer means necessary.
Which is precisely what happened, to the tune of trillions and trillions of liquidity injections, government backstops and loans into what was at that moment a financial system which was operating but whose equity was for all intents and purposes utterly worthless.
* * *
Which takes us to the Court's closing arguments:
the Court finds that the first plaintiff class prevails on liability because of the Government’s illegal exaction, but recovers zero damages.

As the Court noted during closing arguments, a troubling feature of this outcome is that the Government is able to avoid any damages notwithstanding its plain violations of the Federal Reserve Act. Closing Arg., Tr. 69-70. Any time the Government saves a private enterprise from bankruptcy through an emergency loan, as here, it can essentially impose whatever terms it wishes without fear of reprisal. Simply put, the Government often may ignore the conditions and restrictions of Section 13(3) knowing that it will never be ordered to pay damages. 

And there you have it in a nutshell: 103 years after the Aldrich Plan to create a National Reserve Association in which private, commercial banks could create money out of thin air, failed to pass and instead an "impartial" Federal Reserve was created, the US Central Bank is nothing more than what its founder on Jekyll Island first envisioned: a private enterprise above the law, which caters entirely to commercial bank, bails them out, or nationalizes them illegally as it sees fit, and generally does whatever it wishes without any public oversight.
As to the Fed's take on just how illegal its actions were, or if - gasp - it learned its lesson and will no longer illegally "bail out" this bank or that, here is the answer.

The Federal Reserve strongly believes that its actions in the AIG rescue during the height of the financial crisis in 2008 were legal, proper and effective.  

And judging by the public's response to the events of 2008, where it is clear that not only the Fed but nobody learned anything, the next bailout of US commercial banks will proceed very much like the previo sone. And the next. And the one after that.
Source: Starr International Company v The United States