The $12 Trillion Fat Finger: How A "Glitch" Nearly Crashed The Global Financial System - A True Story

Submitted by Tyler Durden on 08/08/2015 17:43 -0400

http://www.zerohedge.com/news/2015-08-08/12-trillion-fat-finger-how-glitch-nearly-crashed-global-financials-system-true-story

 For all the talk of how the financial world nearly ended in the aftermath of first the Lehman bankruptcy, then the money market freeze, and culminating with the AIG bailout, and how bubble after Fed bubble has made the entire financial system as brittle as the weakest counterparty in the collateral chain of some $100 trillion in shadow liabilities, the truth is that despite all the "macroprudential" planning and preparations, all the world's credit, housing, stock, and illiquidity bubbles may be nothing when compared to the oldest "glitch" in the book: a simple cascading error which ends up taking down the entire system.

Like what happened in the great quant blow up August 2007.

For those who may not recall the specific details of how the "quant crash" nearly wiped out all algo and quant trading hedge funds and strats in a matter of hours if not minutes, leading to tens of billions in capital losses, here is a reminder, and a warning that the official goalseeked crisis narrative "after" the fact is merely there to hide the embarrassment of just how close to total collapse the global financial system is at any given moment.

The following is a true story (courtesy of b3ta) from the archives, going all the way back to 2007:

I.T. is a minefield for expensive mistakes

There's so many different ways to screw up. The best you can hope for in a support role is to be invisible. If anyone notices your support team at all, you can rest assured it's because someone has made a mistake. I've worked for three major investment banks, but at the first place I witnessed one of the most impressive mistakes I'm ever likely to see in my career. I was part of the sales and trading production support team, but thankfully it wasn't me who made this grave error of judgement...

(I'll delve into obnoxious levels of detail here to add color and context if you're interested. If not, just skip to the next chunk, you impatient git)

This bank had pioneered a process called straight-through processing (STP) which removes the normal manual processes of placement, checking, settling and clearing of trades. Trades done in the global marketplace typically have a 5-day clearing period to allow for all the paperwork and book-keeping to be done. This elaborate system allowed same-day settlement, something never previously possible. The bank had achieved this over a period of six years by developing a computer system with a degree of complexity that rivalled SkyNet. By 2006 it also probably had enough processing power to become self-aware, and the storage requirements were absolutely colossal. It consisted of hundreds of bleeding edge compute-farm blade servers, several £multi-million top-end database servers and the project had over 300 staff just to keep it running. To put that into perspective, the storage for this one system (one of about 500 major trading systems at the bank) represented over 80% of the total storage used within the company. The equivalent of 100 DVD's worth of raw data entered the databases each day as it handled over a million inter-bank trades, each ranging in value from a few hundred thousand dollars to multi-billion dollar equity deals. This thing was BIG.

You'd think such a critically important and expensive system would run on the finest, fault-tolerant hardware and software. Unfortunately, it had grown somewhat organically over the years, with bits being added here, there and everywhere. There were parts of this system that no-one understood any more, as the original, lazy developers had moved company, emigrated or *died* without documenting their work. I doubt they ever predicted the monster it would eventually become.

A colleague of mine one day decided to perform a change during the day without authorisation, which was foolish, but not uncommon. It was a trivial change to add yet more storage and he'd done it many times before so he was confident about it. The guy was only trying to be helpful to the besieged developers, who were constantly under pressure to keep the wretched thing moving as it got more bloated each day, like an electronic ‘Mr Creosote’.

As my friend applied his change that morning, he triggered a bug in a notoriously crap script responsible for bringing new data disks online. The script had been coded in-house as this saved the bank about £300 per year on licensing fees for the official ‘storage agents’ provided by the vendor. Money that, in hindsight, would perhaps have been better spent instead of pocketed. The homebrew code took one look at the new configuration and immediately spazzed out. This monged scrap of pisspoor geek-scribble had decided the best course of action was to bring down the production end of the system and bring online the disaster recovery (DR) end, which is normal behaviour when it detects a catastrophic 'failure'. It’s designed to bring up the working side of the setup as quickly as possible. Sadly, what with this system being fully-replicated at both sites (to [cough] ensure seamless recovery), the exact same bug was almost instantly triggered on the DR end, so in under a minute, the hateful script had taken offline the entire system in much the same manner as chucking a spanner into a running engine might stop a car. The databases, as always, were flushing their precious data onto many different disks as this happened, so massive, irreversible data corruption occurred. That was it, the biggest computer system in the bank, maybe even the world, was down.

And it wasn't coming back up again quickly.

(OK, detail over. Calm down)

At the time this failure occurred there was more than $12 TRILLION of trades at various stages of the settlement process in the system. This represented around 20% of ALL trades on the global stock market, as other banks had started to plug into this behemoth and use its capabilities themselves. If those trades were not settled within the agreed timeframe, the bank would be liable for penalties on each and every one, the resulting fines would eclipse the market capital of the company, and so it would go out of business. Just like that.

My team dropped everything it was doing and spent 4 solid, brutal hours recovering each component of the system in a desperate effort to coax the stubborn silicon back online. After a short time, the head of the European Central Bank (ECB) was on a crisis call with our company CEO, demanding status updates as to why so many trades were failing that day. Allegedly (as we were later told), the volume of financial goodies contained within this beast was so great that failure to clear the tradeswould have had a significant negative effect on the value of the Euro currency. This one fuckup almost started a global economic crisis on a scale similar to the recent (and ongoing) sub-prime credit crash. With two hours to spare before the ECB would be forced to go public by adjusting the Euro exchange rate to compensate, the system was up and running, but barely. We each manned a critical sub-component and diverted all resources into the clearing engines. The developers set the system to prioritise trades on value. Everything else on those servers was switched off to ensure every available CPU cycle and disk operation could be utilised. It saturated those machines with processing while we watched in silence, unable to influence the outcome at all.

Incredibly, the largest proportion of the high-value transactions had cleared by the close of business deadline, and disaster was averted by the most "wafer-thin" margin. Despite this, the outstanding lower-value trades still cost the bank more than $100m in fines. Amazingly, to this day only a handful of people actually understand the true source of those penalties on the end-of-year shareholder report.Reputation is king in the world of banking and all concerned --including me-- were instructed quite explicitly to keep schtum. Naturally, I *can’t* identify the bank in question, but if you’re still curious, gaz me and I’ll point you in the right direction…

Epilogue… The bank stumped up for proper scripts pretty quickly but the poor sap who started this ball of shit rolling was fired in a pompous ceremony of blame the next day, which was rather unfair as it was dodgy coding which had really caused the problem. The company rationale was that every blaze needs a spark to start it, and he was going to be the one they would scapegoat. That was one of the major reasons I chose to leave the company (but not before giving the global head of technology a dressing down at our Christmas party… that’s another QOTW altogether). Even today my errant mate is one of the only people who properly understands most of that preposterous computer system, so he had his job back within six months -- but at a higher rate than before :-)

Conclusion: most banks are insane and they never do anything to fix problems until *after* it costs them uber-money. Did I hear you mention length? 100 million dollar bills in fines laid end-to-end is about 9,500 miles long according to Google calculator.

* * *

And here is Zero Hedge's conclusion: the next time you think all those paper reps and warranties to claims on billions if not trillions of assets, are safe and sound in some massively redundant hard disk array, think again.

NYU Professor Uncovers How The FDA Systematically Covers Up Fraud & Misconduct In Drug Trials

Submitted by Tyler Durden on 02/18/2015 18:30 -0500
http://www.zerohedge.com/news/2015-02-18/nyu-professor-uncovers-how-fda-systematically-covers-fraud-misconduct-drug-trials
Submitted by Mike Krieger via Liberty Blitzkrieg blog,

That misconduct happens isn’t shocking. What is: When the FDA finds scientific fraud or misconduct, the agency doesn’t notify the public, the medical establishment, or even the scientific community that the results of a medical experiment are not to be trusted. On the contrary. For more than a decade, the FDA has shown a pattern of burying the details of misconduct. As a result, nobody ever finds out which data is bogus, which experiments are tainted, and which drugs might be on the market under false pretenses. The FDA has repeatedly hidden evidence of scientific fraud not just from the public, but also from its most trusted scientific advisers, even as they were deciding whether or not a new drug should be allowed on the market. Even a congressional panel investigating a case of fraud regarding a dangerous drug couldn’t get forthright answers. For an agency devoted to protecting the public from bogus medical science, the FDA seems to be spending an awful lot of effort protecting the perpetrators of bogus science from the public.

The sworn purpose of the FDA is to protect the public health, to assure us that all the drugs on the market are proven safe and effective by reputable scientific trials. Yet, over and over again, the agency has proven itself willing to keep scientists, doctors, and the public in the dark about incidents when those scientific trials turn out to be less than reputable. It does so not only by passive silence, but by active deception. And despite being called out numerous times over the years for its bad behavior, including from some very pissed-off members of Congress, the agency is stubbornly resistant to change. It’s a sign that the FDA is deeply captured, drawn firmly into the orbit of the pharmaceutical industry that it’s supposed to regulate. We can no longer hope that the situation will get better without firm action from the legislature.

From the Slate article: Are Your Medications Safe?

In the past week or so, I’ve come across several important articles that will leave any rational observer increasingly skeptical of the entire medial industry in the U.S. This isn’t something I say lightly, and I think it’s an absolutely horrific development for our society.
Just last week, Liberty Blitzkrieg published an article titled, Introducing “Physician Dispensing” – The Latest Troubling Medical Industry Scam, which expounded on why an erosion of trust in doctors is so troubling. If you missed that piece, I suggest going back and reading it. Here’s an excerpt:

Once the corruption reaches a certain level of societal saturation, you create a culture in which people simply stop trusting everyone and everything. For obvious reasons, this is a very dangerous development. There are people whom you need to trust for any civilization to function reasonably well. Police are one, but doctors are another. I can speak for myself when I say that I am not convinced that any medical professional I see has only my best interests at heart. I seriously wonder how he or she is balancing my health with the ability to earn more money. From conversations with friends and family, I have found that this is much more widespread than we would like to admit. This is incredibly bad and incredibly sad.

While that article was bad enough, it is nothing compared to what I just read by Charles Seife, a journalism professor at New York University. He and his students set out to research the FDA and how it deals with evidence of fraud and misconduct in pharmaceutical drug trials. What he found will shock and disturb even the most hardened cynic. If you are one of the 70% of Americans that take at least one prescription drug, brace yourself…
From Slate:

Agents of the Food and Drug Administration know better than anyone else just how bad scientific misbehavior can get. Reading the FDA’s inspection files feels almost like watching a highlights reel from a Scientists Gone Wildvideo. It’s a seemingly endless stream of lurid vignettes—each of which catches a medical researcher in an unguarded moment, succumbing to the temptation to do things he knows he really shouldn’t be doing. Faked X-ray reports. Forged retinal scans. Phony lab tests. Secretly amputated limbs. All done in the name of science when researchers thought that nobody was watching.

That misconduct happens isn’t shocking. What is: When the FDA finds scientific fraud or misconduct, the agency doesn’t notify the public, the medical establishment, or even the scientific community that the results of a medical experiment are not to be trusted. On the contrary. For more than a decade, the FDA has shown a pattern of burying the details of misconduct. As a result, nobody ever finds out which data is bogus, which experiments are tainted, and which drugs might be on the market under false pretenses. The FDA has repeatedly hidden evidence of scientific fraud not just from the public, but also from its most trusted scientific advisers, even as they were deciding whether or not a new drug should be allowed on the market. Even a congressional panel investigating a case of fraud regarding a dangerous drug couldn’t get forthright answers. For an agency devoted to protecting the public from bogus medical science, the FDA seems to be spending an awful lot of effort protecting the perpetrators of bogus science from the public.

We didn’t have to search very hard to find FDA burying evidence of research misconduct. Just look at any document related to an FDA inspection. As part of the new drug application process, or, more rarely, when the agency gets a tipoff of wrongdoing, the FDA sends a bunch of inspectors out to clinical sites to make sure that everything is done by the book. When there are problems, the FDA generates a lot of paperwork—what are called form 483s, Establishment Inspection Reports, and in the worst cases, what are known as Warning Letters. If you manage to get your hands on these documents, you’ll see that, most of the time, key portions are redacted: information that describes what drug the researcher was studying, the name of the study, and precisely how the misconduct affected the quality of the data are all blacked out. These redactions make it all but impossible to figure out which study is tainted. My students and I looked at FDA documents relating to roughly 600 clinical trials in which one of the researchers running the trial failed an FDA inspection. In only roughly 100 cases were we able to figure out which study, which drug, and which pharmaceutical company were involved. (We cracked a bunch of the redactions by cross-referencing the documents with clinical trials data, checking various other databases, and using carefully crafted Google searches.) For the other 500, the FDA was successfully able to shield the drugmaker (and the study sponsor) from public exposure.

Think about that. Despite all that digging, they were able to link questionable data to specific drugs in only 20% of the cases examined.

It’s not just the public that’s in the dark. It’s researchers, too. And your doctor. As I describe in the current issue of JAMA Internal Medicine, my students and I were able to track down some 78 scientific publications resulting from a tainted study—a clinical trial in which FDA inspectors found significant problems with the conduct of the trial, up to and including fraud. In only three cases did we find any hint in the peer-reviewed literature of problems found by the FDA inspection. The other publications were not retracted, corrected, or highlighted in any way. In other words, the FDA knows about dozens of scientific papers floating about whose data are questionable—and has said nothing, leaving physicians and medical researchers completely unaware. The silence is unbroken even when the FDA itself seems shocked at the degree of fraud and misconduct in a clinical trial.

Such was the case with the so-called RECORD 4 study. RECORD 4 was one of four large clinical trials that involved thousands of patients who were recruited at scores of clinical sites in more than a dozen countries around the world. The trial was used as evidence that a new anti-blood-clotting agent, rivaroxaban, was safe and effective. The FDA inspected or had access to external audits of 16 of the RECORD 4 sites. The trial was a fiasco. At Dr. Craig Loucks’ site in Colorado, the FDA found falsified data. At Dr. Ricardo Esquivel’s site in Mexico, there was “systematic discarding of medical records” that made it impossible to tell whether the study drug was given to the patients. At half of the sites that drew FDA scrutiny—eight out of 16—there was misconduct, fraud, fishy behavior, or other practices so objectionable that the data had to be thrown out. The problems were so bad and so widespread that, contrary to its usual practice, the FDA declared the entire study to be “unreliable.” Yet if you look in the medical journals, the results from RECORD 4 sit quietly in The Lancet without any hint in the literature about falsification, misconduct, or chaos behind the scenes. This means that physicians around the world are basing life-and-death medical decisions on a study that the FDA knows is simply not credible.

It’s not just one study, either. The FDA found major problems with sites involved in the other three clinical trials that were used to demonstrate rivaroxaban’s safety and effectiveness. RECORD 2, for example, was nearly as awful as RECORD 4: Four out of 10 sites that the FDA inspected showed evidence of misconduct, or other issues grave enough to render the site’s data worthless—including clear evidence of data falsification at one site. In aggregate, these problems raise serious doubts about the quality of all four key rivaroxaban studies—and, by extension, doubts about how seriously we should take the claim that rivaroxaban is safe and effective. The FDA is keeping mum, even as wrongful-death lawsuits begin to multiply.

In the decade since the Ketek affair, it’s hard to see any change in behavior by the agency. On occasion, the FDA has even actively approved and promoted statements about drugs that, according to its own inspectors, are based upon falsehoods. At the end of 2011, the FDA learned that an audit of a Chinese site involved in a key clinical trial of a different anti-clotting agent, apixaban, had turned up evidence of fraud: Personnel had apparently been fiddling with patient records. Worse yet, the fraud appeared to invalidate one key finding of the study. Just three months earlier, the researchers running the trial proudly announced in the New England Journal of Medicine that there was a “significant reduction in mortality” among patients who took apixaban compared with those who took the old standby, warfarin. Alas, the moment you exclude the data from the Chinese fraud site, as per standard FDA procedure, that statement went out the window. Yet look at the label for apixaban—the one approved by the FDA after the fraud was discovered—and you read that “treatment resulted in a significantly lower rate of all-cause death … than did treatment with warfarin,” backed up by the data set with the Chinese site included. In other words, the label is carrying a claim that the FDA knows is based upon fraud. In a written response to my questions on this subject, the FDA stated that, “The FDA extended the drug’s review period to address the concerns. However, the review team did conclude concluded [sic] that the data at that site and other sites in China did reflect meaningful clinical information; that was not what was considered unreliable.”

Again, this isn’t an isolated incident. I had previously encountered bogus data on FDA-approved labels when a colleague and I were looking into a massive case of scientific misconduct —a research firm named Cetero had been caught faking data from more than 1,400 drug trials. That suddenly worthless data had been used to establish the safety or effectiveness of roughly 100 drugs, mostly generics, that were being sold in the United States. But even after the agency exposed the problem, we found fraud-tainted data on FDA-approved drug labels. (The FDA still maintains its silence about the Cetero affair. To this day, the agency refuses to release the names of the 100-odd drugs whose approval data were undermined by fraud.)

The most common excuse the agency gives is that exposing the details about scientific wrongdoing—naming the trials that were undermined by research misconduct, or revealing which drugs’ approvals relied upon tainted data—would compromise “confidential commercial information” that would hurt drug companies if revealed. This claim falls apart under scrutiny. The courts have ruled that when information is provided by companies involuntarily, such as the information that an FDA inspector finds, “commercial confidential information” refers to proprietary material that causes substantial, specific harm when it falls into the hands of a competitor. It doesn’t cover embarrassing peccadilloes—or misconduct that might cause bad publicity when word gets out.

As usual, it’s all about protecting corporate profits.

The sworn purpose of the FDA is to protect the public health, to assure us that all the drugs on the market are proven safe and effective by reputable scientific trials. Yet, over and over again, the agency has proven itself willing to keep scientists, doctors, and the public in the dark about incidents when those scientific trials turn out to be less than reputable. It does so not only by passive silence, but by active deception. And despite being called out numerous times over the years for its bad behavior, including from some very pissed-off members of Congress, the agency is stubbornly resistant to change. It’s a sign that the FDA is deeply captured, drawn firmly into the orbit of the pharmaceutical industry that it’s supposed to regulate. We can no longer hope that the situation will get better without firm action from the legislature.

America’s new religion.
*  *  *
For related articles, see:
First is a MUST WATCH hilarious video by John Oliver: Video of the Day – John Oliver on Pharma Company “Marketing to Doctors”
Fraud Alert: FDA Allowed Drugs with Fraudulent Testing to Remain on the Market
The FDA is Caught Spying on its Employees and Creating an “Enemies List”
Introducing “Physician Dispensing” – The Latest Troubling Medical Industry Scam

This Google Motherboard Means Trouble for Intel

Image: Google

http://www.wired.com/2014/04/openpower/?mbid=social_fb

The Intel chip factory in Chandler, Arizona was christened by President Barack Obama.
In 2012, while it was under construction, the President made a pit stop at the plant, known as Fab 42, painting it as a symbol of American optimism. “The factory that’s being built behind me is an example of an America that is within our reach–an America that attracts the next generation of good manufacturing jobs,” Obama said.
It was a noble vision, but for Intel, things didn’t exactly work out as planned. The chip giant eventually mothballed the $5 billion factory, and the construction site is now a symbol of a different kind. Fab 42 represents an Intel in transition, a company that’s struggling to evolve with a changing world.

The big online companies, including Google and Facebook and Amazon, are now looking to run their operations on computer servers that use chips made by someone other than Intel.

It’s not just that people are buying iPads and Android phones built with low-power ARM processors instead of PCs and phones and tablets powered by Intel chips–the main reason the Chandler plant was put on hold. It’s that the big online companies, including Google and Facebook and Amazon, are now looking to run their operations on computer servers that use chips made by someone other than Intel. And the first trend may ultimately feed the second.
The latest blow to Intel’s future arrived on Monday in the form of a red server motherboard touted by Gordon MacKean, the man responsible for building the hundreds of thousands of servers that power Google’s online empire. In a Google+ post, MacKean said he was “excited” to show off the red motherboard, which was built using not an Intel chip, but IBM’s Power8 processor.
To the outsider, the motherboard may not look like much, but the fact that Google has taken the time and effort to port its software to IBM’s architecture and even design a motherboard based on an IBM processor is a big deal. Since its beginning, back in 1998, Google has used servers equipped with Intel processors, and today the company is one of the world’s largest buyers of Intel server chips. The search giant doesn’t make servers for anyone but itself, but it’s likely the fifth-largest Intel server chip customer on Earth.
Why is Google tinkering with a brand new microprocessor? “We’re really driven by an aggressive demand. The growth at Google has been very significant,” McKean says. In other words, Google keeps growing, and so the massive collection of servers that runs Google must keep growing too. Yes, the company can keep expanding its operation using Intel chips. But it behooves Google to use other chip suppliers. That’s a way to cut costs, but it’s also a way to ensure that the chips it uses just keep getting better. Companies like Google don’t want to rely solely on Intel. They want competition in the market. They want to play one chip maker off another.

The Two Intels

For more than a decade, Intel’s chip operation has been a beautiful thing: two parallel lines of business, delivering both staggering volume and high margins. There’s the desktop business, and the server business. But as Intel’s client business struggles, it could affect the server side of the company. As Christos Kozyrakis, a computer science professor at Stanford University, points out, Intel will typically build a desktop chip and then remake it for the server world. “They take exactly the same core with different caches, different memory controllers,” he says, “and they put it on server.”

The difference with ARM and Power is that any outside manufacturers can license the designs and modify them as need be. That’s not the case with Intel’s x86 architecture. The onus is on Intel to innovate.

This lets Intel spread a single chip’s development costs over several parts of the company. And that’s important. Designing a new processor core is a major undertaking, one that can take hundreds of engineers several years to complete. But it’s unclear whether this arrangement will work as well in the future. “Up until now, it seemed to be the case that whatever was good for one segment of the market was good for another,” says Kozyrakis. “Now the question becomes: has this changed?”
Intel says that desktop shipments are rebounding of late, and that server improvements are being cranked out like never before. Indeed, the company has a massive advantage in the server business. Google, Facebook, Microsoft, Amazon — all of the web giants overwhelmingly use Intel-based x86 servers. But as Intel struggles with the desktop market, it’s facing increased competition on the server side. In addition to Google exploring IBM’s Power chips, Facebook has long made noises about using ARM and other low-power chips in its servers. And now it seems Amazon is looking at the same thing.
The difference with ARM and Power is that any outside manufacturers can license the designs and modify them as need be. That’s not the case with Intel’s x86 architecture. The onus is on Intel to innovate. ARM has always licensed out its architecture, and now IBM has formed a group called OpenPower, where memory makers, graphics chip companies, and other component vendors can come together and help build the kind of systems that the Googles of the world are already clamoring for. “If you look at x86, x86 is not creating this open ecosystem environment to let everybody come in and innovate on their platform,” says Brad McCready, an IBM Fellow.
The Christopher Lameter, an R&D team lead with JumpTrading, a Chicago-based high-frequency trading firm, says that he hopes that the OpenPower effort will lead to new types of chip design that will be useful to customers like JumpTrading. He worries that the desktop slowdown will ultimately hurt new Intel developments on the server side, some four-to-six years down the line. And at the same time, he’s excited by some of the new things that have been developed in the mobile phone world. “On the kernel level, it seems that ARM/Android [is] driving innovation,” he says.
But the truth is that today, nobody is certain where the next great server breakthrough will come from. For years, chipmakers got huge performance gains by shrinking the size of their chip components. But today’s chips components are becoming so tiny that they can’t be shrunk for much longer. And the best idea right now seems to be building chips that are custom designed to be really fast at ferrying and processing data for web applications. IBM dreams that OpenPower will do that.
What’s more, if innovation is happening with ARM and Power, there’s pressure on Intel to follow suit.

The Spur of Competition

Intel’s client-side slowdown is real. A decade ago, the company’s client business was growing by 11 percent per year. In 2013, it shrunk by four percent, according to data compiled by Mercury Research, a microchip analyst firm. But Mercury’s principal analyst Dean McCarron doesn’t think that the desktop slowdown is having any effect on server innovation.
But he agrees that there’s one thing that can boosts innovation in the server space: competition. Two years ago, Intel didn’t have much of that. But with OpenPower and ARM pushing into the game, everything is changing. “When there’s a lot of competition. There’s a lot more product innovation,” McCarron says.
That said, it will take a lot more than a red motherboard to displace the king of the hill. Says McCarron: “Intel has every incentive to retain this market because of how lucrative it is.”

Google admits data mining student emails in its free education apps

by Jeff Gould, Peerstone Research Friday, January 31, 2014

When it introduced a new privacy policy designed to improve its ability to target users with ads based on data mining of their online activities, Google said the policy didn’t apply to students using Google Apps for Education. But recent court filings by Google’s lawyers in a California class action lawsuit against Gmail data mining tell a different story: Google now admits that it does data mine student emails for ad-targeting purposes outside of school, even when ad serving in school is turned off, and its controversial consumer privacy policy does apply to Google Apps for Education.
At SafeGov.org our work has long focused on the risks of allowing targeted online advertising into schools. This issue has come to the fore as companies like Google and Microsoft have launched a worldwide race to introduce their web application suites into as many schools as possible. In this article we review the background of this debate and then present important new evidence regarding the practices of one of the leading players, Google.
The suites in question are known as Google Apps for Education and Office 365 Education, respectively, and they include basic apps such as email, word processing, spreadsheets, live document sharing, simple web forms and messaging. Their key selling point is that they offer students something almost as good as a traditional office suite in the convenient format of a browser window, and – best of all for cash-strapped schools – they do so at no cost.
Of course as the economist said there is no such thing as a free lunch, and we must look carefully at the business motives behind these firms’ generosity. Here an important difference between the two leaders emerges. Both Google and Microsoft generate substantial revenues by selling online office suites to government and enterprises for annual subscription fees. If the firms offer essentially the same suites to schools for free, it is surely in part because they hope that when students move into the workplace they will demand the same online tools they learned to use in school. This is a business model that is honest about its intentions and serves the interests of both students and the firms. However, there is an additional component in the Google business model that involves advertising, and this is where the trouble begins.
Both Google and Microsoft offer free ad-based email services to consumers – Gmail and Outlook.com (formerly Hotmail). Google’s Gmail pioneered the technique of targeting ads to users based on profiles of their interests. Google creates the profiles with the help of sophisticated software algorithms that sift through users’ past and present emails, record the things they search for on Google’s search engine, and track the web sites they visit (via the cookies placed on many sites by its DoubleClick ad-serving subsidiary).
The activity performed by these profiling algorithms is known as “data mining”, and their power to make accurate guesses about the tastes and likely behavior of the profiled users is quite remarkable. However, not every free consumer email service uses data mining to target ads. Microsoft’s Hotmail, for example, relied solely on demographic information (such as age, gender and location) provided by users when they register. Hotmail’s successor Outlook.com continues this policy, promising that it “doesn't serve targeted ads based on email contents”. While the ad delivery methods used by the major email providers may differ, the basic idea of offering consumers free email in exchange for ads has proven extraordinarily successful. The top three providers – Google, Microsoft, and Yahoo – together count over one billion users. SafeGov does not take a position on the methods used to target ads in these services. In our view all are legitimate business models, provided that consumers are fully informed of how their data is used and have given their consent.
Whether or to what degree these last two conditions are actually met by specific services such as Gmail or Outlook.com is of course a pertinent question. Currently Google faces legal challenges to its use of consumer data mining in both the U.S. and the European Union. EU data protection authorities in particular have determined that Google fails to inform consumers properly of its conduct or obtain their consent, while a major class action law suit in California advances similar accusations. Although Outlook.com appears to have avoided such challenges to date, we should certainly expect that regulators and courts will hold it to the same high standards as Gmail.
The Google and Microsoft education suites discussed above operate under quite different rules than the firms’ ad-based consumer email services. Office 365, developed from Microsoft’s enterprise server-based software packages such as Exchange and SharePoint, was never designed to serve ads and does not have the functionality to create ad-targeting user profiles based on data mining. Microsoft’s Office 365 web site makes an entirely unambiguous pledge in this regard: “We do not mine your data for advertising purposes.”
Google Apps for Education, by contrast, has a more ambivalent policy regarding advertising. While Google pledges not to serve ads to students without schools’ permission, its Google Apps suite, which is a repurposed version of Google’s Gmail and other consumer services, was designed from the ground up to include ad-serving as well as highly sophisticated user profiling and data mining capabilities. Google explicitly offers schools the option of enabling ad serving to student users of Google Apps for Education. Although it does not yet offer to share the resulting ad revenues with schools that choose the ad-serving option, it has clearly left the door open to such revenue sharing in the future. Indeed, it is hard to see why Google would explicitly write the ad-serving option into its standard contract with schools if it did not hope one day to make ads for students a default and perhaps even mandatory feature of Apps for Education.
Targeted ads are worth more than untargeted ads, because advertisers will pay more to put their ads in front of customers who are more likely to buy. The uncanny power of Google’s data mining and user profiling algorithms to target ads effectively has made it the world’s largest advertising company. To cite just one data point, the Mountain View giant last year generated more ad revenue in the American market than the entire U.S. newspaper industry. While we take Google’s word that it does not serve ads to its student users unless it has permission from schools, an important question that until now has gone unanswered is whether the targeting algorithms that power Gmail are still running in Google Apps for Education even when ad serving is turned off. Google’s own web site once supplied an explicit and quite satisfactory answer to this question. Specifically, in a FAQ on its web site devoted to Google Apps for Education, the firm promised that:

“If you are using Google Apps (free edition), email is scanned so we can display contextually relevant advertising in some circumstances. Note that there is no ad-related scanning or processing in Google Apps for Education or Business with ads disabled.”

However, at some point during the past year the crucial second sentence in this statement was deleted from Google’s web site.
Of course it’s difficult to draw firm conclusions from fleeting changes in the wording on a vendor’s web page. Accordingly SafeGov has been searching for further evidence that would help to resolve one way or the other the question of Google’s data mining practices in Apps for Education. When a trove of court documents from a class action lawsuit against Google in U.S. Federal Court was recently made public, we decided to do a little data mining of our own, albeit with tools less sophisticated than Google’s. What we found is worthy of attention.
In a remarkable pretrial document filed by Google’s lawyers, Google explicitly admits for the first time that it scans the email of Google Apps for Education users for ad-serving purposes even when ad serving is turned off. The issue at stake in the case is whether Google has properly informed its users and obtained their consent for data mining and ad serving in Gmail and, by extension, in Google Apps for Education. In the filing in question Google’s lawyers seek to prove that email users must have consented to Google’s email scanning practices – if only “impliedly” – because these practices have been widely discussed in the press and can thus be considered to be universally known. The lawyers seek to establish this point by supplying a long list of published articles that discuss these practices.
Regarding Google Apps for Education in particular, the lawyers state that schools which contract with Google to provide Google Apps “have a contractual obligation to obtain their students’ and end users’ consent to Google’s automated scanning”. The document then goes on to list a number of examples of how educational institutions have carried out this duty to inform users and obtain their consent for scanning. Notably the Google filing cites the web site of the University of Alaska as an exemplary instance of such compliance:

The University of Alaska (“UA”) has a “Google Mail FAQs,” which asks, “I hear that Google reads my email. Is this true?” The answer states, “They do not ‘read’ your email per se. For use in targeted advertising on their other sites, and if your email is not encrypted, software (not a person) does scan your mail and compile keywords for advertising. For example, if the software looks at 100 emails and identifies the word ‘Doritos’ or ‘camping’ 50 times, they will use that data for advertising on their other sites.” Attached as Exhibit 79 is a true and correct print out of UA’s Google Mail FAQ page, which is also available at www.alaska.edu/google/faqs/general/#mail (last visited Nov. 13, 2013). [Declaration of Kyle C. Wong in Support of Google Inc.’s Opposition to Plaintiffs’ Motion for Class Certification, p. 41]

In other words, Google’s own lawyers here confirm in a sworn public court declaration that even when ad serving is turned off in Google Apps for Education, the contents of users’ emails are still being scanned by Google in order to target ads at those same users when they use the web outside of Google Apps (for example, when watching a YouTube video, conducting a Google search, or viewing a web page that contains a Google+ or DoubleClick cookie). This statement thus appears to be what American lawyers call “an admission against interest”.
Google’s data mining and ad serving practices in the versions of Google Apps it provides to public sector institutions such as government administrations and schools have long been a subject of controversy. Media and regulator interest in the issue surged in early 2012 when Google launched a sweeping consolidation of the many privacy policies governing its individual products into a single overarching document. The new unified privacy policy was intended, among other things, to facilitate Google’s ability to combine information about users extracted from its different services – such as Gmail, YouTube, Google search, etc. – into a single integrated profile of each user, thereby enabling ever more accurate – and so more profitable - ad targeting. However, Google vehemently denied that this new consumer privacy policy would apply to governments and schools. Indeed, a senior Google executive told the Washington Post that:

“Enterprise customers using Google Apps for Government, Business or Education have individual contracts that define how we handle and store their data. As always, Google will maintain our enterprise customers’ data in compliance with the confidentiality and security obligations provided to their domain. The new Privacy Policy does not change our contractual agreements, which have always superseded Google’s Privacy Policy for enterprise customers.”

But Google’s court filings in the California class action suit discussed above unambiguously contradict this statement. In one of these filings, a Google employee states that:

Google and [the University of] Hawaii executed an agreement titled “Google Apps Education Edition Agreement” on or about June 21, 2010… The agreement places the responsibility to obtain “any necessary authorizations from End Users to enable Google to provide the Services” on Hawaii, the “Customer.” The “Services” includes Gmail… The agreement also requires Google to comply with the Customer Privacy Notice… and the End User Privacy Notice…

In other words, Google here acknowledges that its standard consumer privacy policy is an integral part of its standard Google Apps for Education contract. It is still possible that, in contrast to the situation described in the Google court filing quoted above, some educational institutions have managed to strike individual agreements with Google that do indeed “supersede” the standard privacy policy. If they exist, however, Google has curiously not chosen to make any such agreements public. Indeed, there is evidence that Google imposes “gag clauses” on schools that sign contracts for its Google Apps for Education, forbidding them from disclosing the terms and conditions they have received.
In sum, then, we have learned from Google’s own statements that:

1. Ad serving remains a standard option in Google Apps for Education,
2. Even when ads are turned off (as they currently are by default) Google still data mines student emails for ad targeting purposes, and
3. Google’s consumer privacy policy is incorporated in standard Google Apps for Education contracts.

It is a natural and very plausible – though of course not certain – inference from these facts that Google intends one day to make advertising a standard feature of the version of Google Apps it offers to schools.
Where is the harm in allowing targeted advertising in the online web applications that schools provide to their students? This is a vast and important question that we lack the space to address here but will investigate in future work. Suffice it to say as a starting point that SafeGov surveys of parents around the world have unfailingly shown a very high level of parental opposition to such advertising and the intrusive profiling of student online activity that makes it possible – typically in the 80% to 90% range[1]. We believe that policy makers, education authorities and data protection regulators will not choose to ignore the will of parents on this issue. Stay tuned for further research from SafeGov on this vitally important topic.

---

[1] For example, see results from our U.S., Australia, and Malaysia parent surveys – results from other countries are forthcoming
http://safegov.org/2014/1/31/google-admits-data-mining-student-emails-in-its-free-education-apps
Read more Read more

Edward Snowden doesn’t show up once in Google’s list of top 2013 searches

• By Brian Fung December 17 at 12:59 pm

(AP Photo)

This year's National Security Agency revelations have created a firestorm of reports and debates about the state of U.S. surveillance technology and intelligence policy. It set off a brief international manhunt. Entire countries are now building countermeasures to deflect the NSA's gaze. But at least in the eyes of Google, Edward Snowden was hardly a blip on the radar. The search giant's global year-in-review is topped by Nelson Mandela, followed by the late actor Paul Walker and the iPhone 5S. Snowden doesn't make an appearance.

 Okay, the iPhone and the Harlem Shake might be skewing the results. What if we just limited it to people? Still no luck. Oscar Pistorius, the South African athlete under investigation for his girlfriend's murder, ranked higher than Snowden. Maybe "Snowden" is a little too specific. What if we broadened the query to "NSA" or "surveillance" or "spying"? The same pattern will repeat itself if you drill down to U.S.-specific searches. Here's what trends in Washington, D.C. — arguably the city most preoccupied with Snowden this year — looked like, as provided by a Google spokesperson:

While the details of the Snowden saga may have gripped civil liberties advocates and Internet policy types — and although Snowden himself clearly thinks he's still a major subject of debate — the rest of the world seems to think otherwise.
http://www.washingtonpost.com/blogs/the-switch/wp/2013/12/17/edward-snowden-doesnt-show-up-once-in-googles-list-of-top-2013-searches/

Brian Fung covers technology for The Washington Post, focusing on electronic privacy, national security, digital politics and the Internet that binds it all together. He was previously the technology correspondent for National Journal and an associate editor at the Atlantic. His writing has also appeared in Foreign Policy, Talking Points Memo, the American Prospect and Nonprofit Quarterly.

I Googled “Evil” and it Took Me to Google

Posted on December 5, 2013 by DavidSwanson

Google may have been, until now, the Obama of hip internet monopolies.  No matter how many nations the President bombs, people still put Obama peace-sign stickers on their cars.  No matter how many radical rightwing initiatives Google funds, people still think it’s a “progressive corporation” — How could it not be? It’s making progress!

Google is funding Grover Norquist’s Americans for Tax Reform, the Federalist Society, the American Conservative Union, and the political arm of the Heritage Foundation.

And there’s more really bad news: Google is funding ALEC, the powerful, secretive, and destructive lobbying force from which many companies concerned with their public images are fleeing. ALEC is in the news this week, holding its 40th annual meeting. Together with allies, RootsAction.org is applying as much pressure as we can.  And it might just be that the tide is turning.  Google just might have to start worrying about whether its users favor plutocratic plundering or not.

ALEC targets state legislatures around the country to roll back labor rights, environmental protection, civil rights, public health measures and more. Using big money, corporate clout and smooth lobbyists, ALEC teams up with like-minded state lawmakers to draft and enact regressive legislation.

Tens of thousands of people have heeded our request to Tell Google and other huge firms to which we’ll deliver this petition not to participate in ALEC’s corruption of our government:We urge you to stop funding ALEC. With ALEC’s help …
* Tobacco companies get tax cuts.
* For-profit education companies get school privatization.
* Management gets union busting.
* Oil companies get opposition to renewable energy.
* The rich and powerful get the repeal of estate taxes.
But, in the process, democracy gets hijacked — one state at a time.

***

You should read the thousands and thousands of comments people have submitted when they have signed this petition.  And you can, they’re all publicly posted at the link above.

Some people express their great affection for Google, along with disillusionment:

“I really thought that your business was a role model as  progressive, social conscious Corporation…  I was seriously mistaking… -  – It is outrageous that you are supporting these politicians that have injured out country and our people so deeply…  and ALEC???? -  – PLEASE STOP!!!” –J. Carlo Diaz, FL

“Really, Google. Really!? I expect more sense from you!” –Marian Pickett, LA

“As much as I like Google, I’ll be damned if I’ll support in any way the increase in political clout of big and secret money funneled to “money is above all” unbridled capitalism Republican organizations!  There are other choices … and I’ll be switching!!” –William Whitlock, CA

“Any company that funds an organization like Alec does not have an interest in democracy. Alec is in the business of buying votes, and votes for the most un-American causes. My respect for Google has taken a huge hit because of its support of Alec.” –Kathlyn McCaughna

“I used GOOGLE to research ALEC. I am surprised and greatly disappointed that your corporation would support this horror to our democracy. Greed. Cynicism. Arrogance. Downright stupidity. Or as stated in one of your listed sources–policy areas including legislation ‘opposing U.S. consumers’ rights to know the origin of our food,’ ‘undermining workers’ rights,’ ‘stripping environmental protections,’ and ‘limiting patient rights and undermining safety net programs.’ (MediaMatters 12/4/13) I said ‘surprised.’ Perhaps not, just thoroughly disgusted.  – I love GOOGLE; other venues’ ads have not swayed me. However, your support of such a vicious, predatory, manipulative organization has changed the game. –JoAnn Durfee, OR

Some explain to Google what the problem is:

“Alec is for profits at any cost to society. Make your engine work for people, not for evil.” –John Kozub, TN

“Google, Facebook and Yelp are people driven and should not support corporate takeover of the internet or other arenas.” –Amy Whitworth, OR

“Does Google want to be seen to be supporting ALEC? If so they are also supporting this neanderthal approach to energy production: http://www.theguardian.com/world/2013/dec/04/alec-freerider-homeowners-assault-clean-energy  — An alliance of corporations and conservative activists is mobilising to penalise homeowners who install their own solar panels — casting them as ‘freeriders’ — in a sweeping new offensive against renewable energy, the Guardian has learned.” –E Healy, CA

“Funding the most regressive organizations puts your company right there with them: destroyers of democracy, dirty political actors. Think carefully whether this is where you want to be, and where you are going to be seen to be.” –John Prehn, UT

“ALEC has caused so much evil and trouble in Wisconsin that I am sure most people don’t understand, at this point in time. Please do NOT support the agenda of ALEC as it is not for the people but for the FEW!” –Joan Schneider, WI

Some get a bit angry:

“Alec sucks. When I found out the State Farm backed them, we DUMPED State Farm. And we had been with them for 20 years. Don’t miss them a BIT!” –Audrey Lima, FL

“ALEC’s prime mission is to destroy everything decent in this country!  Fascism simply ISN’T the same thing as Democracy!  STOP supporting this evil group!!!!” –Linda Christy, OK

“Stop SCROOGLING the very hard-working Americans who pay taxes that made you a household name. We don’t need ALEC trying to shove their power-hungry, money-centered, and sexist/racist/homophobic agendas onto hard-working taxpayers. And we won’t need GOOGLE either if they get in bed with corrupt ALEC and its members.” –Deirdre McCullough, NC

“Google is EVIL SCUM pretending to be a progressively-minded company while funding organizations that systematically destroy the rights and well-being of everyday citizens everywhere, all to increase their already bloated bottom line.  Screw you–I don’t need your search, your email, your phones, or anything else you can offer.  I NEED DEMOCRACY AND A COMPASSIONATE SOCIETY.” –Ellen Read, NH

Or angry and pithy:

“Dicks.” –Brad Thompson, IA

Some plead with Google:

“Please, Google, do not be a part of this right wing attempt to hijack democracy in your country!” –Sharon Fummerton, BC

“With your power you could (and should) do good instead of furthering ALEC’s poisonous agenda.” –Barbara Coulson, NC

Some propose a course of action:

“I will boycott any company affiliated with ALEC” –Jim Knipe, VA

“Don’t make me start using Bing.” –Liz Neff, CA

Some are taking action already:

“I am already using DuckDuckGo for web searches. I’m pretty sure I can find an email service other than Gmail.” –Dan Starr, IL

“Just changed my search engine.” –Michael Keenan, IL

“I already changed my home page to Yahoo.” –Nadia Daley

“I have stopped using Google completely due to it’s support of these ultra conservative and regressive groups.” –Kelley Dempsey, MD

“Until Google makes it clear that it no longer supports Alec and other anti-democratic, anti labour, and environmentally destructive causes I will conduct my searches through other engines.” –Glenn Ashton

Some have a more serious solution in mind:

“Google is Getting too big and powerful, break it up!” –Stephen Rawlings, FL

“When corporations get a fairer deal than the tax paying population, something needs to CHANGE!” –Debbie Boozer, IN

http://www.washingtonsblog.com/