J. G. Sandom ~ Author

4o4 — How whistleblowers Drake and Crane were so badly treated that no one's incentivized to tell the truth about cost over-runs

Excellent article about NSA whistleblowers Thomas Drake (NSA) and John Crane (DoD) and how the Pentagon punished them for speaking out about cost over-runs. If you want to get a sense of how the government makes it virtually impossible for whistleblowers to come forward, read this piece. When Snowden came forward, many criticized him for not going through standard whistleblower channels. This article spells out why that would have been impossible beyond what he did; yes, Snowden first tried to raise the alarm internally. How whistleblowers are treated, and the constant struggle between security and privacy—these are the issues I wrestle with at length in my latest thriller, 4o4 - A John Decker Thriller, about the surveillance state, recently listed as a Top Ten Amazon Bestseller in Technothrillers. For more on this story, visit The Guardian.

How the Pentagon punished NSA whistleblowers

Long before Edward Snowden went public, John Crane was a top Pentagon official fighting to protect NSA whistleblowers. Instead their lives were ruined – and so was his

• Snowden calls for whistleblower shield after claims by Pentagon source
• Exclusive: Pentagon source goes on record against whistleblower program

by Mark Hertsgaard

By now, almost everyone knows what Edward Snowden did. He leaked top-secret documents revealing that the National Security Agency was spying on hundreds of millions of people across the world, collecting the phone calls and emails of virtually everyone on Earth who used a mobile phone or the internet. When this newspaper began publishing the NSA documents in June 2013, it ignited a fierce political debate that continues to this day – about government surveillance, but also about the morality, legality and civic value of whistleblowing.

But if you want to know why Snowden did it, and the way he did it, you have to know the stories of two other men.

The first is Thomas Drake, who blew the whistle on the very same NSA activities 10 years before Snowden did. Drake was a much higher-ranking NSA official than Snowden, and he obeyed US whistleblower laws, raising his concerns through official channels. And he got crushed.

Drake was fired, arrested at dawn by gun-wielding FBI agents, stripped of his security clearance, charged with crimes that could have sent him to prison for the rest of his life, and all but ruined financially and professionally. The only job he could find afterwards was working in an Apple store in suburban Washington, where he remains today. Adding insult to injury, his warnings about the dangers of the NSA’s surveillance programme were largely ignored.

“The government spent many years trying to break me, and the more I resisted, the nastier they got,” Drake told me.

Drake’s story has since been told – and in fact, it had a profound impact on Snowden, who told an interviewer in 2015 that: “It’s fair to say that if there hadn’t been a Thomas Drake, there wouldn’t have been an Edward Snowden.”

But there is another man whose story has never been told before, who is speaking out publicly for the first time here. His name is John Crane, and he was a senior official in the Department of Defense who fought to provide fair treatment for whistleblowers such as Thomas Drake – until Crane himself was forced out of his job and became a whistleblower as well.

His testimony reveals a crucial new chapter in the Snowden story – and Crane’s failed battle to protect earlier whistleblowers should now make it very clear that Snowden had good reasons to go public with his revelations.

During dozens of hours of interviews, Crane told me how senior Defense Department officials repeatedly broke the law to persecute Drake. First, he alleged, they revealed Drake’s identity to the Justice Department; then they withheld (and perhaps destroyed) evidence after Drake was indicted; finally, they lied about all this to a federal judge.

The supreme irony? In their zeal to punish Drake, these Pentagon officialsunwittingly taught Snowden how to evade their clutches when the 29-year-old NSA contract employee blew the whistle himself. Snowden was unaware of the hidden machinations inside the Pentagon that undid Drake, but the outcome of those machinations – Drake’s arrest, indictment and persecution – sent an unmistakable message: raising concerns within the system promised doom.

“Name one whistleblower from the intelligence community whose disclosures led to real change – overturning laws, ending policies – who didn’t face retaliation as a result. The protections just aren’t there,” Snowden told the Guardian this week. “The sad reality of today’s policies is that going to the inspector general with evidence of truly serious wrongdoing is often a mistake. Going to the press involves serious risks, but at least you’ve got a chance.”

Snowden saw what had happened to Drake and other whistleblowers like him. The key to Snowden’s effectiveness, according to Thomas Devine, the legal director of the Government Accountability Project (GAP), was that he practised “civil disobedience” rather than “lawful” whistleblowing. (GAP, a non-profit group in Washington, DC, that defends whistleblowers, has represented Snowden, Drake and Crane.)

“None of the lawful whistleblowers who tried to expose the government’s warrantless surveillance – and Drake was far from the only one who tried – had any success,” Devine told me. “They came forward and made their charges, but the government just said, ‘They’re lying, they’re paranoid, we’re not doing those things.’ And the whistleblowers couldn’t prove their case because the government had classified all the evidence. Whereas Snowden took the evidence with him, so when the government issued its usual denials, he could produce document after document showing that they were lying. That is civil disobedience whistleblowing.”

Crane, a solidly built Virginia resident with flecks of grey in a neatly trimmed chinstrap beard, understood Snowden’s decision to break the rules – but lamented it. “Someone like Snowden should not have felt the need to harm himself just to do the right thing,” he told me.

Crane’s testimony is not simply a clue to Snowden’s motivations and methods: if his allegations are confirmed in court, they could put current and former senior Pentagon officials in jail. (Official investigations are quietly under way.)

But Crane’s account has even larger ramifications: it repudiates the position on Snowden taken by Barack Obama and Hillary Clinton – who both maintain that Snowden should have raised his concerns through official channels because US whistleblower law would have protected him.

By the time Snowden went public in 2013, Crane had spent years fighting a losing battle inside the Pentagon to provide whistleblowers the legal protections to which they were entitled. He took his responsibilities so seriously, and clashed with his superiors so often, that he carried copies of the Whistleblower Protection Act of 1989 and the US constitution in his breast pocket and pulled them out during office conflicts.

Crane’s attorneys at GAP – who were used to working with all types of government and corporate whistleblowers – were baffled by him: in their experience, most senior government officials cared little for whistleblowers’ rights. So what motivated Crane to keep fighting for the rights of whistleblowers inside the Pentagon, even as his superiors grew increasingly hostile and eventually forced him to resign?

To hear Crane tell it, the courage to stand up and fight runs in his family. He never forgot the story he heard as a child, about his own grandfather, a German army officer who once faced down Adolf Hitler at gunpoint – on the night the future Fuhrer first tried to take over Germany.

A former press aide to Republican members of Congress, John Crane was hired by the Inspector General’s office of the Department of Defense in 1988. Within US government agencies, an inspector general serves as a kind of judge and police chief. The IG, as the inspector general is known, is charged with making sure a given agency is operating according to the law – obeying rules and regulations, spending money as authorised by Congress. “In the IG’s office, we were the guys with the white hats,” Crane said.

By 2004 Crane had been promoted to assistant inspector general. At the age of 48, his responsibilities included supervising the whistleblower unit at the Department of Defense, as well as handling all whistleblower allegations arising from the department’s two million employees (by far the largest workforce in the US government), in some cases including allegations originating in the NSA and other intelligence agencies.

Thomas Drake, NSA whistleblower, in a still from the Robert Greenwald documentary War on Whistleblowers.

By this time, Thomas Drake had proceeded well down the path that would eventually connect him with Crane. Drake’s first day as a fully fledged employee of the National Security Agency was 11 September 2001. Although the NSA would balloon in size and budget as the US responded to the September 11 attacks, the agency already ranked as the largest, most lavishly funded spy organisation on Earth. Created in 1952, the NSA was the government’s code-breaker and all-hearing global “ear”. The NSA intercepted the communications of foreign governments and individuals and translated this raw intelligence into information usable by the CIA, the FBI and kindred government agencies.

Drake, a father of five, had worked for the NSA for 12 years as a private-sector contractor. Now, as a staff member proper, he reported directly to the NSA’s third highest ranking official, Maureen Baginski; she headed the NSA’s largest division, the Signals Intelligence Directorate, which was responsible for the interception of phone calls and other communications.

Tall, sombre, intense, Drake was a championship chess player in high school whose gift for mathematics, computers and languages made him a natural for foreign eavesdropping and the cryptographic and linguistic skills it required. During the cold war, he worked for air force intelligence, monitoring the communications of East Germany’s infamous secret police, the Stasi.

Within weeks of the September 11 attacks, Drake was assigned to prepare the NSA’s postmortem on the disaster. Congress, the news media and the public were demanding answers: what had gone wrong at the NSA and other federal agencies to allow Osama bin Laden’s operatives to conduct such a devastating attack?

As Drake interviewed NSA colleagues and scoured the agency’s records, he came across information that horrified him. It appeared that the NSA – even before September 11 – had secretly revised its scope of operations to expand its powers.

Since its inception, the NSA had been strictly forbidden from eavesdropping on domestic communications. Drake’s investigation persuaded him that the NSA was now violating this restriction by collecting information on communications within as well as outside of the United States. And it was doing so without obtaining legally required court orders.

A straight arrow since high school – he once gave the police the names of classmates he suspected of selling pot – Drake told me he felt compelled to act. “I took an oath to uphold and defend the constitution against all enemies foreign and domestic,” he explained.

To Drake, the President’s Surveillance Program, as it was known inside the George W Bush administration, recalled the mindset of the Stasi. “You don’t spend year after year listening to a police state without being affected, you just don’t,” he told me. “I remember saying to myself, ‘Wow, I don’t want this to happen in our country!’ How could you live in a society where you always have to be looking over your shoulders, not knowing who you could trust, even in your own family?”

Drake’s descent into a nightmare of persecution at the hands of his own government began innocently. Having uncovered evidence of apparently illegal behaviour, he did what his military training and US whistleblower law instructed: he reported the information up the chain of command. Beginning in early 2002, he shared his concerns first with a small number of high-ranking NSA officials, then with the appropriate members of Congress and staff at the oversight committees of the US Senate and House of Representatives.

Drake spent countless hours in these sessions but eventually came to the conclusion that no one in a position of authority wanted to hear what he was saying. When he told his boss, Baginski, that the NSA’s expanded surveillance following 9/11 seemed legally dubious, she reportedly told him to drop the issue: the White House had ruled otherwise.

John Crane first heard about Thomas Drake when Crane and his colleagues at the Pentagon’s Office of the Inspector General received a whistleblower complaint in September 2002. The complaint alleged that the NSA was backing an approach to electronic surveillance that was both financially and constitutionally irresponsible. The complaint was signed by three former NSA officials, William Binney, Kirk Wiebe and Edward Loomis, and a former senior Congressional staffer, Diane Roark. Drake also endorsed the complaint – but because he, unlike the other four, had not yet retired from government service, he asked that his name be kept anonymous, even in a document that was supposed to be treated confidentially within the government.

Binney, Wiebe, Loomis and Roark shared Drake’s concerns about the constitutional implications of warrantless mass surveillance, but their complaint focused on two other issues.

The first was financial. The whistleblowers contended that the NSA’s surveillance programme, codenamed Trailblazer, was a shameful waste of $3.8 billion – it had been more effective at channelling taxpayer dollars to corporate contractors than at protecting the homeland.

Second, the whistleblowers warned that Trailblazer actually made the US less secure. They acknowledged that Trailblazer had vastly expanded the amount of electronic communications NSA collected. But this avalanche of raw data was too much – it left NSA’s analysts struggling to distinguish the vital from the trivial and thus liable to miss key clues.

Drake had discovered a shocking example while researching his postmortem report on the September 11 attacks. Months beforehand, the NSA had come into possession of a telephone number in San Diego that was used by two of the hijackers who later crashed planes into the World Trade Center. But the NSA did not act on this finding.

As Drake later told the NSA expert James Bamford, the NSA intercepted seven phone calls between this San Diego phone number and an al-Qaida “safe house” in Yemen. Drake found a record of the seven calls buried in an NSA database.

US officials had long known that the Yemen safe house was the operational hub through which Bin Laden, from a cave in Afghanistan, ordered attacks. Seven phone calls to such a hub from the same phone number was obviously suspicious. Yet the NSA took no action – the information had apparently been overlooked.

The NSA whistleblowers first sent their complaint to the inspector general of the NSA, who ruled against them. So they went up the bureaucratic ladder, filing the complaint with the Department of Defense inspector general. There, Crane and his staff “substantially affirmed” the complaint – in other words, their own investigation concluded that the NSA whistleblowers’ charges were probably on target.

In the course of their investigation, Crane and his colleagues in the inspector general’s office also affirmed the whistleblowers’ allegation that the Bush administration’s surveillance programme violated the fourth amendment of the US constitution by collecting Americans’ phone and internet communications without a warrant. “We were concerned about these constitutional issues even before we investigated their complaint,” Crane told me. “We had received other whistleblower filings that flagged the issue.”

In line with standard procedure, these investigative findings were relayed to the House and Senate committees overseeing the NSA – and this helped nudge Congress to end funding for the Trailblazer programme. But for the NSA whistleblowers, this apparent victory was the beginning of a dark saga that would change their lives for ever.

Illustration by Nathalie Lees

The Bush administration’s mass surveillance efforts were partly exposed in December 2005, when the New York Times published a front page article by reporters James Risen and Eric Lichtblau, which revealed that the NSA was monitoring international phone calls and emails of some people in the US without obtaining warrants.

Eight years later, that story would be dwarfed by Snowden’s revelations. But at the time, the Bush White House was furious – and they were determined to find and punish whoever had leaked the details to the New York Times.

According to Crane, his superiors inside the Pentagon’s Inspector General’s office were eager to help. Henry Shelley, the general counsel – the office’s top lawyer – urged that the IG office should tell the FBI agents investigating the Times leak about Drake and the other NSA whistleblowers.

After all, the NSA whistleblowers’ recent complaint had objected to the same surveillance practices described in the Times article – which made them logical suspects in the leak. Crane objected strenuously. Informing anyone – much less FBI investigators – of a whistleblower’s name was illegal.

After debating the matter at a formal meeting in the personal office of the inspector general, Shelley and Crane continued arguing in the hallway outside. “I reached into my breast pocket and pulled out my copy of the Whistleblower Protection Act,” Crane recalled. “I was concerned that Henry was violating the law. Our voices weren’t raised, but the conversation was, I would say, very intense and agitated. Henry [replied] that he was the general counsel, the general counsel was in charge of handling things with the Justice Department and he would do things his way.”

Henry Shelley declined my repeated requests for an interview. In an email, he told me, “I am confident when this matter is fully resolved no wrongdoing on my behalf will be identified.”

There the disagreement between Crane and Shelley stalled. Or so it seemed until 18 months later. On the morning of 26 July, 2007, FBI agents with guns drawn stormed the houses of Binney, Wiebe, Loomis and Roark. Binney was towelling off after a shower when agents accosted him; he and his wife suddenly found themselves with guns aimed directly between their eyes, the retired NSA man recalled.

Crane smelled a rat. The investigation that his staff had conducted into the whistleblowers’ complaint had been highly classified: very few people could have known their names, and they would have been inside the IG’s office. After the raids, Crane confronted Shelley and demanded to know whether the IG’s office had given the names to the FBI. Shelley refused to discuss the matter, Crane says.

The battle soon escalated. Four months later, FBI agents stormed Drake’s house in an early morning raid, as his family watched in shock.

After Drake was indicted in 2010, his lawyers filed a Freedom of Information Act request to obtain documents related to the investigation Crane’s office had conducted into the claims of the NSA whistleblowers. According to Crane, he was ordered by his superiors in the IG’s office to delay releasing any documents – which could have exonerated Drake – until after the trial, which was expected to take place later in 2010.

Crane alleges that he was ordered to do so by Shelley and Lynne Halbrooks – who had recently been named the principal deputy inspector general (in other words, the second-highest ranking official in the IG’s office). Crane protested but lost this skirmish as well. (Halbrooks did not respond to repeated requests for an interview.)

In December 2010, nearly five years after the Pentagon’s inspector general’s office had apparently given Drake’s name to FBI investigators, Drake’s lawyers filed a complaint with the inspector general, alleging that Drake had been punished in retaliation for his whistleblowing. According to their complaint, the crimes Drake had been charged with were “based in part, or entirely, on information that Mr Drake provided to the [Pentagon] IG” during its investigation of the NSA whistleblowers.

Crane was at once alarmed and revolted. The complaint from Drake’s lawyers seemed to confirm his suspicion that someone in the IG’s office had illegally fingered Drake to the FBI. Worse, the indictment filed against Drake had unmistakable similarities to the confidential testimony Drake had given to Crane’s staff – suggesting that someone in the IG’s office had not simply given Drake’s name to the FBI, but shared his entire testimony, an utter violation of law.

Drake’s complaint demanded investigation, Crane told Halbrooks. But Halbrooks, joined by Shelley, allegedly rejected Crane’s demand. She added that Crane wasn’t being a “good team player” and if he didn’t shape up, she would make life difficult for him.

But there was even worse to come. As Drake’s trial approached in the spring of 2011, Crane knew that the law required the IG’s office to answer the retaliation complaint filed by Drake’s lawyers. But, Crane says, Shelley now informed him it would be impossible to respond – because the relevant documents had been destroyed. Lower level staff “fucked up”, Crane said Shelley told him: they had shredded the documents in a supposedly routine purge of the IG’s vast stores of confidential material.

Crane could not believe his ears. “I told Henry that destruction of documents under such circumstances was, as he knew, a very serious matter and could lead to the inspector general being accused of obstructing a criminal investigation.” Shelley replied, according to Crane, that it didn’t have to be a problem if everyone was a good team player.

On 15 February, 2011, Shelley and Halbrooks sent the judge in the Drake case a letter that repeated the excuse given to Crane: the requested documents had been destroyed, by mistake, during a routine purge. This routine purge, the letter assured Judge Richard D Bennett, took place before Drake was indicted.

“Lynne and Henry had frozen me out by then, so I had no input into their letter to Judge Bennett,” Crane said. “So they ended up lying to a judge in a criminal case, which of course is a crime.”

With Drake adamantly resisting prosecutors’ pressure to make a plea deal – “I won’t bargain with the truth,” he declared – the government eventually withdrew most of its charges against him. Afterwards, the judge blasted the government’s conduct. It was “extraordinary”, he said, that the government barged into Drake’s home, indicted him, but then dropped the case on the eve of trial as if it wasn’t a big deal after all. “I find that unconscionable,” Bennett added. “Unconscionable. It is at the very root of what this country was founded on … It was one of the most fundamental things in the bill of rights, that this country was not to be exposed to people knocking on the door with government authority and coming into their homes.”

When John Crane put his career on the line by standing up for legal treatment of Pentagon whistleblowers, he was following a moral code laid down 80 years before by his German grandfather. Crane grew up in suburban Virginia, but he spent nearly every summer in Germany with his mother’s extended family.During these summer sojourns, Crane heard countless times about the moment when his grandfather confronted Hitler. His mother and his grandmother both told the story, and the moral never changed. “One must always try to do the right thing, even when there are risks,” Crane recalled being instructed. “And should someone do the right thing, there can of course be consequences.”

Crane’s grandfather was days shy of turning 40 on the night of Hitler’s â€œBeer Hall Putsch”, 8 November, 1923. Plotting to overthrow the Weimar Republic, Hitler and 600 armed members of his fledgling Nazi party surrounded a beer hall in Munich where the governor of Bavaria, Gustav von Kahr, was addressing a large crowd. The rebels burst into the hall, hoping to kidnap Von Kahr and march on Berlin. After his men unveiled a machine gun hidden in the upstairs gallery, Hitler fired his pistol into the air and shouted, “The national revolution has begun!”

Crane’s grandfather, Günther Rüdel, was in the hall as part of his military duties, Rüdel recalled in an eight-page, single-spaced, typewritten affidavit that provides a minute-by-minute eyewitness account of the putsch. (Rüdel was later a government witness in the trial that sentenced Hitler to five years in prison, though he was not called to testify.)

The son of a prominent German general, Rüdel had served with distinction in the first world war, earning two Iron Crosses. By 1923, he was serving as chief political aide to General Otto von Lossow, the German army’s highest official in Bavaria. As such, Rüdel was the chief liaison between Von Lossow and Von Kahr and privy to the two men’s many dealings with Hitler. Suspecting that Hitler and his followers were planning a coup, Lossow and Rüdel had forced their way into the beer hall to monitor developments. The head of Bavaria’s state police, Hans Ritter von Seisser, was also there, accompanied by a bodyguard. Rüdel was standing with Lossow and Von Seisser when armed men burst into the hall, with Hitler in the lead.

“Hitler, with pistol held high, escorted on right and left by armed men, his tunic stained with beer, stormed through the hall towards the podium,” Rüdel wrote in his affidavit. “When he was directly in front of us, police chief Von Seisser’s adjutant gripped [but did not unsheath] his sword. Hitler immediately aimed his pistol at the man’s chest. I shouted, ‘Mr Hitler, in this way you will never liberate Germany.’ Hitler hesitated, lowered his pistol and pushed his way between us to the podium.”

In the surrounding chaos, Hitler’s men tried to force Von Kahr, Lossow and Von Seisser to join the coup, but their uprising soon fizzled. A few days later, Hitler was arrested and charged with treason. He served a year in jail, where he wrote his autobiography, Mein Kampf.

“We are now becoming a police state,” Diane Roark said in a 2014 television interview. Referring to herself and the other NSA whistleblowers, she added, “We are the canaries in the coal mine. We never did anything wrong. All we did was oppose this programme. And for that, they just ran over us.”

“They’re saying, ‘We’re doing this to protect you,’” Roark’s fellow whistleblower William Binney told me. “I will tell you that that’s exactly what the Nazis said in Special Order 48 in 1933 – we’re doing this to protect you. And that’s how they got rid of all of their political opponents.”

These are strong statements – comparing the actions of the US government to Nazi Germany, warning of an emerging “police state” – so it’s worth remembering who made them. The NSA whistleblowers were not leftwing peace nuts. They had spent their professional lives inside the US intelligence apparatus – devoted, they thought, to the protection of the homeland and defence of the constitution.

They were political conservatives, highly educated, respectful of evidence, careful with words. And they were saying, on the basis of personal experience, that the US government was being run by people who were willing to break the law and bend the state’s awesome powers to their own ends. They were saying that laws and technologies had secretly been put in place that threatened to overturn the democratic governance Americans took for granted and shrink their liberties to a vanishing point. And they were saying that something needed to be done about all this before it was too late.

In Washington, top government officials and politicians still insist that the true villain is Edward Snowden. Former CIA director James Woolsey has called for Snowden to be “hanged by the neck until he’s dead, rather than merely electrocuted”.

Democrats are less bloodthirsty, but no more forgiving. President Obama and Hillary Clinton argue that Snowden broke the law when he should have trusted it. “He could have gotten all of the protections of being a whistleblower,” Clinton said in the first Democratic presidential debate last October. “He could have raised all the issues that he has raised. And I think there would have been a positive response to that.”

Tell that to Thomas Drake. Tell it, for that matter, to John Crane.

Halbrooks forced Crane to resign his post in January 2013. After leaving the Pentagon, Crane made his way to the Government Accountability Project, where the erstwhile protector of whistleblowers became a whistleblower himself.

Crane filed a complaint against Shelley and Halbrooks, detailing many more alleged misdeeds than reported in this article. The Office of Special Counsel, the US agency charged with investigating such matters, concluded in March of 2016 that there was a “substantial likelihood” that Crane’s accusations were well-founded. The OSC’s choice of the term “substantial likelihood” was telling. It could have ruled there was merely a “reasonable belief” Crane’s charges were true, in which case no further action would have been required. By finding instead that there was a “substantial likelihood”, the OSC triggered a process that legally required secretary of defense Ashton Carter to organise a fresh investigation of Crane’s allegations. Because no federal agency is allowed to investigate itself, that inquiry is being conducted by the Justice Department.

Incredible as it may sound, Crane aims to get his old job back. His attorney, Devine, thinks that is a fantasy. In Devine’s view, the problems facing whistleblowers are systemic – and the system does not forgive, especially someone who has exposed the system’s corruption as devastatingly as Crane has done.

To Crane, however, it is a simple matter of right and wrong. It was not he who broke the law; it was his superiors. Therefore it is not he who should pay the price but they.

“I just want to see the system work properly,” he says. “I know the system can fail – world war two, Nazi Germany – but I also know that you need to do what is right. Because the government is so powerful, you need to have it run efficiently and honestly and according to the law.”

“What are the odds the system will work properly in your case?” I asked Crane.

“I’m not giving you odds,” he replies with a chuckle. “This is just something that I have to do.”

This article is adapted from Mark Hertsgaard’s new book, Bravehearts: Whistle Blowing in the Age of Snowden (Hot Books/Skyhorse)

4o4 — Even basic phone logs can reveal deeply personal information, researchers find

The mass collection of telephone records by government surveillance programs poses a clear threat to the personal privacy of ordinary citizens, according to US researchers who used basic phone logs to identify people and uncover confidential information about their lives. When this issue was being debated a year or so ago, folks made a big deal about the fact that the government was only gathering metadata. Now, research shows just how dangerous this is to privacy. These are exactly the issues I cover in my latest novel, 4o4 - A John Decker Thriller, about the surveillance state, recently listed as a Top Ten Amazon Bestseller in Technothrillers. For more on this story, visit The Guardian.

The NSA collected calling records of millions of Americans under a controversial programme leaked by former agency contractor Edward Snowden.

Even basic phone logs can reveal deeply personal information, researchers find

Stanford study shows how details gleaned from telephone ‘metadata’ by National Security Agency pose a threat to privacy of ordinary citizens

by Ian Sample, Science editor

The mass collection of telephone records by government surveillance programs poses a clear threat to the personal privacy of ordinary citizens, according to US researchers who used basic phone logs to identify people and uncover confidential information about their lives.

Armed with anonymous “metadata” on people’s calls and texts, but not the contents of the communications, two scientists at Stanford University worked out individuals’ names, where they lived and the names of their partners. But that was not all.

The same data led them to uncover potentially sensitive information about some individuals. One man was found to own a rifle, while another had recently been diagnosed with an irregular heartbeat. Other data pointed to a new pregnancy, a person with multiple sclerosis, and an individual who was gearing up to grow cannabis.

The results highlight the extraordinary power of telephone metadata – the number called, when, and for how long – particularly when it is paired with public information available from services such as Google, Yelp and Facebook. The value of the data, which is not subject to the same legal protections as the content of people’s communications, has long been recognised by the security services. As Stewart Baker, the former general counsel at the US National Security Agency put it in the aftermath of Edward Snowden’s revelations: “Metadata absolutely tells you everything about somebody’s life.”

Patrick Mutchler, a computer security researcher at Stanford, said that while the power of metadata was understood by those gathering the information, the public was largely in the dark because so few published studies have revealed how rich the data are. “That makes it difficult for people with strong opinions about these programs to fight them. Now we have hard evidence we can point to that didn’t exist in the past,” he said.

For the study, the researchers signed up 823 people who agreed to have metadata collected from their phones through an Android app. The app also received information from their Facebook accounts, which the scientists used to check the accuracy of their results. In all, the researchers gathered metadata on more than 250,000 calls and over 1.2m texts.

Analysts who logged into the NSA’s metadata gathering system were initially allowed to examine data up to three hops away from an individual. A call from the target individual’s phone to another number was one hop. From that phone to another was two hops. And so on. The records available to analysts stretched back for five years. The collection window has now been restricted to two hops and 18 months at most.

The Stanford study found that armed with one phone number to start from, the NSA program would initially have given analysts access to telephone metadata for tens of millions of people. Once restrictions came into place, that number fell dramatically, but it still meant that armed with a single phone number, an NSA analyst could retrieve metadata on 25,000 people.

Writing in the journal Proceedings of the National Academy of Sciences, Mutchler describes how on a shoestring budget, he and fellow graduate student, Jonathan Mayer, uncovered a wealth of personal information, some of it sensitive, about people who took part in the study. Through automatic and manual searches, they identified 82% of people’s names. The same technique gave them the names of businesses the people had called. When these were plotted on a map, they revealed clusters of local businesses, which the scientists speculated surrounded the person’s home address. In this way, they named the city people lived in 57% of the time, and were nearly 90% accurate in placing people within 50 miles of their home. Mutchler believes some of the misses came from people not updating their Facebook page when they moved out of their parents’ home, for example, to go to college.

The scientists next delved into more personal territory. Using a simple computer program to analyse people’s call patterns, they inferred who among the study volunteers was in a relationship. Once they knew the owner of a particular number had a partner, identifying the significant other was trivial, they report.

For the final part of the study, the researchers delved even deeper, to see what sensitive information they could glean from telephone metadata. They gathered details on calls made to and from a list of organisations, including hospitals, pharmacies, religious groups, legal services, firearms retailers and repair firms, marijuana dispensaries, and sex establishments. From these, they pieced together some extraordinary vignettes from people’s lives.

The metadata from one person in the study showed they had a long call from a cardiology centre; spoke briefly with a medical laboratory; answered a number of short calls from a local pharmacy, and then made calls to a hotline for abnormal heart-rate monitoring devices. Another participant made frequent calls to a local gun supplier that specialised in semi-automatic rifles, and later placed a number of long calls to the customer support hotline run by a major gun manufacturer that produced the rifles. Another still placed calls to a hardware store, a locksmiths, a hydroponics supplier and a head shop in the space of three weeks. The metadata from two others suggested one had multiple sclerosis and the other had just become pregnant.

“All of this should be taken as an indication of what is possible with two graduate students and limited resources,” said Mutchler, who argues that the findings should make policymakers think twice before authorising mass surveillance programs. “Large-scale metadata surveillance programs, like the NSA’s, will necessarily expose highly confidential information about ordinary citizens,” the scientists write, adding: “To strike an appropriate balance between national security and civil liberties, future policymaking must be informed by input from relevant sciences.”

Ross Anderson, professor of security engineering at Cambridge University, said the study provided numbers that discussions can now be based on. “With the right analytics running over nation-scale comms data you can infer huge amounts of sensitive information on everyone. We always suspected that of course, but here’s the data.”