States of surveillance

indexby Andy Warren

Amongst growing public outrage in the US and international scandal surfacing around previously secret National Security Agency (NSA) activities, let’s take a quick look into the current shenanigans around the NSA, PRISM, Edward Snowden and, in New Zealand, the GCSB amendment.

Firstly, what does it mean when we’re told the government is collecting data about us?  Some history, technical basics and background will be useful.

An exchange of information requires two parties, a message, and a method of transmitting the message. We probably take it for granted in our day-to-day communications, but, in practice we also employ mechanisms for verifying authenciticy and accurate receipt of the message.

In the days of the pony express or telegraph, the message was written on paper and transmitted to a specific destination where the recipient was likely to be. The pony express rider didn’t need to know the contents of the message, merely the delivery location. The rider who first carries the message may not deliver it to its final destination – instead passing it on to another rider at a relay station.

In the case of the telegraph, the act of transmitting the message – converting it into Morse Code – required the telegrapher to read the message and perform the translation and keying of the morse code. The telegrapher hence also had to be trustworthy.

In both these early examples of telecommunication, a number of pieces of information and data can be obtained without the knowledge or agreement of the two communicating parties:

  • Where the parties were when the message was transmitted (and received)
  • The names of the sender and recipient
  • The message itself

A pony express rider could be held-up at gunpoint and forced to hand over the message, a copy of which could be taken and the rider sent on their way. The rider would be aware of this “overt intrusion” and could alert the recipient upon delivery. Alternatively, a covert intrusion might involve entering the rider’s hotel room and reading the messages in their courier bag or replacing one or more of them with fake messages (misinformation) while the rider is asleep.  If the rider was killed, or the message stolen, yet the recipient was expecting the message, then an additional exchange between the recipient and sender would soon pinpoint the intrusion and loss of the information concerned. In the case of misinformation, the question arises as to how the recipient can be certain of the authenticity of the message?

The transmission of a message over the electric telegraph could be intercepted by simply connecting receiving equipment to the wires at some point along the length of the wire.  Any message thus intercepted could be translated from morse code as this was being done at the actual receiving station. A telegraph office could be taken over at gunpoint and then surrendered after an intrusion – this is referred to as a “man in the middle” intrusion.  The sender may be unaware their message has been received by an interloper and any responses untrustworthy.

For obvious reasons, as early as the American Civil war, military communications were encrypted (made unreadable to anyone other than the sending and receiving parties) using some of the earliest “ciphers” or codes. The telegraph wire was also physically protected by cavalry patrols.

The history of the American Civil War includes a number of famous intrusions and the successful deciphering of Confederate messages, so obtained, by Union cryptographers.

In medieval times the message sender would “authenticate” and protect a document by using a wax seal – imprinted by a ring or stamp recognisable by the recipient.  The details of the stamp were sufficiently difficult to forge at that time rendering this a relatively effective method of ensuring the authenticity of a message, and its safe arrival un-altered and private. Earlier still the Romans employed Seal boxes to protect packages, parcels and boxes in transit.

Early radio transmissions were “omni-directional” – and could be received by anyone with suitable listening equipment within reception range.  While the content of a voice transmission could be easily obtained, the location of the transmitter was more difficult to calculate. This required equipment capable of measuring the signal strength and direction, and hence, through triangulation, the location of the transmitter.  During WWII for example, resistance groups would be careful to transmit for only very short periods from the same location – to make it difficult for Axis listening patrols to triangulate their location.  Conversely, they would physically separate themselves from the transmitting equipment to avoid being caught with their transmitter.

Early analog encryption included the technique referred to as “scrambling” which basically comprises the rearrangement of frequency ranges to render the signal unintelligible – the reverse process must occur at the receiving end before a human can understand what is being said. These were replaced by digital signal processing technologies and a few remain in use as relatively cheap basic encryption measures.

The development of directional radio transmitters improved matters (in applications demanding it) by limiting the signal to a much narrower region between the sender and receiver. The radiation pattern emitted by an antenna is a series of “lobes” resembling the petals of a flower. Cellphone towers typically have three directional antennae each covering approximately a third of the tower’s reception area.

It can be argued that very little has changed in respect of the basic parameters of a message exchange, as outlined above, over the internet or modern networks.  The internet itself is similar in structure to the pony express network of relay stations – where messages were handed over by an incoming rider and sent on their way to the next station on the best route to the recipient with another rider.

The phone network made famous by Alexander Graham Bell operates to this day on the same basic technology. As with the telegraph, it is possible to “listen” to a conversation without alerting either party to the intrusion.

The cellular network is based on radio transmission towers and receivers moving relative to them. Your cell phone is constantly communicating with several towers to decide which one is best. For a short time, your phone will send and receive voice data via the tower it is closest to. When you begin moving away from that tower and another comes into range providing a better signal – your conversation will switch towers.  In this manner, once again using triangulation techniques, it is possible to ascertain your position very accurately based on which cell phone towers were in range at a certain time.  Cell phone network providers maintain historical logs of such exchanges which must be surrendered to authorities when requested under warrant.

Where the modern internet differs from the pony express is its use of message fragments.  If I wished to send a sizeable document, written by hand, by pony express, I would be risking a lot by sending it all in a single package with one rider – it could be stolen, damaged or lost. There might also be limitations on how much a single rider could carry as well as the need for a rider to carry many messages simultaneously.  My single large document would monopolise the “available bandwidth” of the rider (network) so slowing the transmission of other messages.

If I broke my single large document up into separate pieces and sent these separately, possibly using additional riders, I would improve the efficiency of the “message transmission network” in several ways:

  • a standard message size – the rider could always fit the same number of messages into their post bag. This makes the behaviour of the network more predictable.
  • messages would be able to move through the network at a predictable rate – without any single message monopolising transmission
  • loss of a message would require retransmission of only a small portion of the overall document
  • the first parts of the message would begin arriving before the last were sent or while some of the message is still in transit – the recipient could begin reading immediately (today this more-or-less describes “streaming” technologies – where, eg., you can begin watching a movie with only a fragment of the whole movie downloaded)
  • If a river, normally passable, was in flood for an extended period, outstanding messages could be rerouted to arrive safely via a longer alternative route. Although these would arrive “out of sequence” relative to the overall document – the pieces could be reassembled without too much difficulty.

I won’t go into the details of modern transmission or routing techniques, nor labour too far, I hope, with the antique examples. Suffice it to say the basic principles remain applicable to this discussion.

With some background in place then, what, in this context, is the NSA up to with PRISM and its various covert strategies to eavesdrop the private exchanges of  (not only) American citizens?

According to various internet sources, mostly quoting snippets from the mouth of Edward Snowden, and variously confirmed by freedom of speech and privacy advocates – PRISM is by no means the only mechanism by which the US government obtains, covertly, copies of private messages. Let’s take a broader look at what the NSA is and has been doing.

There is a distinction here between what is referred to as “upstream surveillance” and direct access to service providers.  Whereas PRISM is allegedly via legislated “back door” access to the servers of service providers based on US soil…

“Other programs — codenamed FAIRVIEW, STORMBREW, BLARNEY, and OAKSTAR — work differently. These programs involve collecting all traffic, either by tapping undersea fiber optic cables or capturing traffic travelling through Internet routers and gateways located in the USA. It’s long been known that the NSA has secret rooms at Internet service providers and routing companies where they can intercept and monitor the data flowing past. Room 641A at the AT&T office in San Francisco was the first such room that we learned about back in 2006.” What Is PRISM? Everything You Need to Know(1) See also “Multi-communication Transactions”, and “…Access to Fiber Optic Cables”

While PRISM has been attracting the lion’s share of coverage, the alarming point should be that much larger surveillance programs predate PRISM and, from an initial reading, dwarf PRISM in the sheer volume of data they can and are collecting.

“Under these Upstream programs, the NSA probably has the ability to capture most of the data being transmitted over the Internet. They’re building a massive data center in Utah, likely to store and analyze all this data. These upstream programs are capturing much more data and surveilling many more people than PRISM is.”(1)

Just how much data is being transmitted daily over the internet and how we might visualise it is helped by a nice infographic.  If Facebook is processing 500 Terabytes of data each day, and nearly 300 Billion emails are sent, how much space would the NSA require if it were to keep up with this data volume – processing as it goes and discarding what it has looked at or what is uninteresting or too old to store (replaced by incoming new data)?  I have no idea how you’d answer this, but it might look like this and judging by how many movies I can fit on a few small USB drives under my TV – they’ll have little trouble and plenty of budget to keep and process as much as they need.

Ok, so, by various means – including covert “wiretapping” the internet backbone through which 99% of the world’s internet traffic passes, and “back door” access via PRISM – what is the NSA doing with all this data once it has it? It sure as hell isn’t reading everyone’s emails and watching hours of ameteur porn and cat videos. The approach here is quite simply a “drag net” using sophisticated search and profiling algorithms not dissimilar to Google’s approach to search engine technology (although Google would be horrified to be mentioned in the same sentence).

What does Google do? The “Google Bot” is an automated process which “reads” web pages and “clicks” on links it finds to build a map of web pages and their interrelationships.

Content relevant to the topic the web page or website appears to be about is “indexed” and relationships between pieces of text and images are built up – an index of relevance.  Relevance is the holy grail in Google’s processes – since it allows them to deliver maximum return on advertising and search results which satisfy a user’s questions or searches.  Google may once have been intended to make searching the internet more efficient – it now focuses increasingly on RoI – this is referred to as a “maturing” business model.

This requires vast storage and massive processing power and has been a fertile area of rapidly developing internet technology for many years. Whereas many internet search engines began life as manually edited catalogues of websites and pages – the sheer number of sites and volume of content soon outstripped these efforts – leaving systems such as Google in a dominant position.  Complaints or alarm from civil liberties groups haven’t had much effect on this growth nor had much effect on the relative nervousness of internet users – the growth of services such as Google having helped revolutionise the usefulness of the internet.  Google has also long enjoyed the benefits of being seen as a “clever startup” run by down-to-earth students – an assumption which is today increasingly strained as Google’s “maturing” business model finds itself at odds with its simple, honest and open founding principals. (Search for “Net Neutrality” and “Google Fiber Terms of Service” for more information.)

The above-referenced page includes an interesting quote given revelations of Google’s cooperation with the NSA: “Very few people have stepped inside Google’s data centers, and for good reason: our first priority is the privacy and security of your data, and we go to great lengths to protect it, keeping our sites under close guard.”

Clearly, it isn’t so much physical access which is the problem.

Where Google maintains a vast database of internet content with relatively little public concern – the NSA is doing a technically very similar thing but with very different political ramifications and relatively massive public concern.

When Google visits and reads a web page – the web page provider is aware of the visit – and is happy because this typically translates into search engine results and positive business or personal gain. Conversely when the NSA reads the web page as it travels down the fiber optic cable – no-one (outside the NSA intelligence community and directly involved networking community) is aware of this happening, no-one can ask why, and no-one can decide if this is for the public good or not. This is listening to the telegraph.

Given the ease with which internet and telecommunications traffic of any sort can be intercepted covertly, without the knowledge of the parties to the exchange, what is the NSA then doing with it?

The first point of reference is probably a “dictionary” approach much like the Echelon Dictionary used as part of NZ’s UKUSA (1949) agreement. Rather than target an individual or group, large volumes of transmissions are processed as they are received, and likely a score is generated by collating the occurrence of “keywords” or phrases of interest. Any message containing a certain number of high-interest keywords will be flagged for more detailed analysis by a more thorough system, if not a human.  Each agency provides its own dictionary and matches against a particular dictionary are sent directly to that agency.

This approach is broadly referred to as “heuristic” – where the sheer volume of data makes thorough analysis of all messages prohibitively expensive (both in time and the expense of processing hardware) – an initial, much simpler and faster approach is employed to identify messages which deserve closer examination.  Any activity involving enough data transmission to be regularly flagged will earn a closer look.

If you wanted to easily cause an email to be immediately flagged as “highly interesting” to the NSA you would simply encrypt it (a simple process using freely available software installed by the sender and the recipient).

The second approach is likely to be mapping or matching.  If you’ve not seen The Battle for Algiers, I suggest you hire or download it and watch it.  You’ll understand how mapping was used as the basis for the French secret police and military’s brutal suppression of the Algerian Liberation movement in the forties.

Mapping is the process of connecting initially disparate data entities. Matching is the process of connecting information about an unknown entity with the actual entity itself.

The same process can be applied in hospital admissions and medical demography – in which a medical event must be connected to an actual patient when identifying data isn’t available.  For example, when a patient is admitted, unconscious with severe brain trauma. Their age is only approximately known – they were brought in by a friend who has known the patient for only a few months, or met them on holiday. The patient has had a kidney either removed or donated. They’ve previously broken their left femur and have had a fractured skull at least ten years in the past. Dental records show a root canal and significant fillings.  How could the medical staff build up a more accurate picture of who this patient is?

This practice is also the technical backbone behind marketing and demographic profiling.  When the phone goes at dinner time and an insurance company asks you to answer a few simple questions – you can be fairly certain your details have been on-sold without your knowledge. A fairly accurate picture of your income and lifestyle, spending patterns and internet preferences will be behind your selection as a cold-calling target (actually in this case probably “warm”).

Matching and profiling is a well-established and sophisticated methodology.  It is already  disturbing enough that companies are doing it – and that we are largely seen as faceless grist for the retail mill – rather than democratic citizens. The demise of the citizen and the rise of the consumer – the satisfaction of our every need through consumption without need of politics or intervention of any sort in the free market.  In fact it is more akin to the observation of wildlife and the management of a wildlife park, or laboratory rats.

Spies, private investigators, auditors, internet “phishing” and confidence scams – all exist on the basis that a plethora of legal, illegal and “grey area” sources can be consulted to build a picture of a person’s (or organisation’s) private life – in some of these examples the information is then used to do things while pretending to be the actual person – credit card fraud, passport theft, social security or social welfare fraud, bank heists, the list is long.

What can they do with this data? In the words of the American Civil Liberties Union: “Each time a resident of the United States makes a phone call, the NSA records whom she called, when the call was placed, and how long the conversation lasted. The NSA keeps track of when she called the doctor, and which doctor she called; which family members she called, and which she didn’t; which pastor she called, and for how long she spoke to him. It keeps track of whether, how often, and precisely when she called the abortion clinic, the support group for alcoholics, the psychiatrist, the ex-girlfriend, the criminal-defense lawyer, the fortune teller, the suicide hotline, the child-services agency, and the shelter for victims of domestic violence. The NSA keeps track of the same information for each of her contacts, and for each of their contacts. The data collected under the program supplies the NSA with a rich profile of every citizen as well as a comprehensive record of citizens’ associations with one another” (

In other words, this is carte-blanche with no specific purpose other than providing as much data as technically possible for any future purpose the NSA or allied agencies decides to pursue.

Solzhenitsyn’s The First Circle describes the efforts of prisoners in a special R&D prison in Moscow to develop a rudimentary voice recognition apparatus which might assist the authorities to identify, by “voice print” a dissident scientist known only by a recorded phone conversation. No doubt the NSA also brings voice recognition to bear on millions of recorded phone conversations – which, I have no doubt, it is recording along with everything else.

The growth of this, I’m sure, has been organic, piece-meal, and uneven over decades. As the political climate, technology and funding have allowed, government agencies whose professional and ideological interests rely on having as much control as possible have, surprise surprise, consistently increased their ability to obtain the information which provides the basis to such social control.

On top of that add the fervent belief amongst spooks convinced that the US is and will be the victim of US-hating terrorism, and you’ve got infrastructure and motivation to dig.

On a slightly less “Us and them” level, employees with access to NSA data have also been caught spying on their ex’s.

Having been caught with the private “meta data” of American Citizens under its fingernails, the NSA has claimed, after the fact, via the Homeland Security get out of jail card, that such measures are necessary for the defence of the American Way of Life from “threats from within”.

This rationalisation has been soundly thrashed by various constitutional and freedom organisations – who point out that dismantling a cornerstone of the constitution is a very high price to pay for the empty promise of security and safety in an increasingly besieged and suspicious nation.  A number of unconfirmed anecdotes from the US already suggest black vans and sedans full of otherwise underemployed Homeland Security agents swooping on unsuspecting proto-terrorists for actions as sinister as mentioning the word “pressure cooker” or “training camp” in emails, blog posts or even overheard conversations of a political nature.  Teenagers and frustrated activists making Facebook threats or tweets of violence against officials will certainly end up enjoying a dawn raid – to underline the point that “Uncle Sam isn’t fucking around” or as John Key said “this isn’t playtime”.

I recently had the unpleasant experience of watching The Act of Killing, a documentary in which Indonesian gangsters are invited to recreate or describe their brutal activities during the murderous military coup of 1965 and subsequent torture and murder of a million communists, activists, teachers and various other groups including Chinese migrants.  I’d previously read other details of this in Naomi Klein’s The Shock Doctrine in which she describes the covert meetings of international political and business leaders to discuss the division of Indonesia’s economy, freshly delivered to them by the country’s brutal new regime, which had been provided with detailed intelligence by MI5 and the CIA to help clear any political impediments to the smooth transition of this massive economy to a state more sensitive to western business needs.

These are the governments now telling their residents that a highly-surveilled society is necessary, that there’s nothing to worry about unless you have something to hide.  I’m reminded of the nervously waiting patient who hears muffled screams from the dentists surgery – you’d be right to be concerned – “trust me”, says the nice man in a white coat, “I’m a professional”.  Yeah, right.

It is a fact that international agreements see domestic intelligence shared with strategic partners without the agreement or oversight of the individuals and organisations whose data they share.  The US applies massive pressure through trade negotiations to leverage legislation and policies in “partner” countries more palatable to US business interests.  Airlines must now provide passenger lists for flights crossing US airspace regardless of whether they are landing at a US airport. The US has leveraged the right to veto any passenger deemed undesirable.  Details of those standards are to remain secret.

Given that the vast bulk of internet services and data reside on servers operated by US companies it is safe to say that most people in the world are now easily targeted regardless of location or nationality.  Failing that, our governments are happy to hand us over in return for the opportunity to sell more parmesan or milk powder to Americans who’ve never seen a cow.

Technically this is all only mildly interesting – what is bad about the internet (infinite perfect copies of data) for the media industry of old – is great for the internet spook industry.  They can soundlessly copy every packet that traverses the internet’s central nervous system and bide their time waiting for their mighty distributed computer arrays to find threats till the corn-fed cows come home.

Governments giving themselves more power? No surprises there either. Advances in telecommunications and processing and storage technology put all this within easy reach of the boys in the shadowy back rooms – their paymasters simply have to say “bring me the dirt on average Jo” and it is done. There is no maybe.

What can be done?

Individually, the knowledge level of your average internet user won’t make them any more or less vulnerable to this vast drag net.

It is highly likely that the internet underworld – occupied by “black hat” and “grey hat” hackers and cyber terrorists and activists – will find new motivations for delivering the fruits of their own pursuits to a wider public keen to avoid detection for whatever reason.  A political underground of this sort already exists in the form of the “TOR” network.  Unsurprisingly, this is an NSA target of interest.

A number of existing technologies – for example Pretty Good Encryption PGP (who came under fire after the WTC attacks) are already highly politically motivated and widely used by groups with real need to keep their activities and information out of the hands of their opponents.

Web browsing “anonymisers” can help hide the identity of an individual as they visit websites – deleting cookies which are used to track visits to sites and past activity, as well as hiding the user’s physical location(1) by acting as a “proxy” for their requests.

The infamous copyright infringers and champions of open internet, The Pirate Bay, hounded and harassed as they have been by US authorities and media corporations in conjunction with Swedish authorities happy to help, have developed a paid service named “iPredator” (named after the European  Intellectual Property Rights Enforcement Directive or IPRED). This paid service provides an encrypted “tunnel” or Virtual Private Network (VPN) to a proxy server somewhere in Sweden. To anyone snooping on your internet browsing, your trail disappears at the remote server – and your actual location remains invisible.

Viruses, trojans, malware, botnets… these covert methods are associated with spammers and hackers who marshal remote armies of compromised computers to launch large scale attacks on web sites targets. It is certainly not beyond the NSA to be also using these techniques – although the counter-measure industry of anti-virus vendors is highly vigilant – this would likely result in yet another embarrassing mess and media hoo-haa for the US authorities.

We’d be naive to believe that, bereft of actual terrorist plots, the NSA and it’s allied armies of zealous field agents will sit twiddling their thumbs waiting for the big opportunity. They will most definitely begin refining their reach and quietly tweaking inconvenient laws around search and seize and entrapment and laws of evidence obtained without warrant – to bring prosecutions against all manner of low level activity – criminal or otherwise.

Surveillance of political groups in the US will escalate along with the heavy jackboot of enforcement for any slight encroachment.

More broadly, the shadow of suspicion grows darker world-wide.

Which organisations are currently fighting against these measures?

  • ACLU – American Civil Liberties Union
  • EFF – Electronic Freedom Frontier
  • According to this EFF page there are 86 organisations demanding an end to the NSA’s surveillance programs

Where does this all sit in the political spectrum?

The freedom of the US free market relies on the freedom to manipulate and bully in the interests of US capital.  The US is a society which must increasingly subvert all rules by which it originally defined itself.  Once those founding ideas are well and truly forgotten, the institutions they give rise to become their own justifying principle. You can expect advances in internet technology to serve these interests before they serve the broader interests of humanity.


(1) Every home internet connection receives a unique “IP address” which can be traced to a country and internet service provider – thus allowing a high degree of targeting or matching which might cross-reference home address and approximate address identified through IP address, perhaps combined with mobile access using email accounts or eg., an Amazon account and past shipments, combined with triangulation data from cell-phone towers – to pinpoint a user’s home address.