Surviving Cyber War: A Primer on DDoS

November 22, 2009 by ADMIN

Share |

By Richard Stiennon, Chief Research Analyst, IT-Harvest

Jomini’s first maxim of the Art of War:

To throw by strategic movements the mass of an army, successively, upon the decisive points of a theater of war, and also upon the communications of the enemy as much as possible without compromising one’s own.

I thought it would be valuable to share this excerpt from my upcoming book Surviving Cyber War (pre-order on Amazon):

At the height of the dot-com boom the lingerie brand Victoria’s Secrets announced a bold initiative to broadcast video of their annual fashion show over the Internet.

As a publicity stunt the event was successful. It was also a demonstration of an inherent weakness of the Internet and the architectures employed to serve up data.

So many people attempted to view the Victoria’s Secrets models strut down the runway that the servers failed.

The crowd of viewers, using web browsers to access the video stream, created what is known as a Distributed Denial of Service (DDoS).

There are many ways to accomplish denial of service. The Victoria’s Secret case could be considered friendly fire. Most other attacks are more malicious.

Cyber activists use DDoS to shut down the servers and networks of political, religious, and corporate organizations.

Cyber criminals attempt to extort cash payments from their targets with the threat of shutting down their business.

Small businesses have been known to hire botnets, collections of compromised computers, to shut down a competitor.

Nations in conflict use crowd sourced denial of service attacks to shut off access to critical web sites in a show of force but also to silence a vocal critic in conjunction with an invasion.

Denial of service attacks are the strongest weapon available for cyber aggression. Countering them is very expensive and could theoretically be impossible.

Yet, there are measures that organizations can take to improve their defensive posture.

Let us look at the defining moments in the development of DDoS as a weapon.In 2003 Barrett Lyon was 25 years old. His work at an IT development company presented him with an interesting task.

One of their clients was being harassed by hackers in Eastern Europe. The client was in the business of gathering and disseminating sports information.

They provided the up to the minute data used by Las Vegas casinos in their book making operations where gamblers place bets on game scores and even detailed performance of individual athletes.

Having reliable Internet access was critical to them. Agents in the field would report every detail of even amateur sports events.

Every pitch, every play would be reported by an army of sports data specialists.

These results would be displayed on big boards within the casinos where gamblers could bet on any aspect of the games.

Barrett’s client first became aware that they had a problem when they received a threatening email, written convincingly in broken English, informing them that hackers had infiltrated their systems and encrypted their database of sports information, demanding that they pay thousands to obtain the key to decrypt the data.

At this point Barrett got involved. Luckily his client was following standard practice of backing up their data and had no problem at all just restoring the critical information.

But Barrett predicted that the hackers would escalate their efforts to the next level: a denial of service attack.

They would use a network of computers they controlled, a botnet, to send millions of requests to their target to effectively deny anyone else’s ability to access sports information. The disruption to their business could spell disaster.

Barrett helped his client quickly bolster their defensive posture.

The key was to have robust web servers, gateway devices that could filter attacks, and lots and lots of available bandwidth.

Within days the hackers did indeed attempt a Denial of Service attack; and, thanks to Barrett’s new architecture, the attack was thwarted.

Barrett was elevated to hero status overnight and word quickly spread to the murky world of online gambling where his services became highly sought after.

Before continuing the story of Barrett Lyon and the development of effective DDoS defenses let us take a moment for an exploration of just why DDoS is such an effective weapon.

The earliest denial of service attack was a ping flood.

Anyone with a fast computer running Unix could execute a simple command that would generate ping packets, small one-way communications used by network monitoring products to check to see if a host is still responding, to completely tie up the resources of the target computer or even completely clog its network connection.

Ping floods are simple to defend against. A single rule in a router or firewall between the attacker and the target can block all pings.

There are, however, some packets that cannot be simply blocked at the firewall. Packets associated with the normal operation of the attacked web site (or other type of server) have to be let through.

In the case of a website there is the TCP (Transmission Control Protocol) packet that initiates a connection between a browser and a web server, the SYN packet.

When a web server receives a SYN packet it begins a three way handshake and waits for a response.

An attacker simply sends millions of SYN packets which tie up the web server to the point where it cannot accept any more connections.

While effective defenses have been developed for blocking SYN floods it still means deploying special equipment in the network path. Another type of attack, the GET flood, mimics thousands of web browsers requesting pages.

This type of attack makes the web server work at maximum capacity serving up its pages and effectively prevents legitimate traffic from getting through.

Flood attacks using SYN and GET can be blocked if the source is known. Once again, just block all traffic from a specific IP address.

Today most firewalls are capable of intercepting SYN requests.

It did not take long for hackers to develop techniques for distributing their attacks among hundreds, thousands and potentially millions of attacking hosts.

These are the most effective attack techniques known, and can be very expensive to counter. The winner is usually the one with the most available bandwidth.

There are two ways to create an army of attacking hosts.

Hackers have been “recruiting” hosts by spreading malware that surreptitiously infects a computer and enlists it in a network that can be controlled from a central point and commanded to launch an attack against a target at the whim of the owner of the army of what are called “bots”.

These “bot armies” are available for hire and have been used to threaten and launch attacks against whitehouse.org and other public entities. The other way to orchestrate a distributed denial of service attack is via crowd sourcing which we discuss in another chapter.

Barrett Lyon, after demonstrating that there are effective counter measures to DDoS, began to get requests from a very specific niche industry: online gaming sites.

In 2003 there was some question about the legality of gambling on line. Enough ambiguity existed that millions of US citizens participated in poker, slot machines, craps and sports betting online.

There were dozens of companies providing such services, most of them hosted off shore in the Caribbean or in Costa Rica.

These were very lucrative businesses.

One small operation consisting of tele-operators and a closet of servers in an office in Costa Rica claimed to do $2 billion in annual revenue.

At that level of turn over it is easy to understand why they were prime targets for extortion threats that targeted their online presence.

Being down for even a day meant millions in lost gaming revenue.

The biggest day of the year for sports betting sites in the US is Super Bowl Sunday. Leading up to Super Bowl XXXVIII in 2004 the gaming sites began to receive extortion emails from Eastern Europe.

The letters said in effect: pay us $30,000 via Western Union by some date or we will take you offline.

The owners of the gaming sites began to call on Barrett Lyon to replicate the defenses he had created for the closely affiliated sports information operation.

It was then that Barrett had his big idea.

Why ask all of these small web site operators to invest in the infrastructure to counter a denial of service attack when he could make that investment once and provide a secure hosting service to all comers? But where to get financing?

From one of those online gaming sites of course! Thus Barrett embarked on a wild entrepreneurial adventure in partnership with a Costa Rican gambling operation.

The new company was named Prolexic Technologies.

Within a year Prolexic hosted 80% of the online gaming websites in the world and succeeded in putting a stop to the nascent extortion racket emanating from Eastern Europe.

His efforts included working with international law enforcement to track down, prosecute and send to prison in Siberia one of the kingpins of extortion, a young man known by his screen name, Ivan.

(The story is told in a new book Fatal System Error: The Hunt for the New Crime Lords Who are Bringing Down the Internet, written by Financial Times Journalist Joseph Menn. You can pre-order it at Amazon.)

The architecture designed by Barrett and his team of network security whizzes used three primary elements to defend against DDoS; elements that are worth studying.

First, Prolexic would proxy their customers’ web servers in their own data centers placed strategically around the world.

A proxy is just a server that mimics the original site. A

proxy, BSD, request for a web page would go to the Prolexic server which would in turn retrieve the relevant web page from the original server in Costa Rica and serve it back to the requestor.

By positioning a proxy server in between all transactions Prolexic could apply various defenses.

These included finely tuned operating systems that would not be vulnerable to common exploits found in off the shelf operating systems.

Barrett called on the expertise of one of the world’s top BSD developers based in Hawaii. BSD is an open source version of UINX.

The community of BSD developers has focused on creating as secure an operating system as possible. Prolexic customized BSD by removing all the components not needed by a web server.

Then they enhanced its ability to thwart the type of resource restrictions (memory, open ports, etc) that usually caused servers to fail when they received too many connections.

They also developed load balancing technology so that an attack of millions of requests could be served across multiple servers.

The next investment Prolexic made was in off-the-shelf network gear from the top providers of denial of service defense products.

These devices could detect attacks, send alerts, and throttle attack packets.

The cost for such devices can exceed $100K and the special security knowledge to run them is not readily available to a typical organization.

Prolexic could make that investment because they were protecting multiple paying clients.

The final component of Prolexic’s defense was bandwidth. The typical heavily trafficked web site uses 10-20 megabits per second of bandwidth.

Through its relationships with major backbone Internet providers Prolexic could use up to 18 gigabits per second of bandwidth, an unprecedented amount.

Most Internet services see the largest amount of bandwidth for outward bound traffic. YouTube, Google’s (GOOG) video hosting service, has to supply terabits of data to its consumers of streaming video.

So negotiating contracts with carriers for large amounts of incoming traffic is relatively easy and inexpensive.

The largest attacks Prolexic experienced was 11 gigs of traffic. Recent reports indicate that DDoS attacks can exceed 30 gigs of traffic.

These measures: hardened, load-balanced servers, defensive devices, and massive amounts of available bandwidth are the core of DDoS defense.

There is an Achilles heel of web infrastructure that attackers have recognized and attacked: the Domain Name Service (DNS).

The Internet is based on protocols that use source and destination packets to route traffic. When a web address, a URL, is entered into a web browser there has to be some way to translate www.threatchaos.com to 72.47.228.221, its IP address, before packets can be exchanged and a visitor can see a web page.

The DNS is a layer of servers all over the world that provide that function. It is a simple protocol but getting complicated in execution as more and more demands are being put on it.

There are multiple tiers to the DNS. The Top Level Domains (TLD) are .com, .net, .gov, .edu, and the many country codes such as .ee for Estonia, and .uk for the United Kingdom.

Each of these top level domains is supported by different organizations. When you type www.threatchaos.com in to the URL window of your browser you generate a request to the .com TLD server (hosted by Verisign in over 400 data centers around the world.)

That server replies with the IP address of the name server that is responsible for keeping track of all of the IP addresses associated with the domain Threatchaos.com.

Your browser quickly checks with that server (NS1.MEDIATEMPLE.NET at 64.207.129.18) which promptly directs you to www.threatchaos.com.

While an owner of a web site could take measures to protect their server they may not own the DNS server that provides the critical function of pointing at the web site.

In other words, an attacker could target the DNS server and effectively take down the web site.

The problem is compounded because a DNS server often provides name service for hundreds, even thousands, of separate domains.

This introduces the concept of collateral damage. You may not know what other domains are served by the DNS machine that you rely on. It could be a political site that attracts the ire of hacktivists.

An attack on the political site could take your own site down.

So one additional measure to countering DDoS is to protect your DNS server with similar defenses: hardened, load balanced servers, specialized network gear, and lots of bandwidth.

A few months after parting ways with the management of Prolexic Barrett’s reputation as the master of DDoS defense brought him into another situation.

A popular ecommerce site headquartered in the San Francisco Bay Area had grown rapidly hosting storefronts for its customers.

These web based storefronts would provide hats, mugs, and t-shirts to consumers imprinted with any image, message, or art work the owners desired.

One entrepreneur chose to sell t-shirts with images that caught the attention of a particular fanatic group who sent emails demanding that the merchandise be removed.

It was only weeks before Christmas, 2006 and the ecommerce site opted to ignore the threats.

When a massive denial of service attack was directed at them they tracked down Barrett and begged him to help.

Barrett was in the middle of launching a new venture, Bitgravity, a content delivery network, so he declined their offers. Until, that is, they told him to name a price.

Consider the importance of being available to customers during the Christmas buying season to a venture backed web company.

A prolonged outage could put them out of business.

Barrett acquiesced and collected his hefty fee. Needless to say he had them back on line within hours, using the techniques he pioneered at Prolexic.

Border Gateway Protocol: the naked under belly of the Internet.

The Internet is a marvel of self organization with many components that work seamlessly on top of each other.

Web servers, layers of protocols, social networks, and routing infrastructure, all work together to provide a communication, business, and social platform that is fueling change in society and the world of commerce.

But those underlying components were designed and deployed before today’s threats were apparent.

There is a weak link in the way the Internet is architected. It is the underlying routing protocol. This weak link is well known by aggressors but has not been exploited in an overt malicious act.

Yet.

YouTube is a popular video sharing site that Google acquired from its founders in October, 2006. Over 100 million people visit YouTube monthly to watch six billion downloaded videos.

On February 24th, 2008 an engineer at an ISP in Pakistan removed YouTube from the Internet. He did this in response to a government decree.

His intention was to follow the letter of the law and block access to YouTube from within Pakistan. There are several ways this could have been accomplished but here is the method he chose.

Packets on the Internet flow through routers. These routers maintain a list of routes based on blocks of IP addresses.

When a packet is received the router reads its intended destination, looks it up in a big table and forwards it on to the next router. Where does that router get that big lookup table?

From other routers, of course.

The protocol used to transmit those route tables is Border Gateway Protocol (BGP). An ISP will have a huge block of IP addresses assigned to them.

But, they do not want to be bothered with updating their routers every time a customer makes a change to the way they use the IP addresses assigned to them so the ISP gives their customers the ability to update the routing tables on their own.

The customer, say Google, then updates their own routers which use BGP to announce which IP addresses it controls to the rest of the routers on the Internet.

A router on the other side of the world would see that ATT (AT&T) owns the big block of addresses that Google owns but would treat Google’s announcement of the IP address range that YouTube resides at as the authoritative source because it is the more granular route announcement.

What happens if an even smaller route announcement is published via BGP? That is exactly what happened in Pakistan.

The engineer at PIENet loaded a new route into his router that said the small block of addresses that contained the IP address of www.youtube.com were controlled by him.

The result was almost instantaneous. His upstream provider in Hong Kong picked up on the new route and broadcast it to the world.

Most routers treated those routes as authoritative because they were more granular than those announced by Google.

Every attempt to watch a YouTube video was routed from anywhere in the world to a small ISP in Pakistan.

Those requests were so numerous that it flooded the links to Pakistan to such an extent that Pakistan was effectively knocked off the Internet as well.

Barrett Lyon was serendipitously at the center of this event as well. After founding Prolexic and building it into a success Barrett launched a new venture.

BitGravity is a content delivery network (CDN) that Barrett created from scratch. As the Internet has begun to carry more and more video the strain put on delivery mechanisms has grown.

Live streaming of a single event like Victorias’ Secrets’ runway show was a first. Now conferences, video chat, online “Internet TV” shows, and services like YouTube and Hulu put even greater demands on infrastructure.

Comedy Central, Revision3, CollegeHumor, and uncountable pornography sites rely on CDN’s to deliver their content.

These services are willing to pay for reliable delivery of their programming. Barrett built BitGravity to serve that market.

BitGravity is backed in part by the largest backbone bandwidth provider in the world, TaTa Communications, based in India.

This gives Barrett access to that bandwidth. His activity also puts him at the hub of a community of people who manage Internet traffic.

So it is not surprising that when YouTube was routed to Pakistan Barrett was on a private chat channel with those operators.

A few quick checks of the routing tables and they had figured out exactly what had happened.

Barrett found himself calling engineers at PCCW in Hong Kong and instructing them in how to filter the route announcements coming from Pakistan.

Thanks to Barrett Lyon the YouTube outage was repaired by the end of that fateful Sunday.

The Pakistan-YouTube incident of 2008 demonstrated just how effective spurious BGP route announcements could be as a cyber weapon.

While not malicious the effects were completely effective in denying access to YouTube.

Preventing malicious abuse of BGP is a concern of the Internet community as a whole and is being addressed but in the meantime it is critical that an organization’s upstream Internet provider follow best practices by installing strict controls over which routers and networks they will accept BGP route announcements from.

If you control your own block of IP addresses and do your own route announcements one defensive measure would be to be ready to announce very small blocks of addresses if a YouTube-like attack occurs.

Another defense could be to move your site to a completely different block of IP addresses that the attacker has not tampered with.

Denial of service is often the intent of Internet attacks. New methods of accomplishing a denial of service are being invented every day.

DNS attacks, crowd sourcing, spurious BGP route announcements, and botnets are today’s weapons of choice.

Future methodologies will become apparent as attackers push the envelope. Luckily, Barrett Lyon and legions of engineers are watching over the Internet.

Relying on ad hoc responses may actually be the appropriate way to respond to the current cyber threat environment.

Just as markets do a better job of regulating good and bad business practices in rapidly evolving economies, the self interested protection of the security community may be the best response to the scourge of Internet pestilence.

Viruses, worms, spam, spyware, and botnets have all changed the Internet and the way organizations use it for profit.

* * *

Stay Informed With ISR News Alerts:

* * *

Announcing the birth of Cyber Defense Weekly, a newsletter created to give participants in this new category a comprehensive summary of the week’s news, product announcements, and escalations in cyber threats.

Simply provide your email address here to become a subscriber.

Comments and input are welcome as always on this critical new category.

Richard Stiennon, security expert and industry analyst, is known for shaking up the industry and providing actionable guidance to vendors and end users. He recently re-launched the security blog ThreatChaos.com and is the founder of IT-Harvest, an independent analyst firm that researches the 1,200 IT security vendors. He was Chief Marketing Officer for Fortinet, Inc. the leading UTM vendor. Prior to that he was VP Threat Research at Webroot Software. Before Webroot, Mr. Stiennon was VP Research at Gartner Inc. where he covered security topics including firewalls, intrusion detection, intrusion prevention, security consulting, and managed security services for the Security and Privacy group. He is a holder of Gartner’s Thought Leadership award and was named “One of the 50 most powerful people in Networking” by NetworkWorld Magazine.

The Publisher gives permission to link, post, distribute, or reference this article for any lawful purpose, provided attribution is made to the author and to Information-Security-Resources.com

These icons link to social bookmarking sites where readers can share and discover new web pages.