Craig Labovitz's Blog

Occasional posts on Internet security, traffic engineering, changing economics of the telecommunication industry and interesting research. Also see the Arbor corporate blog ( and my home page ( for more of my research and writings.

Sunday, Feb 27, 2011 @ 08:17pm     [link]     

Libya Firewall Begins to Crumble?

In what may be an indicator of the rapidly evolving political situation within Libya, Internet traffic in and out of the country climbed over the weekend. Previously, Internet traffic volumes had been operating at 60-80% of normal as the Libyan government reportedly blocked social media and popular video sites after the start of the popular uprising on February 18th.

While media reports and Twitter updates suggest social media sites (i.e. Facebook, Twitter, YouTube, etc) are still blocked in Tripoli, other cities may now have broader Internet connectivity. Twitter updates (see below) appear to confirm the return of a more open Internet to eastern Libya.

Most Libya Internet traffic flows through the state telecom (LTT / AS21003) and then transits out of the country via three main European / Asian providers. The AS21003 network likely includes multiple datacenters and routers in several different Libyan major cities including Benghazi. More information on the network (including the hosting business) is available on the LTT web site at

The below graph shows traffic based on ATLAS GeoIP statistics across sixty or more ISPs around the world. A graph of traffic through AS21003 exhibits similar trends.

Other data sets, including Google's transparency report show a similar increase.

On related note, the LA Times ran an interesting piece this weekend on Libyans smuggling cell phone video to Egypt to upload reports about the ongoing uprising.

Tuesday, Feb 22, 2011 @ 09:26pm     [link]     

Applications Impacted by Libya Internet Outages

A quick update as of Tuesday Feb 22 -- Libya Internet outages continue with a consistent 60-80% drop in traffic volumes.

Looking at TCP and UDP ports, the outage impacts all major applications. Though the percentage drop from "normal" levels is proportionally higher for web and AIM / messaging.

Monday, Feb 21, 2011 @ 05:22pm     [link]     

Libya Internet Traffic Update Feb 20

Sunday, Feb 20, 2011 @ 05:11pm     [link]     

Middle East Internet Scorecard

[A PDF version of this analysis is also available.]

The success of the Tunisian and Egyptian protest movements inspired demonstrations throughout the Middle East last week, including large-scale social media coordinated protests in Libya, Iran, Bahrain, Algeria, Jordan and Yemen. In several of countries, governments responded to the calls for reform with arrests and violent suppression of public demonstrations. Increasingly, several Middle Eastern governments also may be disrupting phone and Internet communication to contain the spread of unrest.

These new Internet filtering efforts come a week after Egypt returned to the Internet following an abortive effort to block protests demanding the then president, Hosni Mubarak, resign. While other countries, including Iran and Myanmar, disrupted telecommunication following social unrest in the past, the Egyptian outage represents a new Internet milestone -- the first highly connected, telecommunication dependent society to intentionally disconnect from the Internet [1,2].

This analysis uses real-time data from the 110 Internet providers around the world to identify possible ongoing Internet traffic manipulation in Middle East countries with active protest movements. More details on our data collection infrastructure and methodology are available in our recent academic paper [3].

Overall, our data shows pronounced changes in Internet traffic levels in two Middle East countries last week: Bahrain and Libya. While network failures and other exogenous events may play a role in decreased traffic volumes, we observe the changes in Bahrain and Libya are temporally coincident with the onset of recent protests. Several Bahrain telecommunication companies blamed the slowdown on "overloaded circuits" and extremely high usage [4].

We note that many countries in the region maintain some level of permanent Internet limits, including blocks on dissident web sites, social media and adult content [5]. The traffic volumes graphed on the following page represent possible traffic manipulation beyond normal filtering practices.

In the below chart, we show the "normal" traffic in and out of each country averaged over the proceeding three weeks in green. The dotted red line in each graph shows the traffic over the last seven days. Orange shaded areas indicated periods of statistically abnormal traffic either last week or the week of February 14. Abnormal traffic volumes may network failures or periods of intentional traffic manipulation. Due to the near complete block of all Internet traffic (January 27 -- February 2), the Egyptian graph shows orange for most of last week as traffic levels climbed to normal. Yemen Internet traffic also exhibited brief, though unusual dips, during the prior week (February 7-11) and also includes an orange period.

While the Internet has proven a powerful tool for rallying social and political change, so too have governments recognized their regulatory and technical capability to disrupt communications. The next few weeks will likely prove a major contest between the continued evolution of the Internet as a vehicle for political change and authoritarian governments? continued assertion of control.

End Notes

[1] Craig Labovitz, "Egypt Loses the Internet". Arbor Networks blog post. Available at January 28, 2011.

[2] James Cowie, "Egypt Leaves the Internet". Renesys blog post. Available at January 27, 2011.

[3] Craig Labovitz, Scott Iekel-Johnson, Danny McPherson, Jon Oberheide, and Farnam Jahanian, "Internet Inter-Domain Traffic". Proceedings of ACM SIGCOMM 2010, New Delhi. August, 2010.

[4] Christopher Rhoads, "Technology Poses Big Test for Regimes". Wall Street Journal. February 12, 2011.

[5] OpenNet Initiative. Web site at


Sunday, Feb 20, 2011 @ 12:01pm     [link]     

Libya Internet Outages

Continuing to track Internet traffic in and out of Libya. Based on ATLAS data from 50 Internet providers around the world, Libyan Internet service was abruptly disrupted last night (February 18) at 7:15pm EST. Some traffic returned early Saturday morning (EST) at 30% of normal levels. All traffic dropped off completely again at 5pm EST tonight (February 19).

Thursday, Jan 27, 2011 @ 05:11am     [link]     

Egypt Loses the Internet

Updated January 31: Added graph and discussion of remaining active paths

Following a week of growing protests and periodic telecommunication disruption, Egypt suddenly lost all Internet connectivity at approximately 5:20pm EST Thursday.

The below graph shows traffic to and from Egypt based on ATLAS data from 80 providers around the world.

Between 3 and 5pm EST, Egyptian traffic rapidly climbed to several Gigabits. At 5:20pm, the all Egyptian transit providers abruptly withdrew the major of Egypt?s several thousand BGP routes and traffic dropped to a handful of megabits per second.

At present, the cause of the outage is unknown though many press reports have drawn parallels to the Internet outages following Iranian political protests during the summer of 2009. Further, the simultaneous failure of Internet across multiple different Egyptian ISPs and diverse physical paths (i.e. satellite, fiber optic, etc) suggests this was a coordinated event rather than a natural failure. Typically, telecommunication companies operate under strict regulatory control in many countries around the world.

As of Monday (January 31), Egypt remains disconnected from the Internet. A week view of traffic in and out of Egypt below.

Normally, Egypt enjoys one of the largest and most robust Internet infrastructures in Africa with a dozen major providers, more than 30% consumer penetration, and multiple high-speed paths to Europe and the rest of the world. Egypt also serves as a major terrestrial fiber optic crossing point for traffic to other countries in Africa and the Middle East. Traffic to other countries using these links through Egypt has not been impacted.

While the Egyptian telecommunication market has enjoyed significant liberalization in the last decade, the Egyptian government Telecommunications Regulatory Authority (TRA) continues to assert a strong level of regulatory control over the telecom licensees. See for more information (although the TRA web site is currently unreachable outside Egypt).

Monday, Jan 03, 2011 @ 04:10am     [link]     

China Hijacks 15 Percent of the Internet?

On Wednesday, the US China Economic and Security Review Commission released a wide-ranging report on China trade, capital markets, human rights, WTO compliance, and other topics. If you have time to spare, here is a link to the 324 page report.

Tucked away in the hundreds of pages of China analysis is a section on the Chinese Internet, including the well-documented April 8, 2010 BGP hijack of several thousand routes (starting on page 244).

To review, shortly around 4am GMT on April 8th a Chinese Internet provider announced 40,000 routes belonging to other ISPs / enterprises around the world (though many were for China based companies). During a subsequent roughly 15 minute window, a small percentage of Internet providers around the world redirected traffic for a small percentage of these routes to Chinese address space. RIPE provides a link to a list of some of these prefixes (as well as indicating the impact on European carriers was minimal) and Andree Toonk and his colleagues at BGPmon have a nice synopsis at the BGPMon blog.

Following shortly on the heels of the China hijack of DNS addresses in March, the April BGP incident generated a significant amount of discussion in the Internet engineering community.


Any corruption of DNS or global routing data (whatever the motive) is a cause of significant concern and reiterates the need for routing and DNS security. But in an industry crowded with security marketing and hype, it is important we limit the hyperbole and keep the discussion focused around the legitimate long-term infrastructure security threats and technical realities.

So, it was with a bit of a surprise that I watched an alarmed Wolf Blitzer report on prime time CNN about the China hijack of "15% of the Internet" last night. A bit less diplomatic, a discussion thread on the North American Network Operator Group (NANOG) mailing list called media reports an exaggeration or "complete FUD". Also on the NANOG mailing list, Bob Poortinga writes "This article ... is full of false data. I assert that much less than 15%, probably on the order of 1% to 2% (much less in the US) was actually diverted."

If you read the USCESRC report, the committee only claims China hijacked "massive volumes" of Internet traffic but never get as specific as an exact percentage. The relevant excerpt from the report below:

The USCESRC cites the BGPMon blog as the source of data on "massive traffic volumes". But curiously, the BGPMon blog makes no reference to traffic -- only the number of routes.

You have to go to a National Defense interview with Dmitri Alperovitch, vice president of threat research at McAfee, to first come up with the 15% number. Several hundred media outlets, including CNN, the Wall Street Journal, Time Magazine and many more picked up this interview and eagerly reported on China's hijack of "massive Internet traffic volumes of 15% or more".

Now certainly, diverting 15% of the Internet even for just 15 minutes would be a major event. But as earlier analysis by Internet researchers suggested, this hijack had limited impact on the Internet routing infrastructure -- most of the Internet ignored the hijack for various technical reasons.

And indeed, ATLAS data from 80 carriers around the world graphed below shows little statistically significant increase due to the hijack on April 8, 2010. I highlight April 8th in yellow and each bar shows the maximum five minute traffic volume observed each day in April going to the Chinese provider at the center of the route hijack.

china hijack

While traffic may have exhibited a modest increase to the Chinese Internet provider (AS23724), I'd estimate diverted traffic never topped a handful of Gbps. And in an Internet quickly approaching 80-100 Tbps, 1-3 Gbps of traffic is far from 15% (it is much closer to 0.015%).

In fairness, I should note that I don't know how Mr. Alperovitch obtained his 15% number (the article does not say) and a hijack of 40k routes out of a default-free table of ~340K is not far from fifteen percent. But of course, routes are different from traffic. I also add that both China denied the hijack and some Internet researchers suspect the incident was likely accidental.

The global BGP Internet routing system is incredibly insecure. Fifteen years ago, I wrote a PhD thesis (link available here) using experiments in part capitalizing on the lack of routing security. My research injected hundreds of thousands fake routes (harmless!) into the Internet and redirected test traffic over the course of two years. A decade or more later, none of the many BGP security proposals have seen significant adoption due to a lack of market incentives and non-legitimate routes still regularly get announced and propagated by accident or otherwise. Overall, the Internet routing system still relies primarily on trust (or "routing by rumor" if you are more cynical).

We need to fix Internet infrastructure security, but we also need to be precise in our analysis of the problems.

UPDATE: Additional discussion and statistics on the incident are now available in a follow-up blog at

- Craig


Sunday, Oct 03, 2010 @ 06:11am     [link]     

Internet Goes to War

If you weren't paying attention last week, the Internet has gone to war.

ABC News proclaimed "Welcome to Infowar, Version 1.0". Fox warned of the">"growing data war". And the Guardian provided minute by minute coverage on the opening salvos of this first "Internet-wide Cyber War".

Of course, all of the above headlines refer to the rash of DDoS attacks both against the Wikileaks web site and the retaliatory strikes against hosting and commercial institutions that severed ties with the organization.

So are we now in a permanent state of cyber-war? As the San Francisco Chronicle asks, do sixteen year old hackers now control the fate of humanity from their laptops?

Well, this blog uses detailed statistics on the last year of DDoS attacks across the Internet to provide some perspective. I'll compare the Wikileaks and retaliatory DDoS attacks to historical baselines of attack activity and discuss broader DDoS trends.

In general, getting accurate data about Internet attacks can be a challenge. Namely, a) companies avoid publicly discussing most attacks and b) the attacks can be difficult to measure or at least consistently compare. For example, engineering mailing list discussion of ISP security and DDoS attack trends generate a bewildering variety of responses. In one instance, two engineers at the same ISP debated the largest observed botnet attacking their company -- one estimated the size at a few thousand hosts while the other at millions. Later when pressed on the source of their data, both of these engineers readily admitted they were really just guessing (they did not have any infrastructure / tools to actually measure the number of attacking botnet hosts).

In an effort to better quantify DDoS attack trends, two years ago Arbor added support for the export of detailed measurements of confirmed DDoS attacks to our commercial products and ATLAS anonymous statistics (deployed in roughly 75% of all Internet carriers). This blog post provides a first look at quantitative measurements of over 5,000 confirmed (via operator classification or mitigation status) attacks over the last year across 37 large carriers and content providers around the world. We believe this is the largest data set of validated DDoS events ever collected. I presented an earlier version of this blog post at this Fall's NANOG (link to the presentation here) and we're currently working on an academic paper version.

Before diving into the statistics, a bit of background -- our data includes both survey results and two overlapping measurement data sets: alerts and mitigations. At a high level, alert data include the magnitude and fingerprint of a DDoS (i.e. IP header fields and router / interface topological origins of the attack). Mitigation statistics include finer-grain detail on the payload of the attack, including spoofed source IPs, number of valid (i.e. not spoofed) source IPs, connection attempts, bps and pps rates per attacking IP, etc.

In general, we evaluate DDoS attacks using two metrics: the scale and the sophistication of the attack. At the high end in 2010, we observed a number of DDoS attacks in the 50+ Gbps range. These large flooding attacks often exceed the inbound aggregate bandwidth capacity of data centers and carrier backbone links (often OC192 / 10 Gbps). Mitigation of these high end attacks can be a challenge -- carriers generally need specialized, high speed mitigation infrastructure and sometimes the cooperation of other providers to block the attack traffic. The below graph plots the growth DDoS flooding attacks over the last decade (hard to imagine that 400 Mbps was an impressive attack back in 2002).

ddos trends

On the other end of DDoS spectrum, we encounter attacks focused not on denying bandwidth, but the back-end computation, database, and distributed storage resources of large web services. For example, service or application level attacks may focus on a series of web or API calls that force an expensive database transaction or calls to slow storage servers. The attackers then use botnets to inundate the web service with thousands of clients issuing a steady stream of these particularly expensive web / API calls. Other application attacks attempt to overwhelm SIP, HTTP or TCP state (e.g. Slowloris). In many of the more sophisticated application DDoS, attackers perform reconnaissance of the web service for weeks or months before the attack (identifying weak links in the infrastructure). Unlike massive DDoS traffic floods, application attacks can be far more subtle and may only register as increased load on servers or a precipitous drop in five minute real-time sales revenue charts. Also like 10+ Gbps flooding attacks, sophisticated application attacks may required specialized, high speed infrastructure to detect and mitigate the DDoS.

So if we're in a Cyber-War, then very large (50+ Gbps) traffic floods and sophisticated application attacks are the front-lines. Which brings us back to the question of Wikileaks and the retaliatory hactivist attacks. Were these attacks massive high-end flooding DDoS or very sophisticated application level attacks?


Despite the thousands of tweets, press articles and endless hype, most of the attacks over the last week were both relatively small and unsophisticated. In short, other than than intense media scrutiny, the attacks were unremarkable. I note that our ATLAS based observations agree with data from the operators directly involved in mitigating the attacks.

For example, below is a graph of DDoS activity against multiple Wikileaks hosting sites on third day (December 1) following the initial release of "Cablegate" documents. The DDoS traffic (in red) never grew beyond 3-4 Gbps. Today, mitigating attacks of this scale is fairly routine for tier1/2 ISPs and large content / hosting providers (more of an annoyance than an imminent critical infrastructure threat -- or "easy peasy" to block as one Internet engineer explained). Also see earlier blog posts (link available here) for more analysis of the Wikileaks attacks.

day 3

The retaliatory hactivist attacks took a slightly different approach with mostly low-level application layer attacks against a range of companies perceived as anti-Wikileaks, including banks, hosting and credit card companies. The loosely organized Anonymous group called hundreds of volunteer activists to arms with messages like:


[I replaced the target and Russian download site with xx's].

Based on ATLAS data, the majority (70%) of the hactivist application DDoS came from a Mac / PC down-loadable "Low Orbit Ion Canon" (LOIC) program and a web based Javascript version (JS--LOIC). Both LOIC variants sent dozens of web requests per second to the victim web sites. The online web version consists of a simple 100 line Javascript for-loop generating web requests and very few options (though you can append text with an appropriately revolutionary message). The PC version supports slightly more complex options, including randomization of URLs and remote control by IRC botnets ("the hive").

Approximately 20% of retaliatory attack DDoS HTTP requests in one attack last week came from a new variant of LOIC named, predictably, LOIC-2. The new LOIC version (a "total rewrite of LOIC") supports additional "hive" remote control command channels including RSS, Twitter, and Facebook (LOIC only supported irc). More significantly, LOIC-2 supports two new "slow" class of attack methods (i.e., DDoS strategies where the client deliberately elongates HTTP transaction times to burden the victim server).

In addition to LOIC, ATLAS observed Slowloris like TCP attacks and several other tools / scripts generating web or TCP DDoS traffic. A smaller component of the hactivist campaign included DDoS flooding using ICMP Smurf and LOIC operating in UDP flood mode (sending traffic to UDP port 80).

More recently, Anonymous supporters released two more sophisticated HTTP flooding tools: High Orbit Ion Cannon (HOIC) and Geosynchronous Orbit Ion Cannon (GOIC). The new tools support multi-threaded HTTP flooding, simultaneous attacks against up to 265 web sites, plug-ins and an "easy to use interface". However, HOIC and GOIC did not appear to play a significant role in the DDoS attacks last week.

While the last round of attacks lead to brief outages, most of the carriers and hosting providers were able to quickly filter the attack traffic. In addition, these attacks mostly targeted web pages or lightly read blogs -- not the far more critical back-end infrastructure servicing commercial transactions. By the end of the week, Anonymous followers had mostly abandoned their attack plans as ineffective.

Overall, both the attack traffic and the hundreds of volunteers running the software on their PCs were not terribly sophisticated. Most volunteers clearly did not realize the tools do not anonymize their PC source IP address nor that word processors store incriminating meta-data in revolutionary manifestos. In short, not exactly the work of evil criminal masterminds.

So ultimately, I'd suggest the last week of DDoS attacks surrounding Wikileaks supporters and opponents falls far short of a "cyberwar". While it makes a far less sexy headline, cyber-vandalism may be a more apt description. In a similar vein, a Foreign Policy Op-Ed called hactivist DDoS the digital equivalent of a sit-in by youth around the world.

All of the above is not to say DDoS is not a serious problem. The number and firepower of botnets grows dramatically each year as well as the sophistication of application attack toolsets. HOIC and succeeding generations of volunteer botnet controlled PCs may evolve to pose a significant Internet-wide threat. However, traditionally the DDoS threat has come more from increasingly professional criminal hackers than volunteer activists.

With discussion of cyberwar out of the way, I'll compare Wikileaks and related attacks to some of the broader trends we are observing in ATLAS DDoS statistics. The chart below shows the distribution DDoS attack vectors in the 5,000 validated attacks in the ATLAS dataset. Note that this dataset represents a subset of all attacks as not all providers have enabled anonymous export of data and many providers are running earlier versions of the product (i.e., lacking anonymous DDoS statistics export support). See the NANOG presentation (link available here) for more details on the methodology.

As discussed earlier, brute-force flooding continues to dominate most DDoS attacks (60%). Generally, these attacks (including the initial strike against the Wikileaks web site) resemble the early days of DDoS attacks circa 2000 except more distributed (better botnets) and greater use of amplification. As in 2000, most flooding DDoS attempt to overwhelm upstream bandwidth, firewall / load balancer state, or resources on web / application farms.

attack overview

Though traditional DDoS flooding attacks remain popular, most of the recent DDoS activity has included some level of application or TCP layer attack components. Involved in 27% of the confirmed attacks over the last year, application layer attacks are also the fastest growing DDoS attack vector. Open source tools like LOIC / HOIC and large library of more advanced commercial criminal software targets firewall, load balancer and end-system web, database, and TCP state. A tutorial by security consulting company Securitech provides a nice overview and examples of these layer3+ attacks.

Finally, "Other" in the above chart is a bit of a grab-bag, including operator defined policy around allowed traffic levels for things like ASN, GeoIP (countries), ATLAS filters, large lists of ACLs and payload (e.g. DNS, URL) regular expressions. Although designed as a line-speed DDoS mitigation appliance, some providers use the Arbor TMS to effect policies similar to next-generation firewall or carrier-grade IPS. Our analysis generally cannot distinguish between DDoS mitigations and policies enacted for other carrier security strategies.

As discussed earlier, the Wikileaks flooding DDoS components fell into the small or mid range of our yearly survey data (links available here). The chart below shows statistics on the flooding DDoS bandwidth, packets per second and duration for the 5,000 validated attacks. The average DDoS comes in at 300 Mbps and 200 Kpps lasting several hours. Though given the heavy tailed nature of DDoS attack distribution, the mean is skewed by a relatively small number of extremely large DDoS (including one 22 Gbps and 9 Mpps IP fragment attack against a single web farm lasting four days). The median of 30Kpps suggests that the majority of DDoS by number of incidences remain fairly low bandwidth (and likely reflect provider offering DDoS mitigation services for hundreds of small customers).

attack sizes

The next table focuses on the number of unique sources involved in DDoS flooding attacks. Despite the availability of massive botnets, most confirmed attacks in our study involve relatively few, well-connected IPs -- the average is 80 sources generating an average of 162 Mbps and 48 Kpps each. Even the 95th percentile of attacks involves only 300 sources. Why so few botnet hosts in these attacks? I suspect the answer is a) a hundred well-connected hosts is more than sufficient to overwhelm many mid-size web farms (you just don't need more than this) and b) botnets are an increasingly valuable resource to be used judiciously as discussed in this Security Week article.

Though more than 100,000 users downloaded the LOIC software last week, the actual peak number of simultaneous Wikileaks retaliatory attackers was significantly lower. ATLAS data suggests the number of attackers was in the hundreds (i.e., instead of thousands or tens of thousands). In other words, the number of source IPs observed in the Wikileaks retaliation attacks fell into the mid or higher end of the 5,000 validated DDoS last year.

number of flooding source IPs

Of course, just tracking statistics per IP does not tell us if these are real or spoofed source addresses. And indeed, increasingly unrealistic data as we approach the max (4 Gbps per source IP!) in the above chart suggests some degree of either source spoofing (e.g. poorly written attack tools always using the same source address) or large number of hosts behind NAT / mega-proxies. About 10% of attacks fall into this category of unrealistic source IP statistics.

The next table focuses on TCP layer DDoS attack statistics. The first column shows the number of TCP connection attempts per second in each attack and the second column provides the median, mean, 95th percentile and max number of connections that actually pass a range of validation algorithms (i.e. "prove" that the TCP connection is from a real host). Ranging from several hundred thousand to millions of connection attempts per second, the data in above chart suggests most of these Syn floods either use attack tools with incomplete stacks or spoof the source IP address (which is pretty much what you would expect). In the specific case of the Wikileaks retaliatory attacks, we believe most of the traffic did not spoof and used the actual sources IPs.

tcp layer attack statistics

Finally, the last table below provides statistics on two types of application-layer attacks: HTTP and SIP. In general, HTTP attacks involve highly targeted floods of requests for complex / computational expensive web or service queries. Examples of well-known attacks include Slowloris and Slow Post. From the data, web attacks involve relatively low bandwidth (95h percentile is 10Mbps). Further, web attacks involve large number of hosts (414 in the 95th percentile) than zombie and other types of flooding attacks. Both SIP and HTTP layer attacks tend to be long-lived -- targeting infrastructure for days and sometimes weeks. Unlike HTTP, SIP attacks tend to be larger (average 200 Mbps and 77Kpps) and more resemble flooding attacks as hackers attempt to overwhelm SBCs or soft-gateways.

application attack statistics

So what conclusions can we draw from all of the above data?

Like the initial Wikileaks attacks, most DDoS continue to rely on brute force flooding to exhaust link capacity or overwhelm load balancer, firewall and web server state. Further, despite the conventional wisdom in the security community that spoofing is no longer common (because botnets are so prevalent), analysis of 5,000 validated DDoS attacks suggests a significant percentage of attackers still take advantage of a lack of BCP-38 and generate large volumes of spoofed DDoS traffic.

While the Wikileaks and retaliatory attacks may not represent the start of "cyberwar", governments clearly view cyberspace as the battlefield of the future. The trend towards militarization of the Internet and DDoS used as means of protest, censorship, and political attack is cause for concern (the world was a simpler place when DDoS was mainly driven by crime, irc spats and hacker bragging rights). Overall, DDoS fueled by the growth of professional adversaries, massive botnets and increasingly sophisticated attack tools poses a real danger to the network and our increasing dependence on the Internet.

- Craig

Credit to Joe Eggleston, Jose Nazario, Jeff Edwards, Roland Dobbins and Mike Hollyman for their contributions to this analysis.

Saturday, Oct 02, 2010 @ 06:12pm     [link]     

Google Sets Traffic Record

In their earnings call last week, Google announced a record 2010 third-quarter revenue of $7.29 billion (up 23% from last year). The market rejoiced and Google shares shot past $615 giving the company a market cap of more than $195 billion.

This month, Google broke an equally impressive Internet traffic record -- gaining more than 1% of all Internet traffic share since January. If Google were an ISP, as of this month it would rank as the second largest carrier on the planet.

Only one global tier1 provider still carries more traffic than Google (and this ISP also provides a large portion of Google's transit).

In the graph below, I show a weighted average percentage of Internet traffic contributed by the search / mobile OS / video / cloud giant. As in earlier posts, the Google data comes from 110+ ISPs around the world participating in ATLAS. The multiple shaded colors represent different Google ASN and reflect ongoing global traffic engineering strategies.


Google now represents an average 6.4% of all Internet traffic around the world. This number grows even larger (to as much as 8-12%) if I include estimates of traffic offloaded by the increasingly common Google Global Cache (GGC) deployments and error in our data due to the extremely high degree of Google edge peering with consumer networks. Keep in mind that these numbers represent increased market share -- Google is growing considerably faster than overall Internet volumes which are already increasing 40-45% each year. More data on general Internet growth trends is available in some of our earlier papers and blog posts.

While its not news that Google is Big, what is amazing is how much bigger Google continues to get.

A quick analysis of the data also shows Google now has direct peering (i.e. not transit) with more than 70% of all providers around the world (an increase of 5-10% from last year). In fact, the only remaining major group of ISPs without direct Google peering are several of the tier1s and national PTTs -- many of whom will not settlement-free peer with Google due to regulatory prohibitions or commercial strategy.

While the business press may debate Google's future (i.e. can it expand beyond search and continue its earnings growth?), for now Google's traffic growth continues apace with massive corresponding impact on the network topology, peering arrangements and the overall Internet infrastructure.

  - Craig    

Friday, Apr 02, 2010 @ 07:12am     [link]     

Battle of the Hyper-Giants

My blog post last month on the rapid growth of Google generated a bit of discussion around Google and its competitors. In particular, this Wired article ("Google?s Traffic Is Giant") suggests Google's infrastructure should "frighten the world?s current ISPs" and content distributors (i.e. CDNs like Akamai and Limelight). Going even further, a panicked "EatMoreBeef" Wired reader warned "I?m selling my Akamai stock!"

As Google grows towards 10% of all Internet traffic, will the multi-media search giant squash all competitors under its chrome-plated multi-terabit steamroller?

Or will the global zeitgeist tire of kitten videos and plow YouTube under the treads of hundreds of millions of virtual tractors tending to their farms and social networks on Facebook?

Or will Microsoft's desktop OS juggernaut link with a growing Azure Cloud and pink phone to form an impenetrable competitive enterprise and consumer road block?

I have no idea.

Given my previous market predictions ("Google will never go above $200!"), I'm not going to try and predict the winners / losers in today's Hyper Giant fight.

But I do know that that the future of the Internet is being decided today by billions of dollars of investments in data centers, backbone infrastructure and alliances / contracts with other content owners and last-mile providers. And increasingly, Hyper Giant strategies are coalescing around similar infrastructure investments as the giants compete on content, capacity (bandwidth, storage, compute), cost and performance. In other words, Google is not unique in their infrastructure ambitions.

In the next couple of blog posts, I'll look at several of the "Hyper Giants" to help put all of this in perspective.

The below graphic shows market data and Internet routing and traffic statistics for Google (Alexa #1), Facebook (Alexa #2) and Microsoft (Alexa #5). [To save you from having to go to the Alexa web site, #3 is Yahoo and #4 is Google's YouTube]. I have also included Akamai -- one of the largest Internet infrastructure providers most consumers have never heard of -- in the list below. fight

Famously started in Harvard dorm room in 2005, Facebook has grown well beyond its Ivy League roots to become the daily required Internet stop for hundreds of millions of consumers. Facebook content has also evolved beyond short text updates bragging about last night's party to include thousands of applications, games and petabytes of pictures and video.

And a lot of Internet traffic.

The below graph shows Facebook as a weighted average percentage of all Internet inter-domain traffic. As in previous blog posts, I'm using data from 110 Internet providers around the world anonymously sharing coarse grain traffic engineering statistics. I have also included MySpace traffic as a point of reference.


Between March of 2007 and April 2010, Facebook grew from zero to more than .5% of all Internet traffic globally -- placing the dominant social media site well in the top 50 Internet Hyper Giants. And this number does not include the significant volumes of Facebook CDN traffic.

Given the expense and time required for new Internet scale datacenter construction ($500 million or more), most Internet content companies begin life using colo (e.g. Twitter) or leased wholesale space (e.g. Facebook). Many nascent Internet companies (Facebook included) also start out leveraging third-party distribution infrastructure like LimeLight or Akamai (currently the dominant CDN used by Facebook).

But as computing, storage and distribution demands increase, small differences in capital / operational expense become large competitive differentiators at Internet scale (this 2008 NANOG presentation provides a nice overview of datacenter / colo pricing pressures). As Facebook crosses the 30,000 server mark, the company's strategy has increasingly shifted to focus on its own proprietary infrastructure. Earlier this year, Facebook began construction of its first datacenter to "deliver a faster, more reliable experience worldwide." Facebook is rumored to have plans on the drawing books for another four mega-scale Internet data centers.

Also like Google, Facebook has aggressively pursued direct peering with last-mile / consumer networks. As of March 2010, Facebook uses direct peering for more than 25% of its traffic (up from 5% in 2009). Like other content heavy Hyper Giants, Facebook also offers a liberal peering policy with a presence at more than 15 public exchange points.

While Facebook may not yet have the same infrastructure footprint as Google or other larger Hyper Giants, the game is clearly afoot. Leveraging wholesale datacenters, third-party CDNs and a raft of partnerships and alliances, Facebook may yet outgrow competitors with an all encompassing social media cum application platform. As of last month, Facebook reportedly surpassed Google as the most visited site on the Internet.

Friday, Aug 08, 2008 @ 05:10am     [link]     

The End is near, but is IPv6?

As of this blog posting, exactly 900 days remain until the end of the Internet, or at least the exhaustion of IPv4 registry allocations. And you don't have to take my word for it, even the normally staid London Times and Fox News proclaimed, "Internet meltdown... The world is heading for a digital doomsday".

Heady stuff.

Of course, IPv6 (or the new IPv4) was supposed to take care of all of this -- billions of billions of new IP addresses, hardened security built in from the start, and an elegant new architecture to replace all of IPv4's hacks.

So what happened to IPv6?

Well, it has been a strange, long year...

The year began with fears over the "end of the Internet" (due to lack of IPv6 adoption) and ends this month with renewed IPv6 enthusiasm centered around the Olympics and a successful US government IPv6 mandate. In between these two extremes of IPv6 despair and enthusiasm, IPv6 generated a surge of news coverage (see graph below). At its peak this past June, print media around the world published nearly 3,000 articles a month on IPv6 (almost twice as much as the comparatively uninteresting IPv4).

Much of the recent coverage has focused on the summer Olympics this week. Chinese organizers have planned the summer Olympics game as a showcase for IPv6 technology. From a recent article, "... IPv6 will herald the arrival of China as a major center for technological and scientific advancement in a way that will overshadow its own unbeatable record as a world leader...". Through China's Next Generation Internet (CNGI) initiative, China has reportedly spent several billion dollars making sure they got a national IPv6 backbone right.

In the US, the recent government deadline for IPv6 compliance also generated a flurry of IPv6 activity: All major vendors publicly declared their IPv6 readiness. Popular press and industry magazines filed thousands of stories on IPv6. US Federal Departments officially declared success and the Internet IPv6-ready this past June 30th.

So has imminent collapse of the Internet has been avoided?

Is the Internet moving full steam ahead towards IPv6?


The truth is that as an industry we don't have a good measure on the relative adoption success of IPv6 with respect to Internet traffic.

No idea really.

We do have some anecdotal measurements on IPv6 registry allocation and BGP announcements. But, very little data on actual IPv6 usage.

As our small effort to fill this gap, we spent much of the last year looking for IPv6 traffic in the Internet. In cooperation with the University of Michigan and close to 100 Internet providers, we leveraged commercial traffic probes across the world to measure inter-domain IPv6 traffic in the Internet. We believe this is the largest study of IPv6 and Internet traffic in general to date (by several orders of magnitude).

Our dataset covered 87 ISPs including one quarter of the tier1 ISPs and a sizable percentage of the regional / PTT providers in North America and EMEA. In all, we monitored 2,389 peering and backbone routers, 278,268 customer and peering interfaces, and an aggregate 15 exabytes of Internet inter-domain traffic at an average daily rate of 4 terabits per second (we spoke about some of this measurement infrastructure at a recent NANOG). We believe this gave us a pretty good view of overall IPv6 traffic trends in the Internet.

You can view the full technical report at

What did we find?

Not much. In fact, less than not much -- very, very little.

The below shows the percentage of IPv6, both native and tunneled, as a percentage of all Internet traffic. At its peak, IPv6 represented less than one hundredth of 1% of Internet traffic. This is somewhat equivalent to the allowed parts of contaminants in drinking water (my household water comes from the Detroit river).

IPv6 Traffic Graphed as Percentage of IPv4

Now the above graph may not be completely fair since many of the ISPs do not have infrastructure to monitor native IPv6 (more about this later). But our numbers seem to agree with data from a variety of other sources on IPv6 adoption rates.

Some related IPv6 statistics:
Percentage of ASN with IPv6 BGP announcements 3%
Percentage of Internet2 sites with passing IPv6 grade 1%
Percentage of Alexa Top 500 websites using IPv6 0.4%
IPv6 DNS queries as percentage of IPv4 DNS load 0.2%
IPv6 as a percentage of all Internet traffic 0.002%
We are not the first to raise concern over the small amount of IPv6 traffic (see Geoff's slides last month) -- just the first to have Internet wide IPv6 traffic usage measurements.

And the lack of IPv6 traffic is not for lack of trying. Many organizations and individuals offer a range of lures to encourage IPv6 adoption. For example, the next generation research and education backbone in the US, Internet2, offers free transit for IPv6 traffic. And unlike IPv4, many large ISPs have very liberal IPv6 peering policies.

The single greatest lure? For ISPs or large multi-homed enterprises struggling to justify just one more tiny, little IPv4 /16 allocation, the minimum IPv6 allocation is /32 or a staggering 2^64 larger than the entire IPv4 Internet today.

On the less pragmatic side, other IPv6 proponents offer free high quality IPv6 porn. Others yet provide ASCII animation of Star Wars movies (IPv4 users get only black & white -- make sure you watch the "Return of the Jedi" version). And, of course, the dancing turtle. Several web sites provide more listings of exotic IPv6 only Internet content.

But none of these efforts have been enough to generate any significant IPv6 traffic.

So, why so little IPv6 traffic?

Well, the biggest issue is money. Specifically, the department of commerce estimates it will cost $25 billion for ISPs to upgrade to native IPv6.

And this massive expense comes without the lure of additional revenue since IPv6 offers diminishingly few incentives nor new competitive features to attract or upsell customers. In many ways, IPv6 in the United States is much like the high definition television federal mandate (but without the mandate or the really crisp looking football games).

The harsh logic of the Metcalfe Effect also applies. With so few web sites and end users currently supporting IPv6, the incremental value to any single new IPv6 end site is limited. For many end users, v6 is an IP version all dressed up but with nowhere to go.

The third major issue is technical. While most vendors passed the OMB IPv6 requirements, it kind of depends on what you mean by "IPv6" and "requirement".

For example, some backbone routers "support" IPv6 forwarding, but not in hardware (at least not at line rate). Or IPv6 "support" does not include ACLs nor other critical security features. An ICANN survey of security vendors found that less than one in three commercial firewalls support IPv6.

Maybe you want an IPv6 MPLS VPN backbone? Sorry, not supported.

And even if your router supports IPv6, you might not be able to test or monitor it. Few vendors offer complete IPv6 SNMP / MIB support and even fewer support IPv6 Flow export (in fairness, V9 flow support is included on many Cisco cards today and Juniper has announced IPFIX support sometime in the next year). We blogged about many of these deployment issue earlier this year and Randy gave a presentation on the topic at a recent NANOG. The CAIDA and ARIN IPv6 Survey also has a nice summary on market / business forces limiting ISP IPv6 adoption.

Perhaps the biggest problem is that IPv4 works. And works well.

While IPv4 addresses are still relatively plentiful and cheap, ISPs and end customers have few incentives to migrate to IPv6. Some recent research even suggests IPv4 addresses may be more plentiful than previously believed. This tech report found that less than 4% of allocated addresses are actually occupied by visible end hosts. The authors concluded that most Internet space is likely, in fact, unused (though allocated).

All of this lack of IPv6 adoption has lead to quite a bit of hand wringing in the ISP technical community. While not declaring IPv6 a failure, discussions do wander into questions about "premature extinction of IPv6" or whether "IPv6 is an exercise in futility".

Imminent Collapse

Predicting the imminent collapses of the Internet has a long and storied history over the last twenty years. But despite all of these predictions, the Internet has survived. Sure we crashed a few routers, announced some bogus routes, and dropped a few bits here and there. But the Internet survived -- even grew a bit and gained some new users. I saw Bob Metcalfe eat a tie.

And the Internet will undoubtedly change and evolve past the impending IPv4 exhaustion.

But how?

Well, the questions is more about market forces than technology. IPv4 address allocations already have a minimal cost ($18,000 for ARIN large allocation). And growing registry management justification requirements and shrinking allocation size have steadily increased the overall cost of address space to ISPs over the last ten years. During the heady Internet technology bubble days, several companies made acquisitions in significant part based on the valuation of large legacy IP allocations.

Many think the price of IPv4 and scarcity will lead to open or if not sanctioned, black markets for address space. And debates continue whether an open market for IPv4 would be good or bad thing for Internet policy. Personally, I think an IPv4 market is inevitable.

The Future of IPv6

It is now clear the original optimistic IPv6 deployment plans have failed.

While the end of the Internet is not near, neither is IPv6. At the current rate of adoption, we are a decade or more away from pervasive adoption of dual stack support for IPv6. As Alain correctly notes in a recent IETF draft, "The IANA free pool of IPv4 addresses will be depleted soon, way before any significant IPv6 deployment will have occurred".

So IPv6 adoption will take far longer and will look far different than most of us expected back in 1994 when the IAB announced the selection of IPv6. Clearly things need to change, including IETF and vendor exploration of other technologies to facilitate IPv6 adoption such as better NAT interoperability or lighter weight dual stack.

Number of IPv6 News Articles per Month

Still, despite some of the rather anemic IPv6 traffic statistics above, IPv6 is growing. The graph above shows the number of print media articles per month mentioning IPv6 and IPv4 in the first 30 words (source MetaNews). Note that IPv6 is running almost two to one against IPv4. If judged purely by public interest, IPv6 is a winning (by comparison, DNSSEC averages only 50 articles per month and barely peaked at 150 during the DNS crisis. BGPSEC fared even worse).

The below graph shows the aggregate average daily IPv6 (tunneled and native) traffic across 87 ISPs over the last year. Since July 2007, IPv6 traffic has grown by nearly a factor of 5 to an average of 100 Mbps per day. BGP tables show an even larger proportional growth. Though not a landslide of adoption, it is still something.

While it is easy to poke fun at predictions of the "Imminent Collapse of the Internet", the eventual exhaustion of IPv4 allocations is real. We need to do something. And IPv6 is our best bet.

Aggregate IPv6 Traffic Internet Wide

So, I'll end with my top four predictions on IPv6 growth:
  1. Islands are beautiful too. IPv6 may succeed in the same way multicast failed. And by multicast failing, I really mean multicast succeeded. Though multicast never evolved a business model to justify its originally envisioned Internet wide inter-domain deployment, multicast has been astonishingly successful within the enterprise and ISP service infrastructure. Almost all Fortune 500 enterprises use multicast in some form to broadcast company video or update applications. Similarly, multicast is at the core of most ISP IPTV infrastructure.Like multicast, we are seeing rapid adoption of IPv6 within consumer, IPTV and 3G / 4G mobile providers for management of their internal infrastructure. You can pick your favorite market driver: 3G / 4G Mobile IP, Digital Radio, RFID, Control Networks, etc. But the numbers for globally unique end devices is staggering no matter which trend you choose.

    For example, Comcast has migrated all of their internal control network to IPv6 with plans to manage 100 million cable modems.

    Various estimates place number of network devices that will need to be managed at 12 billion by 2012. Note that these devices may not need global visibility, but providers will need to at least internally provision and manage (and RFC1918 space is not a reasonable option).
  2. Market trumps technology. And politics trumps markets. The future of the Internet is not fixed line devices. Nor is it users in the United States.The future of the Internet is likely both mobile devices and emerging Internet countries like China which reportedly surpassed the number of US web users at 253 million last month.While politics and government mandates do not always drive technology (see GOSIP or metric system in the United States), sometimes they do (see metric system in United Kingdom).

    Throughout the world, government mandates are spurring IPv6 adoption. China's CNGI initiative and billions in spending uses IPv6 as the centerpiece. Similarly, Japan, Korea all have major IPv6 initiatives. The EU called for mass migration to IPv6 by 2010.

    The important bit to realize about governmental focus on IPv6 is that it is not about technology nor is it even really about IPv6. Many governments view IPv6 through the prism of politics. These countries, rightly or wrongly, view ICANN and the large US centric legacy IPv4 allocations as instruments of US control. For China, Japan and many EU nations, IPv6 is really about no less than who will control the future of the Internet.
  3. IPv6 has already succeed. You can now get native IPv6 from many providers, including Verizon, Tata (formerly VSNL/Teleglobe), Global Crossing and others. Over half of surveyed providers say they have plans to roll out commercial IPv6 offerings in the next year. As more vendors integrate IPv6 into their products lines, the ISP IPv6 tax has correspondingly dropped. For many, IPv6 comes with latest refresh of hardware which ISPs generally amortize over 5-8 year periods. While it will be many years before the millions of embedded end devices support IPv6, your backbone routers likely already do.Most encouraging of all, there is finally the beginning of real IPv6 content. Or at least you can now use IPv6 to search content (as long as the indexed content is IPv4). At the Philadelphia IETF this year, Google announced support for production quality IPv6 search at
  5. And the final reason IPv6 will succeed? No one has the stomach to spend another twenty years replacing IPv6 with something else.
Personally though, I just configured our local router for IPv6 so I can watch Michael Phelps (former University of Michigan athlete) win eight golds this week at http:2001:252:0:1::2008:6.

Full disclosure -- I worked on the failed TUBA counter-proposal to IPv6 and still harbor a grudge.

Earlier Posts