How the Internet Died

Tuesday, August 12 and Wednesday, August 13 the internet seems to be hiccuping along and generally less stable than normal. This isn’t the first or even second time the internet broke on a global level. Due to 10 year old default configuration and business inaction many ISPs and enterprises encountered network outages.

First, some history…

Part 1: The routers
In networking there are two types of routers: software based and hardware based. I cover these in more detail in a presentation I’ve delivered at Cisco Live. Most routers connected to the internet are hardware based. On hardware based routers all routes are installed in specialized TCAM (or some sort of high speed memory) for an ASIC or specialized processor to perform lookups. TCAM is a limited (and expensive) resource. Because of this, there is a fixed number of routes that can be installed into TCAM. More TCAM means more routes, which means more money. If you’ve ever upgraded a 6500 from a 3B to 3BXL module, what you are buying is more TCAM.

Part 2: BGP and the internet
BGP announces every route on the internet. The internet works by having every router on the internet know about (almost) every route and how to get there. The number of routes you hear from your ISPs will vary based on your provider and where you are located in the internet. Either way, give or take a few hundred routes, everyone will have a similar picture of the global internet. Services like the CIDR Report help track the growth of the size of the internet routing table. In March the CIDR Report showed the BGP table crossing the 500,000 route mark for the first time.

What happened?
In 2004 Cisco came out with the Supervisor 720-3BXL module. At the time, this was cutting edge technology in its ability to forward packets and store routes. At the time the BGP table size was around 125,000 routes. Announcing a router module that supported 512,000 routes by default, expandable to 1,000,000 routes seems ahead of its time.

Fast forward 10 years later, the Sup720-3BXL has become a workhorse in both Service Provider and Enterprise networks. Although no longer the fastest box on the market, it has supported interface modules fast enough (multiple 10gig) that there hasn’t been a strong need for most organizations to spend the money to replace them. While the Sup720-3BXL has been running well, the internet table has continued to grow closer and closer to the default TCAM allocation size of 512,000 routes. Unfortunately only users looking for the problem would notice it.

On August 12, Verizon announced new prefixes to the internet, causing the BGP table size to cross that 512,000 route mark. Suddenly hundreds of routers have exceeded their default TCAM memory space. It’s important to note that this isn’t a permanent problem. The amount of TCAM can be changed as described in a document I wrote on cisco.com. This memory space is shared between storage for IPv4 and IPv6 routes. You can reduce the space for IPv6 routes and increase the space for IPv4 routes.

This issue also impacted the Trident line card on the ASR9000. Just like the 6500 the TCAM space can be adjusted to solve or avoid the issue.

Impact of Exhausting TCAM
On a computer, you create swap space to take over when you run out of much faster RAM, causing a massive drop in performance. On hardware based routers a similar, but much worse, problem occurs.

When the number of routes exceeds the available space in hardware, the router can no longer forward packets using the specialized processor or ASIC.The general purpose CPU now begins forwarding packets. On the Sup720 this CPU is 600 Mhz, slower than an iPhone 4. The Sup720 ASIC can forward millions of packets per second. The general purpose CPU can only forward a few hundred packets per second (at best). The sudden flood of traffic will not only cause packet loss due to congestion to the CPU but can also cause routing protocol peers to drop, making problems even worse. Finally, because the data that is in hardware can’t be trusted, the only way to recover from a TCAM exception state is a complete reboot of the router.

“Older” Network Equipment
Multiple news outlets have used the term “older network equipment” to describe the routers impacted by this issue. While not entirely wrong, it gives the impression that it’s the networking equivalent of an eMachine chugging along struggling to do it’s job. This isn’t the case for most customers. There will come a time where the number of routes on the internet will force hardware to be replaced, that time won’t be for a few more years. If customers had taken the time to schedule a maintenance window to reallocate TCAM space this issue would never have been seen. Of course this doesn’t address the problem of informing customers of the problem on the horizon.

And Newer Platforms?
Some platforms are now based on dynamic high speed RAM, like the ASR1000 can can support larger tables, or at least fail in a more graceful fashion.

The Sup2T for the 6500/6800 is based on dynamic TCAM and is shared between Ipv4 and IPv6, so there is no need to recarve TCAM.

The ASR9k story is a little more complicated but newer line cards have much larger TCAM space.

Advertisements

5 thoughts on “How the Internet Died

  1. It’s a shame we all speak abut the 6500/sup720 failure only, but I’m pretty sure the issue was caused also by other’s vendor equipment. Clear and concise first post thanks.

    Like

  2. @feedtheswitch, I don’t mean to imply it’s only a Cisco problem. Any system is going to have limits and most of those limits are going to be on some byte boundary. My background is almost 100% Cisco and I can’t speak from experience regarding other vendors.

    Like

  3. Pingback: Show 201 - Internet Dies at 512K, Long Live the Internet - Packet Pushers Podcast

  4. Pingback: ASR1000 Nuts and Bolts | Excessive Redundancy

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s