But why is my music offline? BGP tracing post-Hurricane Sandy

At the time, I couldn’t reach them from California either. That presented a bit of curiosity, so I started with a traceroute to figure out what might have happened.

$ traceroute di.fm
 traceroute to di.fm (107.6.85.40), 30 hops max, 60 byte packets
 5 xe-10-1-0.edge1.SanJose1.Level3.net (4.71.118.45) 5.906 ms 5.902 ms 6.568 ms
 6 vlan90.csw4.SanJose1.Level3.net (4.69.152.254) 6.524 ms 6.278 ms vlan80.csw3.SanJose1.Level3.net (4.69.152.190) 6.688 ms
 7 ae-71-71.ebr1.SanJose1.Level3.net (4.69.153.5) 6.478 ms 4.454 ms ae-61-61.ebr1.SanJose1.Level3.net (4.69.153.1) 4.625 ms
 8 ae-2-2.ebr2.NewYork1.Level3.net (4.69.135.186) 77.596 ms 77.703 ms 77.687 ms
 9 ae-92-92.csw4.NewYork1.Level3.net (4.69.148.46) 78.191 ms ae-62-62.csw1.NewYork1.Level3.net (4.69.148.34) 93.865 ms 93.851 ms
 10 ae-81-81.ebr1.NewYork1.Level3.net (4.69.134.73) 77.554 ms 77.763 ms ae-71-71.ebr1.NewYork1.Level3.net (4.69.134.69) 77.761 ms
 11 ae-2-2.ebr1.Newark1.Level3.net (4.69.132.98) 89.285 ms 78.158 ms 88.893 ms
 12 ae-1-51.edge3.Newark1.Level3.net (4.69.156.11) 77.901 ms 77.647 ms 78.381 ms
 13 * * *

It looks like I can get to Level 3′s network, but something is breaking either within their system or when it hands off to another network. Often the first step is to check from a few other locations. I tend to log into my EC2 instance in Virginia when I want a view from somewhere else, but I’m confident right now the whole network connection is going wrong since I’m seeing statements on Twitter too. So, we’ll move to the next step: the Looking Glass.

Looking glasses are machines on provider networks that provide information to the public and other providers to help with diagnosing problems. BGP4.as provides a good list of looking glass systems.

I first started with the Hurricane Electric looking glass. Their system won’t let me link, but typing in 107.6.85.40 and selecting “BGP Route” would give me the information I was looking for. The key point is that the address I was looking for was advertised by AS29791. Voxel.net is now part of Agile Hosting Solutions which is owned by Internap and indeed visiting Voxel.net will bring you to Internap.com.

We can learn lots about an Autonomous System through its Peering Database entry as well. Between the traceroute and the peering database entry, we can presume that their system went offline in the 165 Halsey Street “meet-me” room in Newark, NJ.

Knowing this, we can try to find other peering points that might work besides, letting us know if the network is down only for people entering via Level 3. Searching for networks Voxel peers with or other exchange points can be helpful for this. A less involved method is to cue up a looking glass site and select many different locations to ping from at once.

I tried to paste some text, but WordPress decided to use the “Format Like Hell” option and manually pasting in the CSS was… unpleasant. Regardless, from the screenshot we can see that the site is now online, but still hobbling from Fremont. Every time I run traceroute, the mci1.he.net route (Level3 Kansas City) is ok, but the fmt1.he.net (Hurricane Electric Fremont 1) link shows packet loss in the Voxel network.

As a network provider, this would be a time to possibly kill the route. As an end-user, if it’s bad enough I can try to call any of the providers in my upstream path to them as seen from my traceroute and convince them to change the peering. If I can’t get any help, I can also tunnel my connections to a machine that does have a working path. Most people shouldn’t be trying to get BGP routes updated, but network administrators should be familiar and know who to call that is on a network operations center list. See also the NANOG mailing list.

With that, I can now find a better way to stream my online music and know what part of the Internet is broken.

  1. This is a great article, and reminds me of my days working in a NOC. AS outages, internally or at peers, was always a problem. Looking Glasses made my job much easier.

  2. Aw, this was a really good post. Finding the time and actual effort to generate a superb article… but what can I say… I hesitate a lot and don’t seem to get anything done.

  3. If my problem was a Death Star, this article is a photon topdreo.

Leave a Comment


NOTE - You can use these HTML tags and attributes:
<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>