The Internet's success can be contributed to a simple abstraction it provides: any host can send packets to any other host, on any type of link, anywhere in the world. However, Internet infrastructure is the tip of the iceberg. Internet service providers (ISPs) conduct and take response of maintaining these infrastructures, but they never share the information with any of their customers nor any government organization. Regional access network, as one of the major components of Internet infrastructure, plays the key role in serving users and bridging customers to the backbone network. In the rapidly evolving landscape of Internet infrastructure, especially during COVID time, understanding how regional access networks are structured is crucial for enhancing connectivity and service quality.
What is Regional Access Network
An ISP's Internet infrastructure could be roughly divided into two main components - regional access network and backbone network. Regional access networks serve as vital conduits, bridging the gap between end-users and the broader Internet backbone while backbone network serves as packet highway which interconnects regional network globally.
Regional access networks are not just passive channels; they actively manage and aggregate traffic to balance performance with cost, especially across vast geographic regions. There are many hardwares in regional network including switches, routers, transformers, DSLAM, fibers etc. ISPs will construct a building to assemble all the centralized hardware. We define "Central Office (CO)" which is a center who contains a number of hardwares like switches and routers, in an area. In theory as well as from our research, we found ISPs have three tier levels of COs including backbone CO, aggregation CO and edge CO. They all together serve and manage traffic among their customers.
An ISP network can be divided into backbone and regional access.
However, theory is just a guidence for pratical. How ISPs deploy and organize their access network in the real world is an unknown to public. Moreover, regional access network's complexity and opacity pose significant challenges for academic research aimed at improving network performance and reliability. This research aim to provide a replicable third party strategy to infer the regional access network topology.
McTraceroute and ShipTraceroute
The research team employed a variety of advanced Internet measurement tools to infer the topologies of regional access networks operated by major U.S. ISPs including Comcast, AT&T, Spectrum for wireline providers while AT&T, Verizon and T-Mobile as wireless service providers. The two major difficulties of revealing regional access network topology are lacking of "good" vantage points(VPs) and finding "right" targets. Finding "right" targets is mainly by using DNS name to infer the related regional infrastructure and using traceroute results to find the right set of IPs.
The reason "good" VPs are important is because most of the ISPs will block external packet probing their access network infrastructure to avoid attack and reduce traffic. Therefore, to reveal regional access network topology requires us to have VPs inside of ISPs. Our ways to overcome it is by proposing McTraceroute and ShipTraceroute.
McTraceroute aims at revealing wireline service providers' access network topology. It leverages local fast food restaraunts' free Wi-Fi as they use service from local ISPs. Besides, fast food restaraunts are usually evenly spread across a region which gives us a chance to view the region's network from multiple vantages.
McTraceroute leverages local fastfood restaurants' Wi-Fi to probe ISPs' regional access network infrastructure.
ShipTraceroute uses the movability of mobile devices to uncover cellular service providers' network topology. It puts phone into box and uses ground shipping to send devices across the country. In the meanwhile, phone keeps sending out measurement packets and using duty cycle to save power. By leveraging this method, we have a view of the topology of mobile access network.
ShipTraceroute puts phone into a package and uses ground shipment to probe ISPs' mobile network infrastructure.
If you want to use McTraceroute and ShipTraceroute to uncover the network in your region, please check our github repo for more information.
Key Findings and Implications
One of the study's significant revelations is the diverse strategies ISPs employ to manage and aggregate traffic. For instance, we figured out there are three different topologies for wireline network providers in practical including stars, dual stars and multiple layers of stars with increasing number of customers.
Three different types of wireline network topology
We also find mobile regional access network is a surprisingly centralized, all the mobile traffic from base stations will first aggregate at an edge CO before routing to the backbone CO in one region.
Cellular regional access network topology
This network topology design creates a significant differences in latency from Edge Central Offices (COs) to Backbone COs, which have direct implications for cloud service performance experienced by end users. For instance, comcast in east coast aggregates VT, NH, MA and CT's network into a region. Their backbone network is in Maryland. Based on our cloud servers probe experiment, we can clearly see that users in VT has generally higher latency than users in MA. And this latency is unavoidable because all the Comcast's user in VT has to go through aggregation central office in Maryland before accessing to the rest of the Internet.
Comcast uses multiple layer stars to aggregate VT, NH, MA and CT's network into one region which causes user in VT has higher latency than other states to have access to cloud services.
Same situation is being observed in mobile network as well. We found AT&T divided the whole US into 11 regions while for Verizon we observed 25 regions. T-Mobile is the most distributed as they uses other tier 1 network to provide their backbone services. As we can tell from the following figures, AT&T Mobile users in MT will suffer a high latency as the backbone CO for them are in Chicago. Same for VZW customers in AR as well.
AT&T Mobile customers in MT and VZW users in AR will suffer an unavoidable latencies as their backbone COs are in IL.
Impact on Future Technologies
These findings are not just academic; one of the big security take away from this project is there's no actual redudancy in the building level for backbone CO in each region. If attackers take out the backbone CO building, the whole customers in that region will go offline. This attack happened at AT&T Nashville. In the next blog, we will show how we can use this topology to understand many big network outages happen in the historical news. Understanding the underlying access network topologies helps anticipate challenges and opportunities in enhancing network resilience and reducing latency, crucial for next-generation Internet applications.