Running the Gamut: Legacy Sprint's Journey Through IPv6 Transition Mechanisms

By Ben Bittfield – IPv6 Network Architect, Sprint

Sprint has implemented essentially all the major categories of IP addressing and IPv6 transition techniques on the wireless network over the last ten years. Public IPv4, private IPv4 with carrier-grade NAT, dual stack IPv4v6 (both public & private IPv4 varieties) and finally IPv6-only with NAT64.

With 55 million subscribers, IPv4 was not sustainable long term for the Sprint wireless network. Sprint did not have nearly enough public IPv4 addresses, and RFC 1918 private IPv4 addressing did not scale to the tens of millions required. As IPv6 transition mechanisms matured through the years our strategies may have shifted to match, but it was long known that IPv6-only would be the ultimate target state of IP addressing.

Public IPv4: We’re like a wireline ISP!

Prior to the late 2000s and the avalanche of smartphones, wireless carriers were able to immensely oversubscribe handsets to public IP addresses for flip phones. At this point in time, Sprint continued to allocate publicly routable, dynamic IPv4 addressing on all customer devices. For as long as sustainable, Sprint maintained assigning public IPv4 addressing, and our Sales Department had used this as a differentiator to liken Sprint wireless as a wireline ISP, as other wireless carriers in the United States had all moved on to carrier-grade NAT. It was also cheaper from a mobile operator perspective to remain without a stateful firewall for as long as possible since our public IPv4 traffic is and continues to be stateless, but carrier-grade NAT inherently requires stateful sessions.

Carrier-Grade NAT: A Necessary Evil

After smartphones launched and a basically always-on and 1:1 handset to IP address ratio, it was essential for Sprint to immediately move to a different type of IP addressing as public IPv4 on every handset was no longer sustainable. Smartphones quickly became the predominant handset type, so a long glideslope or timeline was not a luxury for this project. A solution needed to be implemented quickly to prevent the blocking of data sessions for customers due to lack of IP addresses. At this time in the early 2010s, IPv6 in mobile carriers was fairly new, as only Verizon Wireless had implemented this in production among North American carriers. While knowing the endgame was ultimately IPv6, Sprint had to implement an immediate solution, with the lowest cost of entry and lowest risk of impact, so we moved to carrier-grade NAT (CGN). The launch of our CGN solution involved minimal impact to customers, since as mentioned above, Sprint was one of the last wireless carriers to implement CGN.

At this point, Sprint was the remaining wireless provider on public IPv4, so enterprise sales continued to use this as a differentiator for our mobile broadband solution. While handsets would move towards the CGN solution, mobile broadband devices and Wi-Fi tethering would continue to remain on public IPv4 for the foreseeable future to replicate the wireline ISP experience as much as possible. Even when dual stack IPv4v6 was launched on mobile broadband devices like hot spots and data cards, public IPv4 addressing was used.

Initial Steps: IPv6 Addressing Plan & IP Core Infrastructure

Once the short-term disaster of public IPv4 exhaustion was mitigated, a multi-step, multi-year approach to the target state solution of IPv6-only was kicked off. Regardless of the IPv6 transition mechanism chosen, the overall IPv6 addressing scheme needed to be laid out. Based on various case studies and presentations regarding IPv6 address planning, Sprint approached this knowing that the initial address scheme would likely need to be revised over the years as the network evolves, so the layout was adaptable and flexible, with plenty of space in between subnets to allow for growth, and numerous “escape hatches” if the network fundamentally changed completely. We also approached this addressing plan with a consistency and simplicity based approach, rather than designing for capacity in each site. While inefficient, the scale and massive amount of IPv6 space allow for the consistency and simplicity that would help our engineers and operational teams in adopting and learning the new addressing protocol. At first, IPv6 addresses are new, big, and scary, so whatever could be done to make it friendlier to the average engineer, the better.

As other migration approaches and case studies have recommended, Sprint started in the core and worked outwards. In this case, that meant the transport path in the wireless IP core network from the LTE gateway elements to the internet would be required to be dual stacked. Once again, for simplicity, Sprint chose to dual stack and reuse the existing vlans to make things simpler. Each wireless datacenter is identical in the call path from the LTE gateway elements to the internet connectivity and the various vlan numbers were well known across the engineering and operations organizations, so piggybacking on those without adding new vlans was beneficial. There was an unofficial motto of “No layer 2 changes” for our IPv6 efforts, since in the current network architecture it didn’t make sense to create new vlans just for IPv6, as IPv4 already had a call path from the gateways to the internet or services platforms.

Evolution to Dual Stack IPv4v6

At this point in time, an IPv6-only + NAT64 network had not been launched in production on mobile wireless carriers, so the general IPv6 transition conversation was centered on dual stack IPv4v6. While it was known that IPv6-only was the ultimate goal, IPv6-only through 464XLAT (RFC 6877) was new and unproven at scale at the time. Sprint wireless also had significant negative customer experiences through the multi-year Network Vision upgrades and our “Pardon Our Dust” initiative, so the less risky, “make-before-break” approach of dual stack IPv4v6 was the preferred choice; knowing that a second initiative to IPv6-only would be required in the future. There was also significant effort to reduce project costs as much as possible, because IPv6 isn’t the type of project to show immediate monetary benefits in the next quarter.

With dual stack IPv4v6, both IPv4 and IPv6 addresses would simultaneously be assigned to the handset upon LTE attach. The handset connectivity to the far side internet content would be decided by DNS; generally, if an IPv6 AAAA record was available for the website, the handset would prefer IPv6 transport for that content. If no AAAA was found, and an A record was returned by DNS, then IPv4 transport would be used. While that is the overall concept, this is a simplified version of what is called Happy Eyeballs (RFC 6555) where DNS and various timers are used, specified by each handset OEM, OS, or browser, to choose IPv6 or IPv4 while not impairing the customer experience with abnormally higher latency than usual. In the vast majority of cases, this meant that if IPv6 content was available for a dual stack IPv4v6 handset, then IPv6 would be preferred with lower latency than IPv4. This resulted in roughly 70/30 split in traffic, in preference to IPv6. High tonnage content such as Google, YouTube, Facebook, Netflix, etc. are all IPv6 enabled, so the majority of handset traffic followed suit.

In hindsight, dual stack IPv4v6 was a wise decision as there were still significant delays due to vendor issues.  Sprint had initially aimed to release dual stack IPv4v6 handsets in 2013; however, that was delayed a full two years due to above-mentioned vendor issues. Some issues were IP core-centric, while some were regional LTE vendor issues. As Sprint added IPv6 traffic load during a friendly user trial, further issues were uncovered, resulting in additional delays. For all defects or delays, Sprint waited until full nationwide readiness was available so the wireless customer would have the same experience and IP address type nationwide. Finally, once all network issues were resolved, dual stack IPv4v6 was launched on newly released handset models with minimal to no customer issues, thanks to the resiliency of Happy Eyeballs.

Target State of IPv6-Only+NAT64

After successfully launching dual stack IPv4v6, Sprint turned its engineering focus on implementing the target state handset IP addressing of IPv6-only with NAT64+DNS64. By this time, other major carriers such as T-Mobile US and Orange Poland had large-scale production deployments of IPv6-only with NAT64+DNS64 via 464XLAT, with other carriers pursuing the same direction.

464XLAT is a novel solution that involves NAT64+DNS64, but also essentially includes a NAT46 client translator within the end device (Android & iOS, in this case) to allow reachability for hard-coded IPv4 literals via a “fake” IPv4 address that is presented to the application/OS, for cases where DNS does not come into play and DNS64 cannot perform the translation to legacy IPv4 addressing.

Because of foresight by the architects of the IPv4v6 project, the majority of billing changes were accounted for and implemented as part of the previous project. However, there were some provisioning changes to be developed as IPv6-only uses a new APN, as well as implementing the NAT64 & DNS64 architecture, changes to the LTE gateways, and IPv6 connectivity to service platforms such as MMS and Visual Voicemail.

With a few exceptions early on, all Sprint Android handset models released after July 2017 have been on IPv6-only. While the pure NAT64+DNS64 aspects of the IPv6-only handset launches were fairly straightforward, there were a handful of customer impacts regarding these handset launches due to the unfortunate nature of several major network architecture changes occurring simultaneously as these handset models were released. Apple iPhones would not launch on IPv6-only until the following fall in 2018 as the Apple-related service platform network changes were not implemented in time for the 2017 hardware release.

Performance & Metrics

Since Sprint didn’t want to impact customer devices in the field, we released IPv6-only in newly launching handset models and were relying on attrition through handset upgrades to migrate off IPv4/IPv4v6. Within a few months, we started to see a recovery in IPv4 addressing. Peak IPv4 addressing was reached in April 2018, with IPv6 becoming the majority in late December 2018.

Additionally, per the Internet Society’s “World IPv6 Launch” website, nearly 90% of traffic from Sprint wireless to IPv6-enabled content such as Google, Facebook, etc., is IPv6.

Regarding latency and other performance improvements, Sprint sees performance metrics similarly on-par with reported third-parties such as LinkedIn, Facebook, Akamai, and others. Where IPv6 is native end to end, there is no latency delay created by the NAT functionality. For more details regarding these differences, see Lee Howard’s write up or this LinkedIn presentation.

Conclusion

IPv6 adoption is like the proverb about planting a tree. “The best time to plant a tree was 20 years ago. The second best time is now.” Sprint was among the last major wireless carriers in North America to the initial game of IPv6, but now the majority of handsets on the wireless network are IPv6-only. As more end-users obtain IPv6 and more content becomes available on IPv6, the best time to start a migration plan is now. There is a wealth of information available in these case studies here on ARIN, a substantial amount of which have proven significantly useful to Sprint on our rollout of IPv6 throughout the years.

Post written by:

Ben Bittfield
IPv6 Network Architect, Sprint

Any views, positions, statements, or opinions of a guest blog post are those of the author alone and do not represent those of ARIN. ARIN does not guarantee the accuracy, completeness, or validity of any claims or statements, nor shall ARIN be liable for any representations, omissions, or errors contained in a guest blog post.

Recent blogs categorized under: IPv6

• The Business Case for IPv6-Only Enterprise
• 3 Strategies for Ensuring the Security of Your Digital Infrastructure
• The Business Case for IPv6: Crossing the IPv6 Chasm
• Caribbean Digital Progress by the Numbers