From Megabits to Milliseconds: Latency and America’s AI Infrastructure Gap

The following is an excerpt from the Connected Nation white paper: From Megabits to Milliseconds: Latency and America’s AI Infrastructure Gap

The Arithmetic of Lag Time - Why bandwidth is no longer the bottleneck.

Broadband policy has long focused on bandwidth, for good reasons. In the 2010s, demand growth was driven by consumption, especially streaming video. In 2020, remote work elevated the importance of upload speeds as homes became workplaces. Federal programs such as the Rural Digital Opportunity Fund (RDOF) and the Broadband Equity, Access, and Deployment (BEAD) Program reflect these priorities, targeting the performance metrics most relevant to legacy internet uses.

With or without subsidies, fiber optic cable deployment to homes and businesses has accelerated to the point that a majority of U.S. locations now have access to fiber-based services and gigabit download speeds—a share that continues to grow. Fiber providers often describe their networks as “future-proof,” and in one sense this is likely true. The bandwidth fiber networks can deliver far exceed the requirements of any internet use case that is widespread today or plausibly on the horizon, and fiber-based services are upgradable not by replacing the fiber line itself but rather the optical gear used to “light” it. And escalating bandwidths transfer the speed pain point to a different performance metric. This is partly a matter of simple arithmetic.

Total delay in an internet interaction consists of two components: transmission time and latency.

Transmission time depends on how fast a connection can push bits through a medium, such as a fiber optic cable. In contrast, latency depends on distance, routing efficiency, and physical limits—most fundamentally, the speed of light, although many factors drive practical latencies far above that theoretical limit.

On a low bandwidth (and in that sense “slow”) connection, transmission time dominates. For example, a 1 MB payload takes roughly 320 milliseconds to transmit over a 25 Mbps connection. Add 80–120 milliseconds of latency—a realistic figure for underserved areas where traffic must reach a distant hub—and most of the delay is bandwidth-driven.

By contrast, on a faster 1 Gbps connection, that same payload transmits in about 8 milliseconds. With that bandwidth, even a modest 35ms of latency means that over 80 percent of the delay now comes from the network itself rather than the link speed. In underserved areas, like the communities targeted by the BEAD program, round-trip latency to a distant hub commonly runs 80–120 milliseconds, making the case even stronger.

FIGURE 1: The arithmetic of lag time. On a 25 Mbps link, transmission time dominates. On a 1 Gbps link, the same 1 MB payload arrives in a fraction of the time — and latency becomes roughly 80 percent of the delay users feel.

As fiber becomes widespread, this inversion becomes universal. Latency, which has improved only modestly over time even as bandwidths have surged, is probably becoming the dominant source of lag time in most interactions with the internet, although this claim faces both statistical availability and definitional barriers to being firmly established.

Critically, the enormous investments and important innovations in last-mile connectivity do little to address this fact. Latency depends primarily on the length and efficiency of the data journey: middle-mile infrastructure, routing policies, and proximity to Internet exchange points (IXPs). Most milliseconds of latency lie outside the last-mile access network. Throughout this article, “IXP” is used to refer to a spectrum of interconnection facilities—from neutral traffic exchange points where networks peer directly, to carrier hotels, regional colocation facilities, and edge compute sites. These differ in ownership model and function, but share a common characteristic: they are the places where data changes hands between networks, and where proximity and peering density determine how efficiently traffic moves. Being near such places is key to achieving low latency.

Fiber expansion is on track to solving the bandwidth problem permanently in most places. But as deployment expands and bandwidth constraints recede, other aspects of network performance—especially latency—emerge as the new binding constraint.

To read the full white paper please click the link HERE!