Towards Gigabit Networking

Keynote Address - Extended Abstract

Positions On Gigabit Networking

1. Gigabit Bandwidth to the User

"Not only could I use O(1 Gbps) to the application on a desktop workstation, I could have used O(10 Gbps) to my application last year"

-the supercomputing community

"We do need Gigabit per second networks, but only inside the network for links and switches to handle aggregation of traffic; individual users will never need more than O(100 Mbps) each"

-the skeptics

2. Scalability of Current Networks

"Current network architecture and protocols break at high data rates; we need new architectures, mechanisms, and protocols above Gigabit per second rates"

-the high speed protocol community

"There is nothing wrong with current networks; we just need to make minor tweaks (such as resource reservation) and everything will scale well."

-the Internet community

3. ATM vs. IP

"The current Internet works pretty well, and we are in the midst of enhancements (IPv6 and RSVP) which will allow emerging applications to work just fine; ATM merely causes a whole bunch of new problems based on a severely broken model (hard connections, small fixed size cells)"

-the Internet community

"The Internet is slow, insecure, and doesn't handle connection oriented traffic well; a global seamless ATM infrastructure will solve all our problems"

-the ATM community

4. Integration of Traffic and Networks

"Given the right real-time protocols and network support for resource reservation, a single seamless global information infrastructure will be able to support all current and future traffic"

-the Internet and ATM communities

"The telephone network is reliable, predictable, and we understand how to provide and bill for desired QoS; the Internet is too unreliable and unpredictable for real paying customers, and we haven't got the foggiest notion how charge for it"

-the skeptics

These positions will be revisited after discussing the role of ATM and B-ISDN in the emerging global information infrastructure, and how they fit in the larger context of Gigabit networking.

A Brief History of Networking

ATM and B-ISDN in the Global Information Infrastructure

There are, however, a number of serious problems with ATM and B-ISDN, partially based on the current standards. The overhead associated with the small cell size is the most frequent criticism, but there are other more serious problems. The small cell size results in a short cell time (in which to make per cell control decisions), and the payload size is not a power of two; both of these significantly increase the cost and complexity of cell processing by end systems. The requirement for sequenced delivery (which is a result of not having a large enough header to carry a sequence number) makes it difficult to stripe links and switch planes to increase effective bandwidth, prevents multiple connection paths, and complicates connection path reconfiguration for load balancing or recovery. Finally, the signalling (particularly B-ISUP with its SS7 legacy) is overly complex, and was not designed for efficient implementation.

There are a number of other challenges in deploying such a broadband infrastructure. Technical challenges include virtual connection routing in a large hierarchical network (particularly for multipoint-to-multipoint), and traffic management and congestion control. The cost of providing optimal resource allocation may outweigh the cost of providing some over-engineering of the network. Network management, billing based on traffic characteristics (e.g. burstiness), and security (per cell and per connection) are all open issues.

There are also a number of practical challenges, the most important of which is how ATM can possibly co-exist in a rational manner with the current and emerging Internet architecture and protocols (including IPv6 and RSVP). Both ATM and IP multiplex, and there are two sets of incompatible address spaces, routing topologies, and signalling mechanisms. IP currently provides the "hourglass" in the protocol stack that allows many transport protocols and applications to work over a variety of subnetworks; ATM is also attempting to provide such an "hourglass". Considerably more detail on these issues is presented in [1,2] which also contain references to the surprisingly few other ATM critiques in print.

Gigabit Networking Above the Network Layer

We still need to deliver high bandwidth at low latency to applications that need it (as opposed to delivering high bandwidth only to the edge of the network). We still have not solved the latency problem (bandwidth-x-delay product) for WAN applications, or provided for the delivery of this bandwidth to the applications through the host architecture and operating systems.

For the higher layers and end systems, we should consider how much bandwidth a single application can use. A current example is the use of the WWW (World Wide Web), which appears to be the killer application we've been waiting for [3]. While the network bandwidth currently supported is rather low, so is the corresponding response to files and images that are fetched by clicking on hyperlinks. It is not hard to imagine scenarios in which bursts of O(100 Mbps) would be desired by a single user, particularly when large high resolution images or blocks of data to be rendered locally are fetched; parallel prefetching increases the bandwidth demand even more. Telepresence applications can also demand very high bandwidth, particularly when more than simple compressed video is involved. Gigabit applications are discussed in more detail in [4].

We also need to ask what latency is tolerable. The WWW currently behaves as a request/response system, and the standard sub-second response time arguments apply; The Web is increasingly looking like a huge distributed timesharing system. Thus, from the time a user clicks on a hyperlink to the time that the screen is full of whatever was requested should not exceed O(100 ms). Various estimates on the network topology and switch latency, coupled with the distance at most half way around the world give us a network latency of O(10-100 ms). Thus we are potentially close to the O(100 ms) limit, before adding the transmission time of the data and the processing time on the local and server hosts. Similarly, session control and network signalling protocols need to be optimised to a single round trip whenever possible, with cached state (such as URN to URL resolution).

So, we must at least deliver the right piece of information to the user with low latency at the end systems. At best, we should predict and prefetch data that will likely be requested next (e.g. breadth first using all the hyperlinks in the current HTML page). We need to recognise that bandwidth is a resource to be traded against memory and processors, and idle bandwidth can be used to our advantage to effectively beat the speed of light. Bandwidth is not free, but it is getting cheaper all the time, in both absolute and relative cost.

End-to-End Principles and Protocol Research

A number of the bottlenecks in high speed communications are operating system related, and they are being well documented. These primarily relate to avoiding unnecessary context switches and extra data copying. In the ideal case, it is possible to have only a single context switch pair: one when the application requests the communication operation, and one when the operation is successful and the application can resume (in the case of a request for information this is when the first complete piece of information the application can use is in place on the host and available in the application user space). While we know how to engineer operating systems and the host memory hierarchy (cache, main store, backing store) relatively well for data, how to manage audio and video objects is still an active area of research.

There was a flurry of activity in new transport protocol design a few years ago, but the protocols are only one piece of the end system bottleneck. This does not mean that existing high layer protocols in their current form are adequate for high speed networks and the corresponding host and network interface architecture, but rather that when we create or modify protocols it should be for the right reasons. We should also note that the difference between a "new" protocol (e.g. TCP-like replacement) and a highly modified "old" protocol (e.g. TCP++) may be in name only. Furthermore, hacking a new protocol from an old one by simply omitting what can't be done fast does not necessarily result in a useful protocol.

While there is nothing inherently "gigabit" about multimedia (we've got cruddy multimedia on the Internet now), the advent of the broadband network infrastructure will enable the wide scale deployment of a number of new applications (including multimedia) to a degree not possible before. Thus we wish to provide support for new integrated media (data, audio, video, and other sensory) applications over the broadband network infrastructure and the delivery of the high bandwidth directly to the application.

So what do we need in protocols for high speed networks? Generic research on high speed protocol architecture is of course important, as is the optimisation of protocol control mechanisms. Furthermore, in the spirit of not inventing new protocols and interfaces just for the sake of it, the extension of existing protocols (or protocol implementations) such as TCP for high speed should be done when possible. Additionally, protocols which provide support for multicast and multipoint applications will be particularly important in the emerging broadband environment. B-ISDN and ATM networks provide this support natively at the network layer with multipoint connections; IP networks now provide this with the addition of multicast support.

It is important to remember that the point of a transport protocol is to provide the functionality desired to applications (with sufficiently low latency, high bandwidth, and low overhead) while providing the end-to-end transport over the broadband infrastructure. Thus, the transport protocol is really not an end in itself, but rather a "glue" that extends the high bandwidth pipe from the edge of the network through the host, network interface, and operating system to the application.

Back to the Position Statements

1. Gigabit Bandwidth to the User

We've got Gigabit bandwidth in network aggregation now, and are just beginning to see Gb/sec switches and links. There seems no reason for the demand for bandwidth to suddenly flatten; processor, memory, and bandwidth requirements continue to increase. The Web is certain to drive high bandwidth demand for individual users. Today's supercomputing applications are tomorrow's mass applications (e.g. simulation and modelling for CAD and agriculture). Gigabit bandwidth to the application is simply a matter of economics; when it is cheap enough, it will be widely deployed. There is a real problem, however, in having the incentives in place to build the infrastructure on which new applications can emerge.

2. Scalability of Current Networks

Scalability is clearly a hard problem, and there are few people that would seriously defend the status quo as stated at the beginning of this paper. In fact, the Internet community is actively looking at how to scale to a Giganode Internet. IPv6 targets the addressing issues; much work needs to be done in the areas of resource management (RSVP is only the reservation protocol) and routing protocols. Note that scalability in number of nodes is probably far harder than scalability in data rate, and this has nothing to do with whether it is IP or ATM in question.

3. ATM vs. IP

It is clear the ATM and IP can't co-exist well in their current state; the problems with this have already been presented. The ATM and IP communities are trying to solve the same problems, for which two disjoint and incompatible solutions are not helpful. Dismissing the extreme IP-only and ATM-only views for the present, there are a couple of scenarios that can be imagined:

ATM can be used only for the cell-relay functionality in the link and subnetwork layer. In this scenario IP provides all routing and addressing. This requires Gigabit IP routers in the backbone network, and such projects are underway (the Washington University project uses an ATM switch as the core of the IP router). This does raise questions about the need for ATM switches as distinct network elements.

The IP and ATM communities could recognise that they should work together on providing a single solution, particularly in routing and addressing. Fixed (but sensibly sized) cells make switching systems easier to design for high speeds, do provide advantages in traffic aggregation, and can map effectively into host and network interface hardware (cache lines and pages). IP already provides the routing "hourglass" between heterogeneous subnetworks and transport protocols, but work is still needed to scale to a Gigabit Giganode Internet. While there is considerable work on IP over ATM, there unfortunately seems to be little work on the real convergence of IP and ATM.

4. Integration of Traffic and Networks

The current telephone network is reliable and predictable, but is extremely limited in the traffic it will bear (voice and low rate data). The telephone networks have had decades to evolve, the traffic is extremely well characterised, and the addressing and routing algorithms are very restricted. On the other hand, the current Internet carries data, is beginning to carry video and voice, and provides functionality for a wide variety of applications and services. The Internet does not, however, provide the reliability, security, ability to bill, or resource reservations that are ultimately required. What we really need is the best of both worlds, but this is clearly quite difficult. Although we will likely live with several parallel networks for quite some time (Internet, telephone, CATV), we can expect gateways to begin to stitch them together (this is just beginning to happen for telephony to packet voice). Ultimately, the only sensible solution is for a single integrated GII which subsumes the current networks. The Internet is the emerging global information infrastructure.

Acknowledgments

Disclaimer

References

Due to space constraints, detailed references are not provided here. [1] and [2] should be consulted for further references that are highly relevant to this presentation.

1. Sterbenz, James P.G., "Protocols for High Speed Networks: Life After ATM?", in Protocols for High Speed Networks IV, Gerald Neufeld and Mabo Ito, ed., IFIP, Chapman and Hall, London, 1995, pp. 3-18; available as http://www.ccrc.wustl.edu/~jpgs/paper/pfhsnIV.html

2. Sterbenz, James P.G., Henning G. Schulzrinne, and Joseph D. Touch, "Report and Discussion on the IEEE ComSoc TCGN Gigabit Networking Workshop 1995", IEEE Network, Vol.9 #4 July/August 1995, pp. 9-21; available as http://www.ccrc.wustl.edu/~jpgs/paper/network95.html

3. Touch, Joseph D., "Defining High Speed Protocols: Five Challenges and an Example that Survives the Challenges", Proceedings of the Gigabit Networking Workshop '94 and to appear in IEEE Journal of Selected Areas in Communications, June 1995; also available as http://www.ccrc.wustl.edu/pub/ieee-tcgn/gbn94/touch.paper.ps.Z

4. Partridge, Craig, ed., Report of the ARPA/NSF Workshop on Research in Gigabit Networking, Washington, D.C., July 1994, available by anonymous FTP from ftp.std.com:pub/craigp/report.ps; also available as http://www.ccrc.wustl.edu/~jpgs/doc/arpansf94.ps

5. Proceedings of the Gigabit Networking Workshops GBN'94, GBN'95, and GBN'96, located at http://www.ccrc.wustl.edu/pub/ieee-tcgn

James P.G. Sterbenz is the chair of the IEEE Communications Society Technical Committee on Gigabit Networking and is with GTE Laboratories, Waltham (Boston) Massachusetts, USA