Method and apparatus for queuing data flows

1

METHOD AND APPARATUS FOR QUEUING
DATA FLOWS

FIELD OF THE INVENTION

5

This invention relates to cable data systems. In particular, this invention relates to a cable modem termination system (CMTS) and a method for intelligently controlling data flow queues in a CMTS

10

BACKGROUND OF THE INVENTION

A cable television or "CATV" system is a television broadcasting system that uses coaxial cable or fiber optic cable to distribute a broadband signal containing many 15 separate television program channels that are frequencymultiplexed onto the cable. By allocating the frequency spectrum of one or more television program channels for data transmission, CATV systems are also used to carry data to and from CATV subscribers. 20

Data on a cable television system is sent as packets that resemble well-known Ethernet packets. Each data packet can have up to several hundred data bytes preceded by several critical data bytes that comprise the frame's header. Among other things, the header includes a preamble, a frame 25 type, a source address and a destination address. A cyclic redundancy check or CRC is appended to the end of the packet for error detection.

Subscribers' computers are coupled to the cable via a cable modem. Each cable modem is assigned an address by 30 the head end when the modem is turned on. The modem's address effectively identifies the computer to which it is coupled and enables each subscriber to have individualized connections to the Internet.

Downstream data packets are those sent from the head 35 end of the cable system to the subscribers. Upstream packets are those sent from the subscribers to the head end. Once a data packet is physically transmitted onto the cable, all of the computers that are coupled to the cable can "see" the packets. By using Ethernet-like data frames however, the 40 cable modems of the system read only the downstream data packets that are addressed to it and ignore all other packets. When packets are sent upstream, the cable modem provides to the data packets, the address assigned to the cable modem. That address of the sending cable modem effectively iden- 45 tifies the subscriber from which the packets originated and in so doing, also provide an address to which responsive packets can be sent.

At the top or "head end" of a cable television system is a cable modem termination system or "CMTS," a function of 50 which is to provide complementary functionality to the subscriber's cable modems so as to enable data connectivity to a wide-area network like the Internet. The CMTS performs a variety of critical functions in a cable data system, including among other things, cable modem address assign- 55 ments and data packet address translation by which packets received by the CMTS from an Internet connection are addressed (directed) to the proper cable modem. A CMTS also functions as a switch by taking upstream packets from the cable modems and routing them toward their proper 60 destinations.

A CMTS includes one or more ports through which data is received and sent. Different ports are typically coupled to different pathways such that different ports lead to different logical destination addresses. By way of example, data from 65 a subscriber received at one port of the CMTS might need to be routed out a different port in order for the packets to

2

be routed to their destination (as determined by the destination address in the packet's header). In so doing, a CMTS receives upstream data packets, either from subscribers or Internet servers, and routes said packets to the port on which the intended recipient, or server for that recipient, resides as dictated by the packet's destination address. (A description of CMTS functionality can be found on the web site of CableLabs®, which is a non-profit research and development consortium of cable television operators within North and South America. The CableLabs® web site URL is: www.cablelabs.com.)

Most cable data systems now have more data server subscribers than there are channels that are available to be allocated to carrying data and as a result, cable data system subscribers share the data capacity of the frequency spectrum allocated to carry data. As the number of subscribers on a cable data system increases, bandwidth available to each subscriber (i.e., the service level) decreases. Stated alternatively, on a cable data system, responsiveness decreases as the number of users increases.

Those of ordinary skill in the art of computer networks in general, and cable data systems in particular, are familiar with the concept of "quality of service." Quality of service is concept of guaranteeing a throughput or a data transfer level. It is anticipated that next-generation cable data systems will offer customers guaranteed bandwidths (data rates) to customers willing to pay for it. Customers who pay for data rate guarantees will have their data packets routed through the cable data system ahead of lower-paying customers. The concept of being able to guarantee (and to be able to truly provide) different service or priority levels allows cable service providers the opportunity to provide better Internet service, as measured by connection bandwidth, to needy customers while charging a premium rate. Tiered service is expected to increase revenue for cable service providers.

One way of providing service levels is to organize Internet traffic into data flows. Data packets sent from the head end of the cable data system to a particular cable data system subscriber can be considered to be as one "data flow." Similarly, the upstream packets from the same customer, or some other customer can be considered to be another "data flow." By knowing the source and destination addresses of data packets on a cable data system, the data flows that they represent can be allotted different service (i.e., priority) levels by reading from a class of service list that a particular subscriber is entitled to.

Part of the job of a CMTS is to route incoming packets to their destinations, as determined by their packet destination addresses. In a CMTS, this usually requires that packets received at one port need to be switched out of the CMTS on a different port that is connected to a different physical or logical pathway leading to different logical destinations. In order to provide different service levels, data packets are stored in priority queues, which are known to those of skill in the networking art to be multi-element lists, each element of which is assigned a priority value or level that determines when the element is to be removed from the queue

The primary task behind any Ethernet frame-based scheduling algorithm is to examine the contents of the multiple queues that are aimed at a particular shared resource, e.g., a channel time-slot on the cable or a port of the CMTS, and then decide which queue should transmit the next Ethernet frame to that shared resource. A relative scheduling algorithm must make decisions regarding the relative treatment each of the queues based on the priority level associated with 3

that particular queue. In general, any scheduling algorithm must make trade-offs between fairness, delay, and implementation complexity.

Many theoretical models for scheduling have been presented in the literature. The Generalized Processor Sharing 5 ("GPS") algorithm (A. K. Parekh and R. G. Gallager, "A generalized processor sharing approach to flow control—the single node case," in IEEE INFOCOM '92, Vol. 2, pp. 915-924, May 1992.) is generally considered to be a fair scheduling algorithm, because it attempts to follow a fluid 10 flow model that transmits (and mixes) output traffic from various active queues at rates proportional to their assigned weights. Unfortunately, the ideal GPS algorithm requires that the arriving packets be broken down into infinitely divisible parts, which is not practical in a data system using 15 variable size packets like Ethernet frames and tends to complicate the algorithm's implementation.

One approach that attempts to approximate the GPS algorithm is the Weighted Fair Queuing ("WFQ") algorithm disclosed by A. Demers, S. Keshav, and S. Shenker, in 20 "Analysis and simulation of a fair queuing algorithm," published in Internetworking: Research and Experience, Vol. 1, No. 1, pp. 3-26, 1990). A limitation of this implementation is maintaining desirable latency performance of low-latency traffic, such as Voice-over-Internet Protocol 25 (VoIP) service, during periods of traffic bursts.

Another approach that attempts to provide fair service includes the CORR algorithm disclosed by D. Saha, S. Mukherjee, and S. Tripath, as "Carry-Over Round Robin: A simple cell scheduling mechanism for ATM networks" and 30 Deficit Round Robin Queuing (M. Shreedhar and G. Varghese, "Efficient fair queuing using deficit round robin"). However, these algorithms distort fairness when a particular queue is filled with very long packets while other queues are filled with very short packets. In such a situation, the queue 35 with the long frames will steal more than its fair share of bandwidth on the shared resource.

A method and apparatus that monitors the latency of every packet passing through the system and uses that information to dynamically adjust (in real time) the performance of the 40 scheduling algorithm would provide improved queuing of data flows in a cable modem termination system and quality of service level differentiation. Such a method would be an improvement over the prior art.

45

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows a cable data system including a representation of a cable modem termination system employing the queuing method and apparatus disclosed herein. 50

FIG. 2 shows a representation of two queues used to store data packets for each port in a CMTS that recognizes two different priority levels.

FIG. 3 shows an exemplary data packet.

FIG. 4 shows an exemplary data structure. 55

FIGS. 5A and 5B show the steps of a method to normalize data flows through several different queues.

DETAILED DESCRIPTION OF THE

PREFERRED EMBODIMENT 60

FIG. 1 shows a block diagram of a cable data system 10 that provides data flow queuing according to the preferred embodiment. The several computers 12 of the subscribers to the cable data system 10 are operatively coupled to the 65 coaxial cable 16 of a distribution network such as a CATV network (fiber optic cable in some embodiments) over which

4

data packets are transmitted to and received from a data system 18. For illustration purposes, the computers 12 in FIG. 1 are sub-divided into two groups: Group 44 and Group 46. In a preferred embodiment, the data system 18 is a cable modem termination system or "CMTS." The coaxial cable 16 and the CMTS 18 form at least part of a data distribution network. Together, the cable 16 and the CMTS enable the computers 12 to communicate with each other as well as with other computers of the Internet.

The CMTS 18 shown in FIG. 1 has four ports (denominated 0-3) into which packets are sent and received. Ports 0 and 1 are each coupled to respective groups of computers 12. Computers of Group 46 are coupled to port 0; computers of Group 44 are coupled to port 1. Ports 2 and 3 are coupled to different servers that each provide access to the Internet backbone.

One function of the CMTS 18 is to route data packets to their destinations. Data packets can be sent into one port and routed out of another port. In so doing, the CMTS 18 acts as a switch, routing data packets to their logical destinations. Port 0 is coupled to the computers in Group 46. Port 1 is coupled to the computers in Group 44. Ports 1 and 2 are coupled to Internet backbone gateways. By way of example, data packets from group 46 computers (received at port 0) can be switched to output ports 0, 1, 2 or 3. The CMTS 18 therefore not only enables group A computers 46 to communicate with the set of all computers 12, but also to communicate with any other computer linked to the Internet backbone accessible via the internet backbone 14.

In order to provide truly differentiated service levels to different data flows, the CMTS 18 needs to intelligently treat packets in higher-priority data flows with a greater level of precedence than packets in lower-priority data flows. This can result in higher-priority data flows getting preferential treatment in many different ways, ranging from offering them more bandwidth on a shared resource to offering them more rapid transport (lower delay and lower latency) when passing through the CMTS.

Some portions of the CMTS system design (such as the congestion control algorithms that run on each of the input ports) are entirely dedicated to the control of bandwidth associated with each of the data flows. In most commercially-available CMTS systems, the scheduling algorithms that determine the order by which packets are transmitted out of the system are also dedicated to the control of bandwidth associated with each of the data flows.

However, a novel scheduling algorithm that determines the order by which packets will be transmitted out of the CMTS can be used to control both the bandwidth and the delay associated with each data flow. In order to do so, high-priority packets can be written to a high priority queue, while low priority packets can be written to a low priority queue. The service level agreement parameters can be used to define the priority of a packet. The association between a service level agreement and a particular data flow is established when a subscriber's data flow is registered within the CMTS 18. This registering of the subscriber's service level agreement usually occurs when the subscriber's cable modem is first powered up and makes contact with the CMTS 18. When a packet passes through CMTS 18, a classification of the packet into a particular data flow must be performed. Since each data flow is associated with a subscriber who registered for a particular service level, each data flow is then associated with a unique priority. A table look-up (usually into a memory) using the unique priority as a key produces the service level agreement parameters that are associated with the priority. By association, these service