US8351435B2

US8351435B2 - Method for applying macro-controls onto IP networks using intelligent route indexing

Info

Publication number: US8351435B2
Application number: US13/033,694
Authority: US
Inventors: John Mulligan; Han Q. Nguyen; Samir Saad; Jacobus Van Der Merwe; Simon Zelingher
Original assignee: AT&T Intellectual Property I LP
Current assignee: Atlassian US Inc
Priority date: 2008-12-02
Filing date: 2011-02-24
Publication date: 2013-01-08
Also published as: US7916735B2; US20100135304A1; US20110145409A1; US9397924B2; US20130077470A1

Abstract

Systems and methods are described that manage routing information in an IP network using extensible indexing and use the indexing to control the network. The indexing and associated controls apply to any router within the routing domain.

Description

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No. 12/315,351, entitled “Method for Applying Macro-Controls onto IP Networks using Intelligent Route Indexing,” filed on Dec. 2, 2008, which issued on Mar. 29, 2011 as U.S. Pat. No. 7,916,735, the disclosure of which is incorporated herein by reference in its entirety.

BACKGROUND OF THE INVENTION

The invention relates generally to network engineering. More specifically, the invention relates to systems and methods that manage routing information in an IP network using extensible indexing to control the network.

Historically, Internet Protocol (IP) networks managed routes at an aggregate level or at very large subnetworks and could only apply broad-gauge network controls. Today, IP networks operate in data and control planes.

The data plane is the path which IP packets follow as they traverse an IP network. This includes routers and links between routers which carry data traffic. The data plane is organized around paths which are established with the goal of sending a data packet out of any given router on a path that will take it toward the interface(s) that are associated with the destination address of that packet. At any point in time, an IP network's data plane has an understanding of where to forward any packet it receives based primarily on the destination address of that packet.

The control plane is used to exchange routing messages. The routing messages enable the data plane to organize itself in an optimal manner. The control plane allows the data plane to be organized around paths which carry packets from a source to a destination. These paths are composed of numerous forwarding decisions made at each router based on routing messages received from the control plane.

Routers typically contain input ports, output ports, a specialized operating system, memory, and one or more processors. Distributed routers require memory for two purposes besides the operating system itself: storing a routing table which is used to perform routing decisions and program a forwarding table, and storing the forwarding table which is used when making forwarding decisions to forward packets out on particular interfaces. These two functions are critical to the operation of any router employed on an IP network.

For a network router to have complete routing information, every router would have to couple their control plane to every other router (peer with other routers). This is impracticable but in very small networks. This is due to the scalability limits of routers. To compensate, most IP networks use Route Reflectors (RRs) to manage the size of the control plane.

Route reflectors are routers which are dedicated to route exchange only, i.e., they do not pass any customer data packets. Route reflectors operate in the control plane. The route reflectors learn routes by exchanging routing messages from router to router which become route reflector clients. Route reflectors peer with all of the routers in a defined network domain so each has reachability within that defined domain and act as the only gateway for routing information exchange between network domains. FIG. 1 shows a network 101 having data links (solid lines) in the data plane 103 coupling routers R₁, R₂, R₃, R₄, R₅, R₆(collectively R) and control messages (broken lines) between the routers R and a route reflector RR in the control plane 105. In this network scheme all routing decisions are made based upon route updates exchanged using the route reflector RR.

The challenge for network operators is to improve routing efficiency. In order to achieve this, a system and method is needed that allows for the dynamic management of routes by applying network controls to subnets based on dynamic network conditions.

SUMMARY OF THE INVENTION

The inventors have discovered that it would be desirable to have systems and methods that manage routing information in an IP network using extensible indexing, and to leverage this indexing to control the associated IP network. The indexing and associated controls apply to any router within the routing domain.

One aspect of the invention provides a non-transitory computer readable medium having recorded thereon a computer program comprising code means for, when executed on a computer, instructing the computer to control steps in a method for managing route information in an IP network. Methods according to this aspect of the invention include assembling a route information database for the IP network, acquiring routes from the IP network, an indexing stage comprising associating with each acquired route an index having one or more fields, each route index field representing a network/service route attribute and is an alpha-numeric index field that classifies a route relative to the number of route index fields used, associating a network operating condition with a network/service route attribute, and entering applicable network/service route attributes in corresponding index fields for each route based on information associated with the route from the route information database, and a control stage comprising acquiring one or more network operating conditions from the IP network, associating the one or more acquired network operating conditions with corresponding network/service route attributes, comparing the one or more associated network/service route attributes with all route indices, and for routes having index fields matching the one or more associated network/service route attributes, executing a predetermined control action modifying those routes.

The details of one or more embodiments of the invention are set forth in the accompanying drawings and the description below. Other features, objects, and advantages of the invention will be apparent from the description and drawings, and from the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is an exemplary network showing data and control planes.

FIG. 2 is an exemplary system framework.

FIG. 3 is an exemplary method.

DETAILED DESCRIPTION

Embodiments of the invention will be described with reference to the accompanying drawing figures wherein like numbers represent like elements throughout. Before embodiments of the invention are explained in detail, it is to be understood that the invention is not limited in its application to the details of the examples set forth in the following description or illustrated in the figures. The invention is capable of other embodiments and of being practiced or carried out in a variety of applications and in various ways. Also, it is to be understood that the phraseology and terminology used herein is for the purpose of description and should not be regarded as limiting. The use of “including,” “comprising,” or “having,” and variations thereof herein is meant to encompass the items listed thereafter and equivalents thereof as well as additional items.

The terms “connected” and “coupled” are used broadly and encompass both direct and indirect connecting, and coupling. Further, “connected” and “coupled” are not restricted to physical or mechanical connections or couplings.

It should be noted that the invention is not limited to any particular software language described or that is implied in the figures. One of ordinary skill in the art will understand that a variety of alternative software languages may be used for implementation of the invention. It should also be understood that some of the components and items are illustrated and described as if they were hardware elements, as is common practice within the art. However, one of ordinary skill in the art, and based on a reading of this detailed description, would understand that, in at least one embodiment, components in the method and system may be implemented in software or hardware.

Embodiments of the invention provide methods, system frameworks, and a computer-usable medium storing computer-readable instructions that manage routing information in an IP network using extensible indexing and use the indexing to control the IP network. The invention may be implemented as a modular framework and deployed as software as an application program tangibly embodied on a program storage device. The application code for execution can reside on a plurality of different types of computer readable media known to those skilled in the art.

FIG. 1 shows a typical network 101 that is comprised of a plurality of routers R, including a network ingress (Provider Edge (PE)) router R₁, an egress (PE) router R₄and network forwarding (Provider (P)) routers R₂, R₃, R₅, R₆depending on the size of the network 101. P routers are strictly interior routers and do not terminate any external connections. Many P routers may be accessed between two PE routers. For example, traffic may enter a PE router in San Diego, Calif., and need to egress on a PE router in Boston, Mass., and traverse a route via a plurality of P routers from San Diego to Los Angeles to Chicago to New York to Boston. The network 101 provides connectivity between a source network and destination network.

A router is an electronic device and/or software that connect at least two networks, such as two Local Area Networks (LANs) or Wide Area Networks (WANs), and forwards packets between them. Each packet can traverse many routers, making many hops throughout the Internet as well as multiple routers within a large organization.

A next hop is the next router to which a packet is sent from any given router as it traverses a network from its source to its destination. In the event that the packet is at the final router in its journey, the next hop is the final destination. A hop is the trip that a packet takes from one router to another or from the final router to the destination. A packet, also referred to as a datagram, is a fundamental unit of data transmission on the Internet and other Transmission Control Protocol/Internet Protocol (TCP/IP) networks.

Routers forward data packets between networks using headers and forwarding tables to determine the best path to forward the packets. Routers work at the network layer of the TCP/IP model or Layer 3 of the Open Systems Interconnection (OSI) model. Routers also provide interconnectivity between like and unlike media. This is accomplished by examining the header of a data packet, and making a decision on the next hop to which it should be sent. Routers use preconfigured static routes, status of their hardware interfaces, and routing protocols to select the best route between any two subnets.

The next hop for any particular packet at any particular point in its journey is determined, for example, in the Internet by both the IP address of its destination as contained in its header and the routing table in the router at that point. An IP address is a unique numeric identifier for each computer or router on a TCP/IP network. A routing table is a database in a router that stores and frequently updates the IP addresses of reachable networks, called “routes” or “prefixes,” and the most efficient path to them.

The route announcement allows a router to learn the routing information to properly forward subsequently received packets to the same destination. Embodiments configure a complete network by analyzing new or pre-existing network architectures. Each network element may then be configured according to their operating system.

FIG. 2 shows an embodiment of a system framework 203 and FIG. 3 shows a method. The framework 203 which may be part of a network management server includes a network interface 205 coupled to a network 101 and configured to acquire network topology information, network reachability information, as well as network status information to perform network management services. The network interface 205 is coupled to a network information manager 207, a route store 209, a route index store 211 and a processor 213. The processor 213 is coupled to storage 215, memory 217 and I/O 219.

The framework 203 stores acquired network data into a data store 215. The framework 203 may be implemented as a computer including a processor 213, memory 217, storage devices 215, software and other components. The processor 213 is coupled to the network interface 205, I/O 219, storage 215 and memory 217 and controls the overall operation of the computer by executing instructions defining the configuration. The instructions may be stored in the storage device 215, for example, a magnetic disk, and loaded into the memory 217 when executing the configuration. Embodiments may be implemented as an application defined by the computer program instructions stored in the memory 217 and/or storage 215 and controlled by the processor 213 executing the computer program instructions. The computer also includes at least one network interface 205 coupled to and communicating with a network 101 such as shown in FIG. 1 to interrogate and receive network configuration data. The I/O 219 allows for user interaction with the computer via peripheral devices such as a display, a keyboard, a pointing device, and others.

Embodiments provide a system and method that index routes exchanged within an IP network. One or more values are assigned to fields of a multi-value index for every route to capture network/service route attributes associated with that route. Each route 209 and its associated index 211 are stored in the framework 203. For each route, macro-level controls may be applied based on a variety of network events. Examples of macro controls include traffic load shedding (removing a route which in turn removes the associated traffic load from the network), traffic redirection (changing a route and redirecting the associated traffic over an alternate path to its destination) and traffic shaping (changing a route and redirecting it into a path/device which limits the amount of bandwidth that can be consumed). Network events may be applied to each route to determine how the route should be changed and redistributed back to the routers in the network 101. Changes to the routes before they are redistributed include replacing a route with a “black hole” route (a pre-established route that drops traffic rather than forwards it to its destination), replacing a route with a route to an alternate path to its destination, or replacing a route with a route through a choke control/rate limiting network peripheral that can limit traffic volume. These modified routes represent examples of the macro-level control that embodiments employ.

Although these controls are macro in nature when viewed from the perspective of overall impact, embodiments enable these controls to be applied surgically on selected routers within the network 101. That is, the composition of the set of routers upon which the controls are applied can be highly selective and customized based on various network/service route attributes encoded in the route index fields of a given route. For example, one route index field may identify routes distributed to gateway routers to the public Internet. In response to an increase in denial of service traffic toward a particular address, the macro control may be to black hole all traffic toward this address that enters from these Internet gateway routers, thereby blocking traffic from off-network sources while allowing on-network traffic toward that address to flow. Additional network events that trigger these controls may include simple auditing, localized high traffic volumes, network outages, and various other network conditions that require mitigation.

Embodiments provide an index that is applied to all routes in a defined IP network as they are learned. As a route is learned via routing protocols such as Border Gateway Protocol (BGP) and Open Shortest Path First (OSPF), it is stored 209 along with an associated index 211. The index is derived upon learning the route by consulting stored information associated with the network. The associated information includes a customer subscription, application information associated with the access line from where a route is announced from, the place in the network topology where a route is announced from, and others.

The index network/service route attributes capture the service/network context of a route. Routes are more than a path to an interface that is reachable from a given router. The router that a route originates from has context associated with where in the network topology it is being announced from (a PE router vs. a gateway router vs. a P router). The specific service interface that a route is associated with has service context. It may be carrying special traffic associated, for example, with Government Emergency Telecommunications Service (GETS) which takes priority over other services in times of emergency. It may be carrying traffic which is delay tolerant. This network/service context can be derived by consulting stored information when the route is received. The route index fields are intended to capture this context so that later, if network events occur, macro-level controls may be applied based on each route's index.

The associated network information may be applied to each learned route. Route index fields may include whether the route is associated with a simplex or duplex path, whether data traffic associated with the route is Quality of Service (QoS) tolerant (delay, packet loss or jitter), specialized handling for source/destination pairs, the service type associated with the access line the route is coming from, the domain of the route (inter-network, network-wide or regional), a default value of the route and whether customer initiated route updates are allowed. Using a route's index, the framework 203 can implement macro-level network control based on external network conditions or events (triggers). The external triggers may include time of day, network capacity levels, network failures (planned and unplanned) and feedback mechanisms from customers (loading on their networks and/or servers hosting key applications).

The macro-level controls can be made customer-specific so that new services can be built upon this control methodology resulting in new revenue from existing IP customers. These controls can also be based on a wide variety of network conditions so that the associated IP network can dynamically adapt to a wide range of network conditions. This enables a closed-loop feedback mechanism so that network performance, for example, can be used to inform routing decisions.

Embodiments leverage network visibility into routing information to capture all of the routes associated with a network by employing two stages which are associated with the direction of information flow between the data and control planes. The first stage is route indexing which acquires the routing information that is announced into the control plane by the routers in the network 101 via standard routing protocols (BGP and OSPF) and associates one or more network/service route attributes to fields in each index based on stored network information. The second stage is intelligent macro-level control. Indexing takes advantage of the control plane messaging in the return direction from the framework 203 back to the routers in the network 101.

The framework 203 has full route visibility, acquiring routes received by route reflectors RRs (and/or via an embedded RR function within the framework 203) from network routers R and routes sent from route reflectors RRs to the network routers R, and assembles a network information database from information supplied from Operations Support Systems (OSSs) employed in the network (

steps

301, 303, 305). The framework 203 receives each announced route message and compares it to the stored information (steps 307, 309). This comparison is used to derive the network and service context associated with a route so that appropriate network/service route attributes can be entered in the route's index fields.

The comparison results in assigning one or more network/service route attributes to applicable route index fields (step 311). Both the route 209 and index 211 are stored (step 313).

The route index may be an alpha-numeric tag which classifies a route relative to a number of network/service route attributes entered in the index fields. Network/service route attributes comprise the route source geographic location such as the eastern region, central region, mountain region or western region of the US (E, C, M or W), the route destination geographic location interface associated with the route (E, C, M or W), the route priority such as if it is affiliated with GETS or standard Internet services (G or I), whether the route has an alternate path or if it is associated with a single path for forwarding to its destination (A or S), and whether an application associated with the route is delay intolerant such as Voice over IP (VoIP) or delay tolerant such as traditional IP traffic (Sensitive to delay (S) or Tolerant of delay (T)). Other variables may be employed for other network/service route attributes.

For example, one route may have index fields E/E/G/S/S to indicate that its source is the eastern region, its destination the eastern region, it is associated with GETS, it is the only single path to this destination and it is associated with a delay intolerant application. Embodiments allow for the development of specific index network/service route attributes for each route that the framework 203 can observe and stores all routes and indices. Since the indexing stage is part of the framework 203, it may be extended with new index fields with no impact on the underlying network routers R. Indexing is a real-time application and creates a new index for new routes as they are announced from the data plane. The indices for existing routes are updated if their status changes (step 315).

The intelligent control stage monitors both the network 101 and associated OSSs that are responsible for managing the network 101. These sources provide information including any outages that may exist in the network, capacity consumption levels for each of the nodes/links in the network and any pending changes that are about to be made to the network (Planned Maintenance (PM) activities that may take a router or link out of service in order to perform an upgrade). Intelligent control monitors network conditions, including utilization data to detect when different parts of the underlying IP network may be experiencing severe and/or sustained congestion which is resulting in significant packet loss in a localized part of the network. By monitoring the network 101 and associated management systems, predefined operating thresholds serve as setpoints that initiate predetermined functions into action (step 317). If an operating threshold is exceeded (

steps

319, 321, 323), macro controls may be executed on the network by applying common actions to routes having the same network/service route attribute in like route index fields (step 325).

A common action may be to shed network load by eliminating all routes of a given type. For example, all routes with destinations in the Eastern Region (E), but not associated with GETS and not associated with delay sensitive applications. This may be performed by selectively re-announcing modified routes (step 327) that match the index fields for the common action. Route modification would be to replace the route learned from the network 101 with a black hole route which drops that traffic where it enters the network (at the source). These modified routes would be announced into the data plane for routes matching the particular index network/service route attribute, while those that do not match will continue to be announced into the data plane in their unmodified state. In this manner, an entire class of traffic may be removed from the network at a macro level, while preserving other important routes in the same area (step 329).

Since in a typical IP network the overall volume of routes that are dealt with may be in the range of tens of thousands to millions, the indexing stage in conjunction with the intelligent control stage provide an effective means of executing control onto IP networks which were previously considered infeasible.

One or more embodiments of the present invention have been described. Nevertheless, it will be understood that various modifications may be made without departing from the spirit and scope of the invention. Accordingly, other embodiments are within the scope of the following claims.

Claims

1. A non-transitory computer readable medium having recorded thereon a computer program comprising code means for, when executed on a computer, instructing the computer to control steps in a method for managing route information in an IP network comprising:

assembling a route information database for the IP network;

acquiring routes from the IP network;

an indexing stage comprising:

associating with each acquired route an index having one or more fields, each route index field representing a network/service route attribute and is an alpha-numeric index field that classifies a route relative to the number of route index fields used;

associating a network operating condition with a network/service route attribute; and

entering applicable network/service route attributes in corresponding index fields for each route based on information associated with the route from the route information database; and

a control stage comprising:

acquiring one or more network operating conditions from the IP network;

associating the one or more acquired network operating conditions with corresponding network/service route attributes;

comparing the one or more associated network/service route attributes with all route indices; and

for routes having index fields matching the one or more associated network/service route attributes, executing a predetermined control action modifying those routes.

2. The non-transitory computer readable medium according to claim 1 wherein the network operating conditions further comprise dynamic network operating conditions.

3. The non-transitory computer readable medium according to claim 2 wherein associating the one or more acquired network operating conditions with corresponding network/service route attributes further comprises:

associating a predetermined operating threshold for each dynamic network operating condition; and

if a dynamic network operating condition exceeds its predetermined operating threshold, triggering a network/service route attribute corresponding to the exceeded dynamic network operating condition.

4. The non-transitory computer readable medium according to claim 1 wherein a route index further includes customer subscription, application information associated with the access line from where a route is announced from and the IP network location where a route is announced from.

5. The non-transitory computer readable medium according to claim 1 wherein acquired routes originate from route reflector messages.

6. The non-transitory computer readable medium according to claim 1 wherein one route index field is a route source geographic location.

7. The non-transitory computer readable medium according to claim 1 wherein one route index field is a route destination geographic location.

8. The non-transitory computer readable medium according to claim 1 wherein one route index field is a route priority.

9. The non-transitory computer readable medium according to claim 1 wherein one route index field is an alternate route path.

10. The non-transitory computer readable medium according to claim 1 wherein one route index field is if the route is associated with a single path.

11. The non-transitory computer readable medium according to claim 1 wherein one route index field is if the route is delay intolerant.

12. The non-transitory computer readable medium according to claim 1 wherein one route index field is if the route is delay tolerant.

13. The non-transitory computer readable medium according to claim 1 further comprising, if the route information database indicates a change in status for a route, updating the affected route's index accordingly.

14. The non-transitory computer readable medium according to claim 1 wherein a predetermined control action further comprises applying a predetermined logic based on the one or more acquired network/service route attributes.

15. The non-transitory computer readable medium according to claim 14 wherein one predetermined logic sheds network load based on the one or more acquired network/service route attributes.

16. The non-transitory computer readable medium according to claim 1 wherein assembling a route information database further comprises receiving border gateway protocol and open shortest path first messages.

17. The non-transitory computer readable medium according to claim 3 wherein one predetermined operating threshold is time of day.

18. The non-transitory computer readable medium according to claim 3 wherein one predetermined operating threshold is IP network capacity.

19. The non-transitory computer readable medium according to claim 3 wherein one predetermined operating threshold is IP network failure.

20. The non-transitory computer readable medium according to claim 3 wherein one predetermined operating threshold is customer feedback.