US8098584B2 - Optimization of traffic routing for data center services - Google Patents
Optimization of traffic routing for data center services Download PDFInfo
- Publication number
- US8098584B2 US8098584B2 US12/537,479 US53747909A US8098584B2 US 8098584 B2 US8098584 B2 US 8098584B2 US 53747909 A US53747909 A US 53747909A US 8098584 B2 US8098584 B2 US 8098584B2
- Authority
- US
- United States
- Prior art keywords
- paths
- path
- cost
- sites
- available
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active, expires
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L45/00—Routing or path finding of packets in data switching networks
- H04L45/70—Routing based on monitoring results
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/08—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
- H04L43/0823—Errors, e.g. transmission errors
- H04L43/0829—Packet loss
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/08—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
- H04L43/0823—Errors, e.g. transmission errors
- H04L43/0829—Packet loss
- H04L43/0841—Round trip packet loss
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/08—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
- H04L43/0876—Network utilisation, e.g. volume of load or congestion level
- H04L43/0882—Utilisation of link capacity
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/12—Network monitoring probes
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L45/00—Routing or path finding of packets in data switching networks
- H04L45/22—Alternate routing
Definitions
- the data request When a person accesses data on the Internet, that person's computer sends a data request to an online service provider (OSP).
- OSP online service provider
- the data request typically travels through several systems that are connected at various geographic points by routers and switches. Although there may be many different paths that connect the user's computer to the OSP, typically a default path and sourcing point is established for the connection.
- the default path and sourcing point have cost and performance characteristics associated with the transport of data between the user's computer and the OSP. In some instances, the default path and sourcing point is selected by a human.
- GFS global foundation service
- the data centers support high volumes of traffic to facilitate responding to many end user requests for data.
- the GFS typically experiences large costs to transport data across the wide area network between data centers and the Internet at peering sites.
- ISPs internet service providers
- the GFS has peering across multiple sites and backbone links that interconnect the GFS data centers.
- the GFS network has hundreds or more connections to neighbor ISPs.
- An illustrative technique may identify a plurality of available paths and source locations between an online service provider (OSP) and a destination prefix, and a traffic manager may measure a cost for each of the plurality of available paths and/or sources. In some instances, the traffic manager may also measure a performance for each of the available paths.
- OSP online service provider
- the cost is measured as a fee associated with a transmission of data, which is extracted based on a volume of data transfer through network systems such as routers and switches.
- Performance may be measured as a round trip time of data transfer for the available path, throughput, or packet loss.
- Each available path may be represented on a chart based on the cost and performance value.
- An optimization curve may be identified on the chart to indicate optimized combinations of cost and performance.
- the traffic manager may then select one, or multiple, available paths as preferred with a minimized cost for an instance of the performance value when compared to others of the plurality of available paths.
- the preferred path or source may be selected from an available set of paths and sources along the optimization curve.
- the traffic manager may rewrite a routing table or change a DNS response to implement the preferred path or source as the default path or source location between the OSP and the destination prefix.
- FIG. 1 is a schematic diagram of an illustrative environment where a server analyzes a network that connects a client device to an online service provider (OSP).
- OSP online service provider
- FIG. 2 is a flow diagram of an illustrative process of optimizing traffic routing for the OSPs.
- FIG. 3 is a schematic diagram of illustrative network paths between the client device and the OSP.
- FIG. 4 is a flow diagram of an illustrative process of measuring costs and performance of various paths between the OSP and the end user.
- FIG. 5 is a chart showing illustrative cost and latency for various paths between the OSP and the end user.
- FIG. 6 is a flow diagram of an illustrative process of optimizing cost and performance to select a path between the OSP and the end user.
- the global foundation service (GFS) team manages traffic between online service providers (OSPs), hosted by multiple data centers (DCs), and end users.
- a path used to transmit this traffic may be selected to reduce one or both of latency (i.e., increase performance and cost associated with the data transfer.
- Informed path selection may be conducted by quantifying costs of each possible path or source location between the OSP and the end user and measuring a round trip time (RTT) for each path to quantify latency.
- RTT round trip time
- An optimization process may be used to automatically select a preferred (optimal) path or source based on the measured costs and the RTT.
- FIG. 1 is a schematic diagram of an illustrative environment 100 that includes a network 102 that facilitates communications between various computing resources.
- the network 102 may be the Internet, or in some instances, a smaller network having multiple routers that enable path selection between endpoints of a data communication.
- the environment 100 includes online service providers (OSPs) 104 that provide data to end users 106 (or simply “users”) having computing devices 108 .
- the OSPs 104 may include search engines, e-commerce websites, social networking sites, news or entertainment sites, or anything else that provides online data for consumption by users 106 .
- the OSPs 104 may receive requests for data each day, sometimes ranging upwards of millions of requests. Each request may be fulfilled in as efficient a manner as possible to satisfy the requests of the users 106 .
- the OSPs 104 may be hosted on many different DCs, which may be distributed across a large geography or the globe.
- One reason OSPs 104 may use multiple DCs is to place the DCs closer to the end users 106 to minimize latency in fulfillment of data requests.
- DCs are often located near large populations (e.g., large urban areas) for this reason.
- a site selection of a DC often takes into consideration locations where resources (energy) are relatively cheaper than other locations. Because of deployment of multiple DCs for an OSP, requests may be fulfilled from different locations. For example, in the United States, a first DC may be located near the west coast in California while a second DC may be located near the east coast in Virginia.
- geographic proximity may identify the west coast DC as a cheaper and faster access point based purely off of proximity between the DC and the end user.
- geographic proximity may not be a clear indicator for an end user located in Chicago, thus routing decisions may be analyzed to determine one or more optimal paths to provide data from the OSPs (having multiple DCs) to the end user.
- the users 106 may connect with the online service provider via the computing device 108 .
- the computing device 108 may be a server, a desktop computer, a tablet, a mobile computer, a mobile telephone, a gaming console, or a music player, among other possible computing devices.
- ISPs internet service providers
- the request is transmitted through the network 102 and typically routed by one or more internet service providers (ISPs) through multiple routers from a local (relative to the end user 106 ) routing hub to an edge router, core router, or the like, and eventually to a DC where the data that is requested may be accessed to fulfill the user request.
- ISPs internet service providers
- traffic engineering server(s) may observe and interact with the network 102 .
- the TE server 110 may analyze data communications across the network 102 to determine one or more optimal paths for transmitting data between a particular one of the OSPs 104 and the user 106 via the computing device 108 .
- the TE server 110 may exchange data with various routers (core, edge, nexthub, etc.) that transmit data across various paths using different ISPs, and the like, to analyze possible paths for data transmission.
- the TE server 110 may modify routing tables for testing and measurement purposes and then output one or more optimum routing tables based on an analysis, which is further described below.
- TE server 110 may include one or more processors (“processors”) 112 and system memory 114 .
- processors processors
- system memory 114 may be volatile (such as RAM), non-volatile (such as ROM, flash memory, etc.) or some combination of the two.
- System memory 114 may include a traffic manager 116 to perform the functions described herein.
- the traffic manager 116 may include a path analyzer 118 to determine available paths between one of the OSPs 104 (via one or more DCs) and the end user 106 (via the computing device 108 ).
- the traffic manager 116 may include a measurement module 120 .
- the measurement module 120 may be used to measure the cost and performance of various paths between respective ones of the OSPs 104 and respective ones of the computing devices 108 . For example, the measurement module 120 may identify possible paths and then measure traffic along these paths to determine a cost (usage fee) and round trip time (RTT) (i.e., latency measurement) of data transmission using a particular path. The measurement module 120 may then output measured values to the optimization module 122 for further processing.
- RTT round trip time
- the traffic manager 116 may include the optimization module 122 .
- the optimization module 122 may receive the measured values from the measurement module 120 .
- the optimization module may determine a relationship between cost and latency (using the RTT) to enable an optimized selection of paths for routing data between one of the OSPs 104 and the computing device 108 .
- the optimization module 122 may identify a turning point that represents a path which is an ideal path for providing data the transfer at both a relatively low cost and low latency.
- FIG. 2 is a flow diagram of an illustrative process of optimizing traffic routing for the OSPs.
- the process 200 is illustrated as a collection of blocks in a logical flow graph, which represents a sequence of operations that can be implemented in hardware, software, or a combination thereof.
- the blocks represent computer-executable instructions that, when executed by one or more processors, cause the one or more processors to perform the recited operations.
- computer-executable instructions include routines, programs, objects, components, data structures, and the like that perform particular functions or implement particular abstract data types.
- the order in which the operations are described is not intended to be construed as a limitation, and any number of the described blocks can be combined in any order and/or in parallel to implement the process.
- Other processes described throughout this disclosure, in addition to process 200 shall be interpreted accordingly.
- the TE server 110 via the path analyzer 118 , may determine, or otherwise identify, available paths that may be used to transmit data between one of the OSPs 104 (via one or more DCs) and the end users 106 (via the computing devices 108 ).
- the path analyzer 118 may leverage various routing tables to determine possible paths, and then test the paths to determine whether they are viable paths for transmitting the data.
- the TE server 110 may measure costs and performance characteristics for each available path between one of the OSPs 104 and the end user 106 or a destination prefix.
- performance is correlated with latency. For example, improved performance may be achieved by reducing latency.
- the cost may be a fee (dollar amount) to transmit data, which is charged by an ISP.
- the performance may be a RTT of a data transfer along the path.
- the destination prefix may be a routing point where no alternative paths exist between the destination prefix and the end user 106 .
- the destination prefix is a small router destination that services a relatively small number of end users. Thus, the destination prefix may be used synonymously with “end user.”
- the TE server 110 via the optimization module 122 , may perform an optimization to identify one or more optimized paths for transmitting data between one of the OSPs 104 and the end user 106 .
- the optimization module 122 may plot available paths based on cost and latency to determine optimal paths for various cost and latency values.
- the TE server 110 via the traffic manager 116 , may route traffic based on an optimized path as determined from the operation 206 .
- the routing tables may be modified to direct data requests (traffic) along one or more paths that are identified as the most efficient (via a balance of cost and performance (latency)).
- FIG. 3 is a schematic diagram of illustrative network paths 300 between the client devices 108 and the OSPs 104 .
- the TE server 110 may be used to measure in real time, or near-real time, the performance and cost of routing traffic in a GFS network to a destination via any one of many alternative paths. However, the measurement may be conducted by the TE server 110 without actually redirecting the current ongoing traffic to that destination to that alternative path.
- Measurement of a GFS network 302 may be performed by initially determining a default path and then determining alternative paths for each prefix in the GFS network 302 .
- one or more border gateway protocols (BGP) daemons 304 are set up on the TE server 110 to peer with core routers 306 and edge routers 308 , where one BGPD is used per each autonomous system (AS). From peering BGP sessions, the TE server 110 extracts default BGP paths from DCs to each destination prefix (e.g., the end user 106 , destination prefix). In addition, full routing tables from the edge routers 308 are periodically copied (e.g., dumped) by the TE server 110 .
- BGP border gateway protocols
- the TE server 110 can extract an Internet protocol (IP) address of a next hop 310 (i.e., an ingress router of a neighbor ISP 312 ) and an AS path 314 of each alternate path to a destination prefix 316 (a prefix) from the edge router 308 .
- IP Internet protocol
- the TE server 110 may identify each available alternate path from a DC 318 (of the OSPs 104 ) to the destination prefix 316 of an end user 106 , denoted as Path (DC, egress, next hop), where egress is the edge router 308 , and the next hop 310 is a router of the neighbor ISP 312 that is willing to forward traffic from egress to the destination prefix 316 (effectively the end user).
- DC Internet protocol
- egress the edge router 308
- the next hop 310 is a router of the neighbor ISP 312 that is willing to forward traffic from egress to the destination prefix 316 (effectively the end user).
- the TE server 110 may measure a RTT for data transfer to each alternative path.
- the TE server 110 may inject prefixes (or subprefixes) of the destination prefix 316 with alternate paths into the network to enable measurement of a performance (RTT) of these alternate paths by sending probes destined to the prefixes.
- RTT performance
- BGP uses longest prefix matching, the ongoing traffic destined to those prefixes may be affected, i.e., diverted to the injected paths.
- the TE server 110 may inject /32 prefixes, each of which will affect a single IP.
- the above described injection process may be further explained by the following example.
- a default path is Path(DC,E 1 ,N 1 ), where E and N denote the edge router 308 and the next hop router 310 , respectively.
- the traffic manager 116 can inject multiple / 32 prefixes and measure the RTTs of all the injected alternate paths and the default path simultaneously by probing IP-1 and all the injected IP addresses from DC at substantially the same time.
- the TE server 110 may inject (n) /32 prefixes into each of the core routers 306 and 1 /32 prefix into each egress edge router 308 .
- FIG. 4 is a flow diagram of an illustrative process 400 of measuring costs and performance of various paths between one of the OSPs 104 and the end user 106 (or the destination prefix 316 ).
- the TE server 110 and more specifically the path analyzer 118 and the measurement module 120 of the traffic manager 116 may perform some or all of the operations of the process 400 .
- the measurement module 120 may select an interval to perform a measurement of a network, or portion thereof.
- the measurement module 120 may measure cost and latency of the BGP network at intervals such as (and without limitation) daily, weekly, hourly, or more or less frequently.
- the interval may be implemented using a fixed or random schedule.
- the path analyzer 118 may determine the default path by referencing a routing table from the edge router 308 , or other routing location.
- the path analyzer 118 may determine available paths between one of the OSPs 104 and the end user 106 (or the destination prefix 316 ). Again, the path analyzer may extract this information from a router table.
- the traffic manager 116 may select a path for a measurement.
- each of the available paths may be measured via a loop which begins at the operation 408 .
- the measurement module 120 may measure performance (or latency) of the path.
- the performance may be measured as the RTT.
- the RTT may be conducted via data-plane probing. The measurements can then be used to compute the aggregate performance of a given path strategy.
- the measurement module 120 may send a Transmission Control Protocol acknowledgement (TCP ACK) packet to a random high port of an IP address of the end user (in the destination prefix 316 ). This may trigger the destination to return a TCP reset (RST) packet, which can be used to measure the RTT at 410 ( 2 ).
- TCP ACK Transmission Control Protocol acknowledgement
- RST TCP reset
- the RTT measured by TCP ACK/RST may be more accurate to the latency experienced by applications because most OSP applications are TCP-based and ICMP packets may be forwarded in the network using a lower priority setting.
- ICMP Internet Control Message Protocol
- a randomly selected IP address in a prefix may not respond to a probe from the traffic manager 116 .
- the traffic manager 116 may use known techniques that prioritize and order probes to a small subset of IP addresses in a prefix that are likely to respond rather than scanning all the IP addresses or only checking the first IP address in a prefix.
- the measurement module 120 may measure a cost (fee) associated with transmitting data across the path.
- a cost (fee) associated with transmitting data across the path.
- One challenge in optimizing traffic cost is that the actual traffic cost is calculated based on a P95 link utilization over a certain billing period (e.g., a month) while an online TE strategy usually performs optimization for a short internal (e.g., scale of a seconds, minutes, etc), at a time.
- the TE server 110 may use a greedy algorithm to optimize a short-term cost as a meaningful way to compare alternative strategies under a same traffic demand during a predetermined period of time.
- the measurement module 120 may estimate traffic volume.
- the TE server 110 may have to estimate the traffic volume to each end user (prefix) because external links are charged by traffic volume to create a cost.
- the measurement module 120 may estimate traffic volume over a short interval (e.g., seconds, minutes, etc.) by collecting a net flow of data from all the core routers in the network because the data requests (traffic) flows to/from the DCs will traverse at least one core router. Thus, the traffic may be estimated by tallying the net flow.
- the measurement module 120 may compute the traffic cost. For each link, the measurement module 120 may aggregate the traffic volume to all prefixes that traverse that link in a TE strategy. The total traffic cost is the sum of the cost incurred on all the links.
- the measurement module 120 may determine whether another path needs to be measured to determine one of the latency or the cost. When another path is to be measured, the process 400 may loop back to the operation 408 where another path may be measured by the measurement module.
- the measurement module may publish results when all of the paths have been measured via the loop between the operation 408 and 412 .
- the results are published for each interval of the process 400 , as determined by the operation 402 .
- the published results are used by the optimization module 122 as discussed below.
- FIG. 5 is a chart 500 showing illustrative cost and latency for various paths between the OSP and the end user.
- the chart 500 shows an optimization data points for each prefix, which may service multiple end users. Because a network may use many alternative routes to transmit data between a DC and the prefix, the chart 500 includes many data points 502 , which are plotted with respect to cost 504 and performance 506 in RRT.
- the chart 500 may be used to create an assignment for each DC to a prefix via a path (e.g., Path(DC, C 3 , E 2 , N 2 ) of FIG. 3 , etc.).
- Each assignment if deployed, may result in a certain level of aggregate application performance (latency) and incur a certain amount of traffic costs.
- the goal of the assignment is to select a path that conforms to a strategy of minimizing traffic costs and latency (increasing performance).
- the path performance and traffic demands may evolve over time in the network, thus the assignments may be continually adapted in order to optimize performance and cost in the long term.
- online traffic engineering works by dividing the time into fixed time intervals, and at the end of each interval calculating an optimal assignment for routing the application traffic in the next interval.
- the interval may be the same as the interval selected at 402 in the process 400 .
- the data points 502 may be plotted from the published results at the operation 416 , which include latency information from the operation 410 and cost information from the operation 412 .
- each data point represents a path assignment and possible strategy for routing traffic.
- not all strategies are worth exploring because some are not competitive in cost/latency when compared to other available strategies represented by the data points 502 .
- only the optimized data points 508 that form the lower-left boundary, and thus an optimization curve 510 of all data points need to be considered because these optimized data points include either lower cost or lower latency than similarly situated data points.
- a strategy is considered optimal if there is no other strategy that has both lower latency and lower cost for a given cost value. Effectively, the lower-left boundary connecting all the optimal strategies forms the optimization curve on the plane.
- the chart 500 may also enable identification of a turning point 512 .
- a small increase in RTT results in an initial sharp drop in the cost.
- a certain point, defined as the turning point 512 a small reduction in cost results in sharp increase in latency.
- This location may indicate that a preferred strategy, depending on predefined preferences, should be selected near the turning point on the optimization curve 510 .
- This strategy may be defined as a BalanceCostPerf strategy.
- Alternative strategies may see a minimized cost (e.g., LowestCost strategy), seek a minimized round trip time (e.g., LowestLatency strategy), or weigh cost and latency to determine a strategy between these two extremes.
- the chart 500 may include a default point 514 that represents the cost and latency of a default path prior to a new path selection based on a path corresponding to the optimized data points 508 that are located along the optimization curve 510 .
- FIG. 6 is a flow diagram of an illustrative process 600 of optimizing cost and performance to select a path between the OSP and the end user (or the destination prefix).
- the process 600 will be described with reference to the chart 500 of FIG. 5 .
- the TE server 110 and more specifically the optimization module 122 of the traffic manager 116 , may perform some or all of the operations of the process 600 .
- the traffic manager 116 may select an interval to perform an optimization.
- the interval may be similar or identical to the interval selection at the operation 402 of the process 400 .
- the optimization module 122 may locate a range R of RTTs for any strategy on the optimization curve 510 .
- At least two hypothetical strategies may be used to find the minimum and maximum RTT within R.
- a first strategy e.g., HyperBestPerf
- a second strategy e.g., HyperLowestCost
- the optimization module 122 may ignore the link capacity constraint in an assignment process, and hence the two strategies may only be theoretical (not capable of actual implementation for assignment of a path). However, the first and second strategies provide reasonable lower and upper bounds of R.
- the optimization module 122 may iteratively “sweep” between the minimum RTT to the maximum RTT within R with reasonably small increments (e.g., a few milliseconds, etc.) and find the corresponding optimized data point 508 at each increment. A connection of the optimized data points 508 with line segments approximates the optimization curve 510 . Given a RTT within R, the optimization module 122 can compute the cost of a corresponding optimal strategy on the curve, which may be solved as a linear programming problem.
- Equation 1 represents the capacity constraint for each link and ⁇ is a constant ( ⁇ 1) that reserves some spare capacity to accommodate potential traffic variations as follows.
- Equation 3 represents the RTT constraint as follows.
- the optimization module 122 may optionally determine the turning point 512 .
- the turning point may be selected at a point along the optimization curve 510 at which the incremental decrease in cost becomes negligible and the increasing latency becomes substantial, such that the optimization curve (when traced from the left with a low RTT) exhibits an apparent “turn” or corner.
- the turning point may be selected by an algorithm or a human and may be tailored based on preferred cost or latency requirements. However, this is just one possible solution of many for an optimal path.
- the optimization module 122 may adjust (shift) the optimization curve, which is a fractional solution, to accommodate an integer solution, which enables a single path to accommodate all of the predicted traffic along a selected path. This adjustment is explained more fully below.
- the chart 500 shows fractional solutions to optimal BalanceCostPerf strategies using linear programming.
- the traffic to a destination prefix may be required or desired to flow along a single path at a time, and hence variables fkij must be either 0 or 1. This is referred herein as an integer solution.
- the traffic manager 116 may use a heuristic to convert the fractional solution to an optimal strategy with an integer solution.
- the heuristic may use the fractional solution as a starting point to find the integer solution.
- a heuristic starts with the fractional solution and then sorts all of the destination prefixes (d k ) in the ascending order based on Equation 5.
- avail k ⁇ j ⁇ availCap j ⁇ ( d k ) vol k Equ . ⁇ 5
- vol k and availCap j (d k ) denote the traffic volume to dk and the available capacity at link j for carrying traffic to d k , respectively.
- AvailCap j (d k ) is zero if link j cannot route traffic to d k .
- the optimization module 122 then assigns the prefixes to links in a sorted order. In this way, it provides high priority to prefixes with large traffic volume and small available capacity.
- the optimization module 122 randomly assigns all the traffic to d k to one of the paths path(dc i , link j ) that has enough residual capacity for d k with a probability proportional to f kij .
- Another version of the heuristic runs linear programming in a recursive manner. After assigning a prefix, the optimization module 122 updates the capacity constraint 1 and the RTT constraint 2 by removing that prefix from the linear programming, and re-computes all the fkij's for the remaining prefixes in a new linear programming. The optimization module 122 then assigns the next prefix by repeating the simple heuristic above (but again assigning only one prefix). This may help to prevent the final integer solution from deviating too far away from the original fraction solution. The heuristic then continues to a next iteration. This refined version, however, may be more costly than a simple version that is described above.
Abstract
Description
min cost=Σjpricej×ΣkΣi(f kij×volk)) Equ. 1
ΣkΣi(f kij×volk)≦μ×capj Equ. 2
Equation 3 represents the RTT constraint as follows.
ΣkΣiΣj(f kij×volk×rttkij)≦Σkvolk ×wRTT Equ. 3
Equation 4 ensures all the traffic to a destination must be successfully carried as follows.
ΣiΣj f kij=1 Equ. 4
The objective is to find an appropriate set of variables fkij to minimize the total cost.
where volk and availCapj(dk) denote the traffic volume to dk and the available capacity at linkj for carrying traffic to dk, respectively. AvailCapj(dk) is zero if linkj cannot route traffic to dk. The
Claims (15)
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US12/537,479 US8098584B2 (en) | 2009-08-07 | 2009-08-07 | Optimization of traffic routing for data center services |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US12/537,479 US8098584B2 (en) | 2009-08-07 | 2009-08-07 | Optimization of traffic routing for data center services |
Publications (2)
Publication Number | Publication Date |
---|---|
US20110032833A1 US20110032833A1 (en) | 2011-02-10 |
US8098584B2 true US8098584B2 (en) | 2012-01-17 |
Family
ID=43534776
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US12/537,479 Active 2030-02-27 US8098584B2 (en) | 2009-08-07 | 2009-08-07 | Optimization of traffic routing for data center services |
Country Status (1)
Country | Link |
---|---|
US (1) | US8098584B2 (en) |
Cited By (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US9634938B2 (en) | 2013-11-05 | 2017-04-25 | International Business Machines Corporation | Adaptive scheduling of data flows in data center networks for efficient resource utilization |
WO2018208570A1 (en) * | 2017-05-09 | 2018-11-15 | Cisco Technology, Inc. | Routing network traffic based on dns |
WO2020088684A1 (en) * | 2018-11-02 | 2020-05-07 | 华为技术有限公司 | Method and network device for routing processing |
US11070645B1 (en) * | 2018-12-14 | 2021-07-20 | Amazon Technologies, Inc. | Flexible scheduling of data transfers between computing infrastructure collections for efficient resource utilization |
US11456956B2 (en) * | 2014-11-20 | 2022-09-27 | Verizon Patent And Licensing Inc. | Systems and methods for dynamic connection paths for devices connected to computer networks |
Families Citing this family (52)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US8743885B2 (en) | 2011-05-03 | 2014-06-03 | Cisco Technology, Inc. | Mobile service routing in a network environment |
US8949832B2 (en) * | 2011-08-29 | 2015-02-03 | Novell, Inc. | Techniques for workload toxic mapping |
EP2804351A4 (en) * | 2012-01-11 | 2015-07-22 | Nec Corp | Computer system, controller, switch, communication method, and recording medium in which network management program has been stored |
AT512805A1 (en) | 2012-04-19 | 2013-11-15 | Fts Computertechnik Gmbh | Self-organizing method for building deterministic routes in a large computer network |
US20130332619A1 (en) * | 2012-06-06 | 2013-12-12 | Futurewei Technologies, Inc. | Method of Seamless Integration and Independent Evolution of Information-Centric Networking via Software Defined Networking |
AT513314A1 (en) | 2012-06-25 | 2014-03-15 | Fts Computertechnik Gmbh | Method for building optimal timed paths in a large computer network |
US9172743B2 (en) * | 2012-12-31 | 2015-10-27 | Futurewei Technologies, Inc. | Scalable storage systems with longest prefix matching switches |
US9042235B1 (en) * | 2013-03-15 | 2015-05-26 | Genband Us Llc | Determining peer-to-peer communication paths between service providers |
US9794379B2 (en) | 2013-04-26 | 2017-10-17 | Cisco Technology, Inc. | High-efficiency service chaining with agentless service nodes |
WO2015016919A1 (en) * | 2013-07-31 | 2015-02-05 | Adaptive Spectrum And Signal Alignment, Inc. | Method and apparatus for continuous access network monitoring and packet loss estimation |
US9621453B1 (en) * | 2013-08-29 | 2017-04-11 | Google Inc. | Path discovery in multipath networks |
US9525638B2 (en) * | 2013-10-15 | 2016-12-20 | Internap Corporation | Routing system for internet traffic |
US20150156057A1 (en) * | 2013-12-02 | 2015-06-04 | John C. Leung | Executable-Based Platform Selection |
US9887914B2 (en) * | 2014-02-04 | 2018-02-06 | Fastly, Inc. | Communication path selection for content delivery |
WO2015136870A1 (en) * | 2014-03-10 | 2015-09-17 | 日本電気株式会社 | Communication route control device, communication route control system, storage medium storing communication route control program, and communication route control method |
US9553797B2 (en) * | 2014-03-12 | 2017-01-24 | International Business Machines Corporation | Message path selection within a network |
US10417025B2 (en) * | 2014-11-18 | 2019-09-17 | Cisco Technology, Inc. | System and method to chain distributed applications in a network environment |
US9660909B2 (en) | 2014-12-11 | 2017-05-23 | Cisco Technology, Inc. | Network service header metadata for load balancing |
USRE48131E1 (en) | 2014-12-11 | 2020-07-28 | Cisco Technology, Inc. | Metadata augmentation in a service function chain |
JP6733655B2 (en) * | 2015-03-06 | 2020-08-05 | 日本電気株式会社 | Network control device, network control method, and program |
US9762402B2 (en) | 2015-05-20 | 2017-09-12 | Cisco Technology, Inc. | System and method to facilitate the assignment of service functions for service chains in a network environment |
US9929949B2 (en) * | 2015-06-29 | 2018-03-27 | Google Llc | Systems and methods for inferring network topology and path metrics in wide area networks |
US10721098B2 (en) | 2015-08-28 | 2020-07-21 | Vmware, Inc. | Optimizing connectivity between data centers in a hybrid cloud computing system |
US10721161B2 (en) | 2015-08-28 | 2020-07-21 | Vmware, Inc. | Data center WAN aggregation to optimize hybrid cloud connectivity |
US10547540B2 (en) * | 2015-08-29 | 2020-01-28 | Vmware, Inc. | Routing optimization for inter-cloud connectivity |
US11044203B2 (en) | 2016-01-19 | 2021-06-22 | Cisco Technology, Inc. | System and method for hosting mobile packet core and value-added services using a software defined network and service chains |
US10187306B2 (en) | 2016-03-24 | 2019-01-22 | Cisco Technology, Inc. | System and method for improved service chaining |
US10931793B2 (en) | 2016-04-26 | 2021-02-23 | Cisco Technology, Inc. | System and method for automated rendering of service chaining |
US10419550B2 (en) | 2016-07-06 | 2019-09-17 | Cisco Technology, Inc. | Automatic service function validation in a virtual network environment |
US10320664B2 (en) | 2016-07-21 | 2019-06-11 | Cisco Technology, Inc. | Cloud overlay for operations administration and management |
US10218616B2 (en) | 2016-07-21 | 2019-02-26 | Cisco Technology, Inc. | Link selection for communication with a service function cluster |
US10225270B2 (en) | 2016-08-02 | 2019-03-05 | Cisco Technology, Inc. | Steering of cloned traffic in a service function chain |
US10218593B2 (en) | 2016-08-23 | 2019-02-26 | Cisco Technology, Inc. | Identifying sources of packet drops in a service function chain environment |
US10361969B2 (en) | 2016-08-30 | 2019-07-23 | Cisco Technology, Inc. | System and method for managing chained services in a network environment |
US10038792B2 (en) | 2016-11-02 | 2018-07-31 | Microsoft Technology Licensing, Llc | Data center centroid metric calculations for PSTN services |
US11050706B2 (en) * | 2017-03-22 | 2021-06-29 | Salesforce.Com, Inc. | Automated autonomous system based DNS steering |
US10225187B2 (en) | 2017-03-22 | 2019-03-05 | Cisco Technology, Inc. | System and method for providing a bit indexed service chain |
US10333855B2 (en) | 2017-04-19 | 2019-06-25 | Cisco Technology, Inc. | Latency reduction in service function paths |
US10554689B2 (en) | 2017-04-28 | 2020-02-04 | Cisco Technology, Inc. | Secure communication session resumption in a service function chain |
US10511507B2 (en) * | 2017-05-09 | 2019-12-17 | Cisco Technology, Inc. | Routing network traffic based on whether an application associated with traffic is a rerouting application as defined by a policy and whether a second path ranking exceeds a first path ranking |
US10735275B2 (en) | 2017-06-16 | 2020-08-04 | Cisco Technology, Inc. | Releasing and retaining resources for use in a NFV environment |
US10798187B2 (en) | 2017-06-19 | 2020-10-06 | Cisco Technology, Inc. | Secure service chaining |
US10397271B2 (en) | 2017-07-11 | 2019-08-27 | Cisco Technology, Inc. | Distributed denial of service mitigation for web conferencing |
US10673698B2 (en) | 2017-07-21 | 2020-06-02 | Cisco Technology, Inc. | Service function chain optimization using live testing |
US11063856B2 (en) | 2017-08-24 | 2021-07-13 | Cisco Technology, Inc. | Virtual network function monitoring in a network function virtualization deployment |
US10791065B2 (en) | 2017-09-19 | 2020-09-29 | Cisco Technology, Inc. | Systems and methods for providing container attributes as part of OAM techniques |
US11018981B2 (en) | 2017-10-13 | 2021-05-25 | Cisco Technology, Inc. | System and method for replication container performance and policy validation using real time network traffic |
US10541893B2 (en) | 2017-10-25 | 2020-01-21 | Cisco Technology, Inc. | System and method for obtaining micro-service telemetry data |
US10516601B2 (en) * | 2018-01-19 | 2019-12-24 | Citrix Systems, Inc. | Method for prioritization of internet traffic by finding appropriate internet exit points |
US10666612B2 (en) | 2018-06-06 | 2020-05-26 | Cisco Technology, Inc. | Service chains for inter-cloud traffic |
WO2020087002A1 (en) * | 2018-10-26 | 2020-04-30 | Akamai Technologies, Inc. | Dns everywhere |
US11115309B1 (en) * | 2019-06-03 | 2021-09-07 | Amazon Technologies, Inc. | External network route advertisement validation |
Citations (9)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5463686A (en) | 1992-04-17 | 1995-10-31 | France Telecom | Communication routing method with income optimization for telecommunication networks |
US20020042812A1 (en) * | 2000-05-12 | 2002-04-11 | Spencer Mark A. | Apparatus, method and system for voice over a network having provider selection and disputeless billing |
US20030133443A1 (en) * | 2001-11-02 | 2003-07-17 | Netvmg, Inc. | Passive route control of data networks |
US6683945B1 (en) | 2000-12-28 | 2004-01-27 | Bellsouth Intellectual Property Corporation | Routing data based on comparative income values |
US20040042402A1 (en) * | 1997-02-11 | 2004-03-04 | Claude Galand | Method and system for a local and fast non-disruptive path switching in high speed packet switching networks |
US6981055B1 (en) * | 2000-08-22 | 2005-12-27 | Internap Network Services Corporation | Method and system for optimizing routing through multiple available internet route providers |
US7006448B1 (en) | 1999-10-01 | 2006-02-28 | Lucent Technologies Inc. | System and method for measuring network round trip time by monitoring fast-response operations |
US7085837B2 (en) | 2001-12-04 | 2006-08-01 | International Business Machines Corporation | Dynamic resource allocation using known future benefits |
US20100172249A1 (en) * | 2005-11-02 | 2010-07-08 | Hang Liu | Method for Determining a Route in a Wireless Mesh Network Using a Metric Based On Radio and Traffic Load |
-
2009
- 2009-08-07 US US12/537,479 patent/US8098584B2/en active Active
Patent Citations (9)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5463686A (en) | 1992-04-17 | 1995-10-31 | France Telecom | Communication routing method with income optimization for telecommunication networks |
US20040042402A1 (en) * | 1997-02-11 | 2004-03-04 | Claude Galand | Method and system for a local and fast non-disruptive path switching in high speed packet switching networks |
US7006448B1 (en) | 1999-10-01 | 2006-02-28 | Lucent Technologies Inc. | System and method for measuring network round trip time by monitoring fast-response operations |
US20020042812A1 (en) * | 2000-05-12 | 2002-04-11 | Spencer Mark A. | Apparatus, method and system for voice over a network having provider selection and disputeless billing |
US6981055B1 (en) * | 2000-08-22 | 2005-12-27 | Internap Network Services Corporation | Method and system for optimizing routing through multiple available internet route providers |
US6683945B1 (en) | 2000-12-28 | 2004-01-27 | Bellsouth Intellectual Property Corporation | Routing data based on comparative income values |
US20030133443A1 (en) * | 2001-11-02 | 2003-07-17 | Netvmg, Inc. | Passive route control of data networks |
US7085837B2 (en) | 2001-12-04 | 2006-08-01 | International Business Machines Corporation | Dynamic resource allocation using known future benefits |
US20100172249A1 (en) * | 2005-11-02 | 2010-07-08 | Hang Liu | Method for Determining a Route in a Wireless Mesh Network Using a Metric Based On Radio and Traffic Load |
Non-Patent Citations (22)
Title |
---|
"Open IP Networks for Innovation and Profit", Juniper Networks, Feb. 2009, retrieved from the internet at http://www.juniper.net/us/en/local/pdf/whitepapers/2000298-en.pdf, 13 pgs. |
Agarwal, et al., "Measuring the Shared Fate of IGP Engineering and Interdomain Traffic", Proceedings 13th IEEE Intl Conf on Network Protocols, 2005, retrieved from the internet at http://www.ieee-icnp.org/2005/Papers/21-sagarwal-Interdomain.pdf, 10 pgs. |
Akella, et al., "A Comparison of Overlay Routing and Multihoming Route Control", SIGCOMM'04, Aug. 30-Sep. 3, 2004, Portland, OR, retrieved from the internet at http://conferences.sigcomm.org/sigcomm/2004/papers/p397-akella1.pdf, 14 pgs. |
Akella, et al., "A Measurement-Based Analysis of Multihoming", SIGCOMM'03, Aug. 2003, retrieved from the internet at http://www.cs.princeton.edu/~jrex/teaching/spring2005/reading/akella03.pdf, pp. 353-364. |
Akella, et al., "A Measurement-Based Analysis of Multihoming", SIGCOMM'03, Aug. 2003, retrieved from the internet at http://www.cs.princeton.edu/˜jrex/teaching/spring2005/reading/akella03.pdf, pp. 353-364. |
Akella, et al., "Multihoming Performance Benefits: An Experimental Evaluation of Practical Enterprise Strategies", USENIX Annual Technical Conf 2004, Boston, MA, retrieved from the internet at http://www.cs.cmu.edu/~aditya/papers/usenix04.pdf, 14 pgs. |
Akella, et al., "Multihoming Performance Benefits: An Experimental Evaluation of Practical Enterprise Strategies", USENIX Annual Technical Conf 2004, Boston, MA, retrieved from the internet at http://www.cs.cmu.edu/˜aditya/papers/usenix04.pdf, 14 pgs. |
Dhamdhere, et al., "ISP and Egress Path Selection for Multihomed Networks", IEEE 2006, retrieved from the internet at http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&isnumber=&arnumber=4146933, 12 pgs. |
Goldenberg, et al., "Optimizing Cost and Performance for Multihoming", SIGCOMM'04, Aug. 30-Sep. 3, 2004, Portland, OR, retrieved from the internet at http://www.cs.princeton.edu/~jrex/teaching/spring2005/reading/goldenberg04.pdf, 14 pgs. |
Goldenberg, et al., "Optimizing Cost and Performance for Multihoming", SIGCOMM'04, Aug. 30-Sep. 3, 2004, Portland, OR, retrieved from the internet at http://www.cs.princeton.edu/˜jrex/teaching/spring2005/reading/goldenberg04.pdf, 14 pgs. |
He, et al., "Toward Internet-Wide Multipath Routing", IEEE Network, Mar./Apr. 2008, retrieved from the internet at http://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=04476066, pp. 16-21. |
Kallitsis, et al., "Pricing and Measurement-based Optimal Resource Allocation in Next Generation Network Services", Globecom Workshops, Nov. 2007 IEEE, retrieved from the internet at http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&isnumber=4437775&arnumber=4437788, 6 pgs. |
Kandula, et al., "Walking the Tightrope: Responsive Yet Stable Traffic Engineering", SIGCOMM'05, Aug. 21-26, 2005, Philadelphia, PA, retrieved from the internet at http://nms.lcs.mit.edu/papers/sigcomm05-kandula.pdf, 12 pgs. |
Katho, et al., "The Concept and Model of 4 Dimensional Traffic Engineering", 2006 IEEE, retrieved from the internet at http://ieeexplore.ieee.org/Xplore/login.jsp?url=http%3A%2F%2Fieeexplore.ieee.org%2Fiel5%2F11125%2F35640%2F01690501.pdf%3Ftp%3D%26isnumber%3D%26arnumber%3D1690501&authDecision=-203, 6 pgs. |
Liu, et al, "Multihoming Route Control among a Group of Multihomed Stub Networks", Computer Communications, vol. 30, Issue 17, Nov. 2007, retrieved from the internet at http://dropzone.tamu.edu/techpubs/2006/TAMU-ECE-2006-01.ps, 12 pgs. |
Liu, et al., "Route Optimization among a Group of Multihomed Stub Networks", IEEE Globecom 2005, retrieved from the internet at http://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=01577766, pp. 889-893. |
Verkaik, et al., "Wresting Control from BGP: Scalable Fine-grained Route Control", USENIX Annual Technicval Conf Jun. 2007, retrieved from the internet at http://cseweb.ucsd.edu/~snoeren/papers/rcp-usenix07.pdf, 14 pgs. |
Verkaik, et al., "Wresting Control from BGP: Scalable Fine-grained Route Control", USENIX Annual Technicval Conf Jun. 2007, retrieved from the internet at http://cseweb.ucsd.edu/˜snoeren/papers/rcp-usenix07.pdf, 14 pgs. |
Wang, et al, "COPE: Traffic Engineering in Dynamic Networks", SIGCOMM'06, Sep. 11-15, 2006, Pisa, Italy, retrieved from the internet at http://delivery.acm.org/10.1145/1160000/1159926/p99-wang.pdf?key1=1159926&key2=9110285421&coll=GUIDE&dl=GUIDE&CFID=41118936&CFTOKEN=59385253, pp. 99-110. |
Xie, et al., "P4P: Provider Portal for Applications", SIGCOMM'08, Aug. 17-22, 2008, Seattle, WA, retrieved from the internet at http://cs-www.cs.yale.edu/homes/yry/projects/p4p/p4p-sigcomm08.pdf, 12 pgs. |
Zhang, et al., "On Optimal Routing with Multiple Traffic Matrices", 2005 IEEE, retrieved from the internet at http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&isnumber=&arnumber=1497927, pp. 607-618. |
Zhang, et al., "Optimal Routing with Mutliple Traffic Matrices Tradeoff between Average and Worst Case Performance", Proceedings of the 13th IEEE International Conference on Network Protocols (ICNP'05), 2005, retrieved from the internet at http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&isnumber=32971&arnumber=1544622, 10 pgs. |
Cited By (11)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US9634938B2 (en) | 2013-11-05 | 2017-04-25 | International Business Machines Corporation | Adaptive scheduling of data flows in data center networks for efficient resource utilization |
US11456956B2 (en) * | 2014-11-20 | 2022-09-27 | Verizon Patent And Licensing Inc. | Systems and methods for dynamic connection paths for devices connected to computer networks |
WO2018208570A1 (en) * | 2017-05-09 | 2018-11-15 | Cisco Technology, Inc. | Routing network traffic based on dns |
US20180331947A1 (en) * | 2017-05-09 | 2018-11-15 | Cisco Technology, Inc. | Routing network traffic based on dns |
CN110612707A (en) * | 2017-05-09 | 2019-12-24 | 思科技术公司 | Routing network traffic based on DNS |
US10826820B2 (en) | 2017-05-09 | 2020-11-03 | Cisco Technology, Inc. | Routing network traffic based on DNS |
CN110612707B (en) * | 2017-05-09 | 2022-12-02 | 思科技术公司 | Routing network traffic based on DNS |
US11722403B2 (en) | 2017-05-09 | 2023-08-08 | Cisco Technology, Inc. | Routing network traffic based on DNS |
WO2020088684A1 (en) * | 2018-11-02 | 2020-05-07 | 华为技术有限公司 | Method and network device for routing processing |
US11863447B2 (en) | 2018-11-02 | 2024-01-02 | Huawei Technologies Co., Ltd. | Route processing method and network device |
US11070645B1 (en) * | 2018-12-14 | 2021-07-20 | Amazon Technologies, Inc. | Flexible scheduling of data transfers between computing infrastructure collections for efficient resource utilization |
Also Published As
Publication number | Publication date |
---|---|
US20110032833A1 (en) | 2011-02-10 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US8098584B2 (en) | Optimization of traffic routing for data center services | |
US10972387B2 (en) | Application performance based path-selection using dynamic metrics | |
US11863417B2 (en) | Routing mode and point-of-presence selection service | |
US6795858B1 (en) | Method and apparatus for metric based server selection | |
US6831895B1 (en) | Methods and devices for relieving congestion in hop-by-hop routed packet networks | |
US10091096B1 (en) | Routing mode and point-of-presence selection service | |
Zhang et al. | Optimizing Cost and Performance in Online Service Provider Networks. | |
CN104836732B (en) | The automatic selecting method and system of network connection | |
CN104158919A (en) | Webpage access implementation method, server and client | |
US8050193B2 (en) | Method for determining prospective peering partners for an internet service provider | |
JP2018528695A (en) | Method and apparatus for directing real-time traffic using real-time user monitoring data | |
EP1311967A1 (en) | Method and system for optimizing routing through multiple available internet route providers | |
Lodhi et al. | Complexities in Internet peering: Understanding the “black” in the “black art” | |
Yang et al. | Bandwidth–delay constrained routing algorithms | |
Venetis et al. | Mobile agents-based data aggregation in WSNs: Benchmarking itinerary planning approaches | |
JP4419865B2 (en) | Real network traffic management method, program and apparatus for virtual network | |
Karyotis et al. | Utility decisions for QoE-QoS driven applications in practical mobile broadband networks | |
Hillmann et al. | Modeling the location selection of mirror servers in content delivery networks | |
Hsu et al. | DiffServ‐based bandwidth‐constrained anycast routing in a mobile IPv6 network | |
CN105917621A (en) | Methods and systems for data routing | |
Tomar et al. | Conceptual model for comparison of IPv6 ISPs based on IPv4 traffic profiles | |
Luo et al. | Software defined network‐based multipath state‐aware routing with traffic prediction in satellite network | |
Cohen et al. | On the computational complexity and effectiveness of N-hub shortest-path routing | |
Mekbungwan et al. | In-network computation for IoT data processing with ActiveNDN in wireless sensor networks | |
Tomic et al. | Implementation and efficiency analysis of composite DNS-metric for dynamic server selection |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: MICROSOFT CORPORATION, WASHINGTON Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:ZHANG, MING;GREENBERG, ALBERT;MAHAJAN, RATUL;AND OTHERS;SIGNING DATES FROM 20090805 TO 20090806;REEL/FRAME:023068/0233 |
|
FEPP | Fee payment procedure |
Free format text: PAYOR NUMBER ASSIGNED (ORIGINAL EVENT CODE: ASPN); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY |
|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |
|
AS | Assignment |
Owner name: MICROSOFT TECHNOLOGY LICENSING, LLC, WASHINGTON Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:MICROSOFT CORPORATION;REEL/FRAME:034564/0001 Effective date: 20141014 |
|
FPAY | Fee payment |
Year of fee payment: 4 |
|
MAFP | Maintenance fee payment |
Free format text: PAYMENT OF MAINTENANCE FEE, 8TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1552); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY Year of fee payment: 8 |
|
MAFP | Maintenance fee payment |
Free format text: PAYMENT OF MAINTENANCE FEE, 12TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1553); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY Year of fee payment: 12 |