US20140122634A1 - Numa aware network interface - Google Patents
Numa aware network interface Download PDFInfo
- Publication number
- US20140122634A1 US20140122634A1 US13/663,131 US201213663131A US2014122634A1 US 20140122634 A1 US20140122634 A1 US 20140122634A1 US 201213663131 A US201213663131 A US 201213663131A US 2014122634 A1 US2014122634 A1 US 2014122634A1
- Authority
- US
- United States
- Prior art keywords
- nic
- packet
- node
- interconnect
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F13/00—Interconnection of, or transfer of information or other signals between, memories, input/output devices or central processing units
- G06F13/38—Information transfer, e.g. on bus
- G06F13/40—Bus structure
- G06F13/4004—Coupling between buses
- G06F13/4027—Coupling between buses using bus bridges
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L12/00—Data switching networks
- H04L12/54—Store-and-forward switching systems
- H04L12/56—Packet switching systems
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/08—Configuration management of networks or network elements
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L47/00—Traffic control in data switching networks
- H04L47/10—Flow control; Congestion control
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L47/00—Traffic control in data switching networks
- H04L47/10—Flow control; Congestion control
- H04L47/33—Flow control; Congestion control using forward notification
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L69/00—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
- H04L69/30—Definitions, standards or architectural aspects of layered protocol stacks
- H04L69/32—Architecture of open systems interconnection [OSI] 7-layer type protocol stacks, e.g. the interfaces between the data link level and the physical level
- H04L69/321—Interlayer communication protocols or service data unit [SDU] definitions; Interfaces between layers
Definitions
- the field of invention relates generally to computer networking and, more specifically but not exclusively relates to a methods, apparatus, and computer platforms and architectures employing Non-uniform Memory Access (NUMA) aware network interfaces.
- NUMA Non-uniform Memory Access
- LAN Local Area Network
- Internet accessing the Internet via the LAN or other access point
- users are always logged on to at least one service that is accessed via a computer network.
- cloud-based services has lead to even further usage of computer networks, and these services are forecast to become ever-more prevalent.
- broadband network backbones typically support bandwidths of 10 Gigabits per second (Gbps) or more, while the standard for today's personal computers is a network interface designed to support a 1 Gbps Ethernet link.
- processors capabilities have been increased through both faster clock rates and use of more than one processor core.
- today's PCs typically employ a dual-core processor or a quad-core processor, while servers may employ processors with even more cores.
- processors it is common to employ multiple processors to enhance performance.
- it is envisioned that much if not most of the future processor performance increases will result from architectures employing greater numbers of cores, and that future servers may employ greater numbers of processors.
- NIC Network Interface Controller
- server NICs have been designed to support for many optimizations for multi-core, multi-processor platform architectures. These optimizations include Receive Side Scaling (RSS) and Application Targeted Routing (ATR). These optimizations were designed around the prior art front-side bus (FSB) platform architecture, as illustrated in FIG. 1 .
- RSS Receive Side Scaling
- ATR Application Targeted Routing
- FIG. 1 depicts a simplified front-side bus architecture diagram for a symmetric multiprocessing (SMP) platform.
- the architecture includes multiple processors 100 coupled to a front-side bus (FSB) 102 .
- FSB front-side bus
- North bridge 104 which in turn is coupled to memory 106 , a high-bandwidth Input/Output (I/O) interface (as depicted by a Platform Component Interconnect Express (PCIe) x8 interface 108 ), and a South bridge 110 .
- South bridge 110 was typically configured to interface with various platform I/O devices and peripherals, such as depicted by PCIe x4 interfaces 112 and 114 .
- the network interface controllers were attached via a PCIe interface to either North bridge 104 or South bridge 110 , as depicted by NICs 116 and 118 . In either case, the NICs communicated to a uniform memory 106 via North bridge 104 . All processor 100 accesses to memory 106 were also via North bridge 104 . Implementation of RSS and ATR distributed network workloads across cores and, although cache impacts were considered, the primary goal was workload distribution.
- processor architectures have also changed in recent years, moving from discrete components toward a highly integrated approach.
- the North-bridge, South-bridge architecture was implemented using physically separate chips for North bridge 104 and South bridge 110 using wired (i.e., board traces) interconnects for the FSB and the interconnect between the North and South bridges.
- wired interconnects for the FSB and the interconnect between the North and South bridges.
- a processor employing one or more processor cores and logic providing functionality somewhat similar to a North bridge and South bridge are integrated on a single chip with corresponding interconnect wiring embedded in the chip.
- the processor cores are referred to as the “core” and the rest of the processor circuitry is referred to as the “uncore.”
- FIG. 1 is a schematic diagram of a conventional symmetric multi-processor platform architecture
- FIG. 2 is a schematic diagram of a NUMA platform architecture, further depicting processing of a packet under a conventional approach;
- FIG. 3 is a schematic diagram of a NUMA platform architecture employing NUMA aware network interfaces, according to one embodiment
- FIG. 4 is a schematic diagram illustrating further details of the NICs and Node system memories of FIG. 3 , according to one embodiment
- FIG. 4 a is a schematic diagram illustrating further details of NIC B of FIG. 3 , according to one embodiment
- FIG. 4 b is a schematic diagram illustrating one embodiment of a NUMA aware NIC employing two NIC-to-NIC interconnect interfaces
- FIG. 5 is a flowchart illustrating operations and logic performed during handling of packets using the NUMA platform architectures of FIGS. 3 , 4 , and 4 a.
- FIG. 6 is a schematic diagram of a NUMA platform architecture employing four nodes including NUMA aware network interfaces, according to one embodiment.
- FIG. 7 is a schematic diagram illustrating a ring interconnect architecture used to interconnect multiple NUMA aware network interfaces, according to one embodiment.
- NUMA Non-uniform Memory Access
- FIG. 2 illustrates a Non-Uniform Memory Access (NUMA) multi-processor platform architecture 200 employing two NUMA nodes 202 a and 202 b , also labeled ‘A’ and B′.
- NUMA Non-Uniform Memory Access
- reference to nodes herein may be to their letter label rather than their reference number.
- reference to a component that is common across nodes will be made to the base reference number, while a letter ‘a’, ‘b’, will be added to the reference number in the drawing Figures to identify that component on a particular node.
- each of nodes A and B include a processor 204 , which is labeled 204 a for the processor for node A and 204 b for the processor for node B.
- each node includes system memory 206 and a NIC 208 .
- Each processor 204 includes a core portion including a plurality of processor cores 210 , each including a local level 1 (L1) and level 2 (L2) cache.
- L1 and L2 level 1 cache
- the remaining portion of the processor is referred to as the uncore, and includes various interconnect circuitry and interfaces for connecting various functional blocks on the processor in communication.
- this interconnect circuitry is depicted as an interconnect 212 ; however, it will be understood that interconnect 212 may be representative of one or more interconnect structures, such as buses and single or multi-lane serial point-to-point or mesh interconnect structures.
- a portion of the uncore circuitry is configured to handle many of the North-bridge functions under the legacy architecture of FIG. 1 , including memory control and PCIe interfaces for devices such as NICs.
- Corresponding exemplary functional blocks depicted in the processor uncores in FIG. 2 include a PCIe interface (I/F) 214 , a last level cache (LL Cache) 215 , a memory controller (MC) 216 , and a socket-to-socket link interface (S-to-S I/F) 218 .
- I/F PCIe interface
- LL Cache last level cache
- MC memory controller
- S-to-S I/F socket-to-socket link interface
- each processor 204 would include many other functional blocks that are not shown for clarity.
- Each of processors 204 is operatively coupled to a printed circuit board called main board 220 via a socket, or otherwise coupled to the main board via a direct coupling technique, such as flip-chip bonding. In either case, it is common practice to refer to the processors themselves as sockets, and thus the usage of the terminology socket-to-socket links and interfaces for coupling processor nodes in communication.
- Main board 220 includes electrical wiring (i.e., traces and vias) to facilitate electrical connections corresponding to the physical structure of various interconnects depicted in FIG. 2 .
- interconnects include PCIe interconnects 222 between PCIe interfaces 214 and NICs 208 , interconnects 224 between memory controllers 216 and system memory 206 , and a socket-to-socket link 226 coupled between socket-to-socket interfaces 218 a and 218 b .
- socket-to-socket interfaces 218 a and 218 b employ the Intel Quickpath Interconnect (QPI)® protocol and wiring structure.
- QPI Quickpath Interconnect
- processors are enabled to access different memory resources distributed across the platform.
- the memory resources may be considered local memory resources (i.e., memory resources on the same node as a processor or core) or non-local memory resources (i.e., memory resources on other nodes).
- system memory 206 a comprises a local memory resource
- system memory 206 b comprises a non-local memory resource.
- non-local memory resources may also be shared between processors while not being associated with a particular processor or node.
- software running on a processor core in one node may be allocated a memory address space in system memory nominally associated with another node.
- the address space for a portion of the memory resources for a platform may be managed as a single global pool, with unique addresses for each memory address location within the pool.
- a portion of the address space may be considered local address space that employs local addressing (e.g., address space allocated to an operating system), while another portion may be addressed using global addressing.
- network ports may be assigned to software application that are running on processors (or processor cores) that are on nodes that are different than the node associated with the assigned port.
- processors or processor cores
- the NIC receiving the packets and the destined memory resource in which the packet data is to be written are on separate nodes.
- NICs are configured to provide an interface with a computer network using a corresponding network protocol, such as the Ethernet protocol depicted in FIG. 2 .
- each NIC is associated with an operating system (OS) NIC (device) driver that is logically located in an OS kernel.
- OS operating system
- the NIC driver is used as an abstracted interface between the operating system software and the NIC, which is a hardware device.
- a NIC driver may provide access to registers on a NIC, provide a program interface to the NIC, etc.
- the NIC driver also facilitates handling and forwarding of data received via packets from the network to consumers of that data, such as a software application.
- NIC input port For instance, under conventional usage packets are received at a NIC input port and buffered in an input buffer and then copied to a memory buffer in system memory that is allocated to the NIC driver by the operating system. Additionally processing of the packet data may also be performed by the NIC driver.
- FIG. 2 which further shows how data corresponding to a packet 228 that is received by a NIC at a first node but contains data that is to be written to system memory or cache on a second node is handled under a conventional approach.
- each of NICs A and B are connected via respective Ethernet links 230 to a network 232 .
- network 232 is shown as two separate clouds; however, it will be understood that these represent connections to the same network.
- the depiction of separate clouds in the Figures herein may be interpreted as identifying connections to separate networks, regardless to whether they are depicted using the same reference number.
- the handling of packet 228 proceeds as follows, with corresponding operations depicted by encircled numbers in FIG. 2 .
- the packet is received from network 232 via Ethernet link 230 a at an input port of NIC 208 a and is buffered (i.e., temporarily stored) in an input buffer.
- a packet data of packet 228 is then copied from the input buffer and written to system memory 206 a using a DMA (Direct Memory Access) write operation. This involves forwarding packet data corresponding to the packet from NIC 208 a via PCIe link 222 a and PCIe interface 214 a to memory controller 216 a via interconnect 212 a .
- DMA Direct Memory Access
- Memory controller 216 a then writes the packet data to system memory 206 a , as depicted at an operation 2 a .
- the packet data is written to LL Cache 214 a , as depicted by an operation 2 b .
- the use of a DMA write operation enables the memory write to be performed independent of an operating system.
- forwarding packet data from one storage location to another is commonly termed forwarding the packet, regardless of the number of copies of the packet data that are employed. Accordingly, forwarding a packet or forwarding packet data in the detailed description and the claims includes implied data copy operations, as applicable. In addition, forwarding a packet or forwarding packet data may include transferring a portion or portions of the data in the packet, the complete packet, and/or meta-data associated with the packet such as classification results, hash values, etc.
- a software entity running on one of cores 210 a e.g., a NIC driver
- a software entity running on one of cores 210 a will either detect or be apprised of the arrival of packet 228 in system memory 206 a or LL Cache 214 a and determine that the packet data is actually associated with (i.e., the consumer of the packet is) a consumer software application having a memory allocation including a portion of the memory address space of system memory 206 b on node B. Or more simply, a determination will be made that the packet data is to be written to an address within an address range associated with system memory 206 b .
- the data for packet 228 is copied from system memory 206 a or LL Cache 214 a to a first buffer on socket-to-socket interface 218 a on node A and sent to socket-to-socket interface 218 b on node B over interconnect link 226 , where it is received in a second buffer.
- the buffered copy of packet 228 is either then written to system memory 206 b or LL Cache 214 b during a fifth operation, as depicted by operations 5 a and 5 b .
- the packet data may be accessed by the software application that is a consumer of the packet via its location in either system memory 214 b or LL Cache 214 b.
- packets received at a NIC for a first node that are destined to be written to system memory or cache corresponding to a second node are detected at the NIC of the first node and transferred to the NIC for the second node over a high-speed NIC-to-NIC interconnect between the NICs.
- the packet is received at the second NIC, it is handled as if the packet was received from the network via the second NIC to begin with.
- this process is transparent to other components and software entities running on both the first and second node.
- FIG. 3 shows a NUMA multi-processor platform architecture 300 that is configured to implement the foregoing approach, according to one embodiment.
- Aspects of platform architecture 300 are similar to platform architecture 200 , where like-numbered components perform similar operations, while augmented and new components are shown in light gray.
- FIG. 3 further shows how data corresponding to a packet 228 that is received by a NIC at a first node (A) but contains data that is to be written to a memory resource on a second node (B) is handled under NUMA platform architecture 300 .
- handling of packet 228 proceeds as follows. As before, the packet is received at NIC 308 a from network 232 via Ethernet link 230 a and is buffered in an input buffer on the NIC during a first operation. During a second operation corresponding to an initial packet processing operation performed by NIC 308 a , the packet header and/or packet payload for packet 228 is inspected, and applicable filter match input data is generated based on packet header field data and/or packet payload data, which is then employed by packet processing logic on NIC 308 a as an input lookup against filters and/or rules defined in node filter table 304 that is maintained in system memory 206 a .
- the filters and/or rules in node filter table 304 enables the node of the memory resource to which applicable packet data is to be written to be identified.
- the memory resource node is node B, which has a corresponding NIC 308 b . Accordingly, during a third operation, data corresponding to a copy of packet 228 is forwarded from the input buffer (or another buffer) on NIC 308 a to an input buffer on NIC 308 b .
- the copy of packet 228 is then processed during a fourth operation as if it was received by NIC 308 b from network 232 via Ethernet 230 b , resulting in a copy of packet 228 being written using a DMA write operation to either system memory 206 b (operation 4 a ) or LL Cache 215 b (operation 4 b ).
- the packet handling technique illustrated for NUMA platform architecture 300 enhances platform performance by eliminating the copying of the packet into system memory of the incorrect node (A) and then having to forward the packet data to its appropriate destination node (B) over the socket-to-socket interconnect 226 . This reduces both the bandwidth consumption on the various interconnects that were employed under the conventional NUMA platform architecture 200 , as well as packet handling operations performed by an applicable software entity or entities on the incorrect node.
- FIG. 4 shows further details of NICs 308 a and 308 b and system memories 206 a and 206 b of NUMA platform architecture 300 , according to one embodiment.
- Each of NICs 308 a and 308 b includes a Physical layer (PHY) interface 400 to which a Media Access Control (MAC) block 402 is operatively coupled.
- MAC block 402 includes a First-in, First-out (FIFO) input buffer 404 and filter table registers 406 .
- Each NIC further includes packet processing logic 408 , a DMA engine 410 , a processor interconnect interface 412 , and a NIC-to-NIC interconnect interface 414 .
- NICs 308 a and 308 b also include an optional node filter table cache 416 .
- Node A system memory 206 a includes various portions of memory address space that is allocated to store corresponding data. These include a portion of memory address space allocated to software applications A and B, and a node filter table 304 a .
- node B system memory 206 b includes memory address space allocated to a NIC B device driver, a software application C, and a node filter table 304 b .
- a portion of the memory address space for each of system memory 206 a and 206 b would be allocated to one or more operating systems, and the NIC B device driver may be located in a kernel for one of the operating systems.
- a NIC device driver may be logically associated with a virtual machine manager (VMM) or the like.
- VMM virtual machine manager
- one or more entries in filter table registers 406 identify the location (i.e., starting address or address range) of the node filter table 304 for the node.
- FIG. 4 a depicts a NIC 308 b ′ illustrating further details of a Node 308 in FIG. 4 , according to one embodiment.
- MAC 402 b ′ is further depicted to include a second FIFO input buffer 418 b to which a PHY 420 a of NIC-to-NIC interconnect interface 414 a is coupled.
- Packet processing logic 408 b includes a packet classifier 422 b and a hash engine 424 b .
- FIG. 4 a shows further details of optional node filter table cache 416 b.
- FIG. 4 b shows further details of a NIC 309 b , according to one embodiment.
- NIC 309 b is similar to NIC 308 b ′ of FIG. 4 a , but with the further addition of a second NIC-to-NIC interface 415 b including a PHY 421 b and a FIFO input buffer 419 b .
- each NIC-to-NIC interconnect interface is operatively coupled to a respective FIFO input buffer.
- a NIC may have a third NIC-to-NIC interface including a PHY that is operatively coupled to a third FIFO input buffer (not shown).
- a NIC-to-NIC interconnect interface may be multiplexed either before or after a PHY, enabling a single NIC-to-NIC interconnect interface to be shared for multiple NIC-to-NIC interconnect links.
- FIG. 4 b also depicts an operating system 432 including a NIC B driver 434 , and software applications 436 and 438 .
- Each node will typically host one or more operating system instances.
- One of the operating system instances may generally be employed to facilitate platform forwarding configuration information for the NICs.
- the operating system and/or NIC driver may keep track of which NIC is allocated for sending and receiving network traffic corresponding to a software application consumer.
- NIC B driver 434 operates as an interface between operating system 432 and the NIC associated with the node the operating system is running on, e.g., NIC B in FIG. 4 b .
- NIC driver 434 B may also be configured to update one or more of node filter table cache 416 b , packet processing logic 408 b , and filter table registers 406 b.
- processing of a packet 228 proceeds as follows.
- packet data including a packet header 426 and payload 428 is received in the form of an analog modulated Ethernet signal at an input port 430 that is communicatively coupled to PHY 400 a of NIC 308 a .
- the PHY is used to convert the Ethernet signal via which packet data is conveyed from an analog signal to a digital data stream, and then deserialize the data stream, forwarding the deserialized digital data as blocks of data to MAC block 402 .
- MAC block 402 performs additional processing, including detection of the start and end of packets to extract packet data that is buffered in FIFO input buffer 404 a.
- packet header 426 and/or packet payload 428 are inspected, and the packet is classified in a block 504 by packet classifier 422 .
- packet classification There are various schemes well known in the art that may be used for packet classification, such as a 5-tuple classification of an Internet Protocol (IP) packet header (source address, destination address, source port, destination port, and transport protocol), or classification based on one or more header fields.
- IP Internet Protocol
- data corresponding to packet payload 428 may also be considered during packet classification.
- packet classification data may be encoded in a packet header or the packet payload.
- packet classification is implemented by performing a hash on one or more header fields via hash engine 424 .
- a hash may be performed on a concatenation of the 5-tuple IP field values.
- hash engine 424 may also be used to generate a hash value based on data in any combination of packet header fields and packet payload data.
- the result of packet classification is the generation of a filter match input value (i.e., filter match input data) that is used as a lookup input to a lookup table comprising node filter table 304 a or a packet flow table including node filter columns, as depicted in a block 506 .
- a filter match input value i.e., filter match input data
- the corresponding data for the NIC column and/or Node column could be included in a packet flow table that is used to associate packets with particular header and/or payload data to corresponding packet flows. Since flow table filters and rules are typically derived via packet classification, a conventional packet flow table could be modified to include additional columns corresponding to the illustrated NIC and Node columns.
- a lookup into node filter table cache 416 a is first made to determine whether there is a matching node filter table entry.
- the use of a cached node filter table may be advantageous under various server implementations. For example, it is common to classify packets associated with a client-server connection session into corresponding flows based on a combination of header field values, such as a 5-tuple hash.
- the flowchart logic proceeds to decision block 510 . If not, the logic proceeds to a decision block 512 in which a look up against a node filter table or packet flow table in system memory (e.g., node filter table 304 a ) is performed. (Although depicted in the drawings herein in system memory for simplicity, if is possible that data corresponding to a node filter table or packet flow table may actually be accessed from a processor memory cache under some embodiments.) If a match is found, the logic proceeds to an optional block 514 in which an entry in the node filter table cache 416 a is added.
- system memory e.g., node filter table 304 a
- each added entry requires an eviction of a current entry, which may be selected using any well-known cache eviction policy, such as least recently used (LRU), least frequency used (LFU), or FIFO, for example.
- LRU least recently used
- LFU least frequency used
- FIFO FIFO
- a new node filter entry including corresponding filter match value and/or rule is generated in a block 516 .
- This process may typically be handled by a NIC driver or other entity that is employed for managing the node filter table for the node.
- the new entry will include a match filter or rule and identification to which node the classified flow is assigned to.
- a connection setup and teardown feature may be employed that enables applicable packet classification information to be determined before any packets are transferred. Accordingly, under this approach a new node filter table entry or packet flow table entry may be added during the connection setup operation, rather than in response to a lookup table miss.
- node filter table cache 416 a global copies of the node filter table are maintained across all of the nodes on a platform.
- copies of node filter tables may be shared among neighboring nodes. Accordingly, in an optional block 518 a copy of the new entry is forwarded to other node(s), as applicable. The new entry may also be added to node filter table cache 416 a if this is implemented and depending on the cache replacement policy. As before, the logic then proceeds to decision block 510 .
- the node corresponding to the NIC that receives the packet from the network e.g., Node A in this example
- the logic proceeds to a block 522 in which a determination of the NIC via which the correct node can be accessed is made, followed by forwarding a copy of the packet data from the first NIC to that (second) NIC via one or more NIC-to-NIC interconnect links between the first and second NICs.
- a copy of the packet data would be forwarded from NIC 308 a to NIC 308 b via NIC-to-NIC Interconnect 302 .
- forwarding a packet from a first NIC to a second NIC may traverse more than one NIC-to-NIC interconnect link under some embodiments.
- the packet is received at the NIC-to-NIC interconnect interface of the correct (second) node, and the packet is buffered in an input buffer.
- the packet is received and processed by PHY 420 b and then buffered in FIFO input buffer 418 b .
- the packet could be buffered in the same buffer as the packets received at an input port 430 b for NIC 308 b ′ using a multiplexing scheme that multiplexes inputs from PHY 400 b and PHY 420 b (not shown).
- a NIC may have a PHY or PHYs coupled to multiple input ports, each with one or more input buffers and multiplexers (as applicable). Generally, one of these buffers could be shared with a NIC-to-NIC interconnect interface, or a separate input buffer or buffers may be used.
- the logic returns to block 504 , and further handling of the packet proceeds as if the packet was originally received at the correct NIC to begin with.
- the packet data is written to a memory location specified by a memory address associated with the NIC B driver via a DMA write operation facilitated by DMA engine 410 b and then the packet payload is copied to a memory address associated with the software consumer of the packet, depicted as software application C.
- the packet payload may be copied directly to an applicable memory address allocated to the software consumer application, or to an applicable address in the LL Cache for the second Node (B).
- a node filter lookup table may include filter and/or rule data that considers a memory address that is determined during packet processing, with corresponding table entries identifying nodes and/or NICs that packet is to be forwarded to.
- a filter column may include address range maps to corresponding nodes and/or NICs.
- NUMA platform architecture employing two or more nodes.
- An exemplary four-node NUMA platform architecture 600 is shown in FIG. 6 , including nodes 302 a , 302 b , 302 c , and 302 d , wherein each of the nodes has a similar configuration including a respective processor 304 , memory 206 , and NIC 309 .
- socket-to-socket interconnect interfaces 218 on processors 304 a and 304 c are depicted as being mirrored with mating socket-to-socket interconnect interfaces 218 on processors 304 b and 304 d , it will be understood that in one embodiment processors 304 a - d have similar configurations; and the use of mirroring the interfaces in FIG. 6 is used to make the interconnect routing simpler and easier to understand.
- nodes 302 c and 302 d are depicted as having a mirrored configuration to nodes 302 a and 302 b , it will be understood that this representation is merely for illustrative purposed to make the interconnects and Ethernet connections to network 232 more direct.
- each of processors 304 a - d is operatively coupled to a main board 602 via a respective socket (not shown) or otherwise mounted to main board 602 via an appropriate bonding scheme, such as flip-chip bounding.
- the components for a given node may be mounted or otherwise operatively coupled to a daughter board, which in turn is coupled via one or more connectors to a main board.
- main board 602 includes wiring traces to facilitate interconnects 226 xy between pairs of socket-to-socket interfaces 318 , wherein x and y correspond to the respective socket nodes for the processors being linked in communication via the interconnect.
- interconnect 226 ac is an interconnect between socket-to-socket interfaces 318 a and 318 c .
- platform architecture 600 includes four NICs 309 a - d , which are also labeled NIC A, NIC B, NIC C, and NIC D.
- each of NICs 309 a - d is connected to two other NICs via a NIC-to-NIC interconnect 302 nm, where m and n identify the two NICs that are interconnected.
- These NIC-to-NIC interconnects include interconnects 302 ab , 302 ac , 302 cd , and 302 bd .
- each NIC is interconnected to the other three NICs, as depicted by additional interconnects shown with dashed lines, labeled 302 ad and 302 bc.
- the physical structure of a NIC-to-NIC interconnect may comprise one or more wired or optical pathways between a pair of NICs (forming a pathway between to NIC-to-NIC interface PHYs).
- wired pathways are provided by wire trances in main board 602 .
- an optical cable is coupled to the NIC-to-NIC interfaces via appropriate optical connectors (not shown).
- a wire cable may be used for the wired pathway.
- a wired NIC-to-NIC interconnect link may be implemented over a single path or multiple paths or “lanes” routed in parallel.
- a PCIe link is used, such as a PCIe 1x, 2x, 4x, 8x, or 16x link, were the n in nx corresponds to the number of lanes.
- Other types of high-speed serial interconnects and associated protocols may also be used, including but not limited to Intel's QPI or Keizer Technology Interconnect® (KTI), or an Open Core Protocol (OCP) interconnect.
- NICs A-D may be interconnected to form a ring-type interconnect. This approach may be extended to link larger numbers of NICs, as shown in further detail in FIG. 7 , wherein the top portion of the Figure depicts a plurality of NUMA aware NICs 700 (also labeled NIC A, NIC B, etc.) interconnected via NIC-to-NIC interconnect links 302 x - y .
- the lower portion of FIG. 7 shows details of a NIC 700 , according to one embodiment.
- NIC 700 includes a first NIC-to-NIC interconnect interface 702 including a PHY 704 to which an input buffer 706 and an output buffer 708 is coupled.
- NIC 700 also includes a similarly configured NIC-to-NIC interconnect interface 710 including a PHY 712 coupled to an input buffer 714 and an output buffer 716 .
- input buffer 706 is connected to output buffer 716 and output buffer 708 is connected to input buffer 714 .
- NIC-to-NIC interconnect interfaces 702 and 710 are configured to implement an interconnect protocol employing node-based addressing, where each NIC 700 represents a ring stop node on the ring interconnect. Accordingly, packet data may be forwarded between any NIC 700 to any other NIC 700 via the ring interconnect. Moreover, in one embodiment the NIC-to-NIC interconnect interfaces 702 and 710 are configured to bypass the FIFO input buffers when packet data is forwarded to a NIC 700 that is an intermediate node along a path between a forwarding NIC node and NIC node that is targeted for receiving the forwarded packet data. For example, suppose a packet is to be forwarded from NIC A to NIC N ⁇ 1 via NIC N.
- NIC-to-NIC interconnect interface 702 inspects the destination node address for the packet data and determines 1) that NIC Nis not the destination NIC node, and 2) that the destination NIC node (NIC N ⁇ 1) can be reached by forwarding the packet outbound from NIC-to-NIC interconnect interface 710 . Accordingly, when the packet data is received at NIC N it is buffered in input buffer 706 and then forwarded to output buffer 716 , whereupon the packet data is temporarily stored until it can be send outbound NIC-to-NIC interconnect interface 710 . In this manner, packet data can be forwarded between any NIC nodes on the ring without having to be processed by packet processing logic on the intermediate NIC nodes along the forwarding portion of the ring.
- the NIC-to-NIC interconnects may be configured to support other node configuration.
- star-type node configuration and hierarchical node configuration may be implemented with NIC-to-NIC interconnects and corresponding forwarding logic on the NICs.
- the foregoing embodiment provide enhanced performance over the conventional NUMA approach under which the NICs are agnostic to being deployed in a NUMA platform architecture.
- the NICs are configured to be NUMA aware, they are enabled to forward packet data to the correct node for handling the data for situations where they are not associated with the correct node to begin with.
- This approach avoids additional processing operation and interconnect usage under the conventional approach, such as reducing QPI traffic employed needed for cache coherency.
- it also increases the effectiveness of DDIO since packet data may be directly written to the correct processor's last level cache.
- NUMA architectures While the embodiments depicted herein are referred to as NUMA architectures, the teachings and principles may be applied to other non-NUMA architectures under which forwarding of packets between NICs via NIC-to-NIC interconnects are implemented.
- such architectures will include multiple nodes, each including one or more processors and/or processor cores including local caches.
- system memory may be configured such that it is accessed in a distributed manner across nodes, wherein at least a portion of the memory is not associated with a particular node.
- a last level cached or the like may be configured in a distributed manner across nodes.
- the NIC-to-NIC forwarding feature may be implemented in a similar manner under such non-NUMA architectures as described in the embodiments herein. Under these architectures NIC-to-NIC forwarding may be implemented such that the packet data may be accessed by the processor or processor core running the software application that is a consumer of the packet data. Accordingly, the forwarding table data is configured in consideration of the location of the processor or processor core associated with a software consumer (for a given packet).
- embodiments of the invention may be configured to support network connections via wireless links, such as via wireless connections over an IEEE 802.11-based (aka WiFi) wireless link, a WiMAX link, or various mobile telecommunication-based data links, e.g., GSM, CDMA, GPRS, W-CDMA, EDGE, CDMA2000, UMTS, HSUPA, HSDPA, EV-DO, LTE, etc.
- a NIC may be configured to process corresponding radio frequency (RF) signal data.
- RF signal data processing may be implemented via a separate component or components that are communicatively coupled to a NIC.
- the elements in some cases may each have a same reference number or a different reference number to suggest that the elements represented could be different and/or similar.
- an element may be flexible enough to have different implementations and work with some or all of the systems shown or described herein.
- the various elements shown in the figures may be the same or different. Which one is referred to as a first element and which is called a second element is arbitrary.
- Coupled may mean that two or more elements are in direct physical or electrical contact.
- Coupled may mean that two or more elements are in direct physical or electrical contact.
- coupled may also mean that two or more elements are not in direct contact with each other, but yet still co-operate or interact with each other, while “operatively coupled” may mean that two or more elements are directly or indirectly connected when a component or device is operating.
- An embodiment is an implementation or example of the inventions.
- Reference in the specification to “an embodiment,” “one embodiment,” “some embodiments,” or “other embodiments” means that a particular feature, structure, or characteristic described in connection with the embodiments is included in at least some embodiments, but not necessarily all embodiments, of the inventions.
- the various appearances “an embodiment,” “one embodiment,” or “some embodiments” are not necessarily all referring to the same embodiments.
- embodiments herein may be facilitated by corresponding software and/or firmware components and applications, such as software running on a server or firmware executed by an embedded processor on a network element.
- embodiments of this invention may be used as or to support a software program, software modules, firmware, and/or distributed software executed upon some form of processing core (such as the CPU of a computer, one or more cores of a multi-core processor), a virtual machine running on a processor or core or otherwise implemented or realized upon or within a machine-readable medium.
- a machine-readable medium includes any mechanism for storing or transmitting information in a form readable by a machine (e.g., a computer).
- a machine-readable medium may include a read only memory (ROM); a random access memory (RAM); a magnetic disk storage media; an optical storage media; and a flash memory device, etc.
Abstract
Description
- The field of invention relates generally to computer networking and, more specifically but not exclusively relates to a methods, apparatus, and computer platforms and architectures employing Non-uniform Memory Access (NUMA) aware network interfaces.
- Access to computer networks has become a ubiquitous part of today's computer usage. Whether accessing a Local Area Network (LAN) in an enterprise environment to access shared network resources, or accessing the Internet via the LAN or other access point, it seems users are always logged on to at least one service that is accessed via a computer network. Moreover, the rapid expansion of cloud-based services has lead to even further usage of computer networks, and these services are forecast to become ever-more prevalent.
- Expansion of network usage, particularly via cloud-based services, as been facilitated via substantial increases in network bandwidths and processor capabilities. For example, broadband network backbones typically support bandwidths of 10 Gigabits per second (Gbps) or more, while the standard for today's personal computers is a network interface designed to support a 1 Gbps Ethernet link. On the processor side, processors capabilities have been increased through both faster clock rates and use of more than one processor core. For instance, today's PCs typically employ a dual-core processor or a quad-core processor, while servers may employ processors with even more cores. For some classes of servers, it is common to employ multiple processors to enhance performance. In addition, it is envisioned that much if not most of the future processor performance increases will result from architectures employing greater numbers of cores, and that future servers may employ greater numbers of processors.
- In computer systems, network access is typically facilitated through use of a Network Interface Controller (NIC), such as an Ethernet NIC. In recent years, server NICs have been designed to support for many optimizations for multi-core, multi-processor platform architectures. These optimizations include Receive Side Scaling (RSS) and Application Targeted Routing (ATR). These optimizations were designed around the prior art front-side bus (FSB) platform architecture, as illustrated in
FIG. 1 . - In further detail,
FIG. 1 depicts a simplified front-side bus architecture diagram for a symmetric multiprocessing (SMP) platform. The architecture includesmultiple processors 100 coupled to a front-side bus (FSB) 102. Also coupled to FSB 102 is aNorth bridge 104, which in turn is coupled tomemory 106, a high-bandwidth Input/Output (I/O) interface (as depicted by a Platform Component Interconnect Express (PCIe) x8 interface 108), and a Southbridge 110. Southbridge 110 was typically configured to interface with various platform I/O devices and peripherals, such as depicted byPCIe x4 interfaces - Under this legacy architecture the network interface controllers were attached via a PCIe interface to either North
bridge 104 or Southbridge 110, as depicted by NICs 116 and 118. In either case, the NICs communicated to auniform memory 106 via Northbridge 104. Allprocessor 100 accesses tomemory 106 were also via Northbridge 104. Implementation of RSS and ATR distributed network workloads across cores and, although cache impacts were considered, the primary goal was workload distribution. - Processor architectures have also changed in recent years, moving from discrete components toward a highly integrated approach. For example, for many years, the North-bridge, South-bridge architecture was implemented using physically separate chips for North
bridge 104 and Southbridge 110 using wired (i.e., board traces) interconnects for the FSB and the interconnect between the North and South bridges. Under a typical highly integrated design employed by today's processors, a processor employing one or more processor cores and logic providing functionality somewhat similar to a North bridge and South bridge are integrated on a single chip with corresponding interconnect wiring embedded in the chip. Under this highly integrated architecture, the processor cores are referred to as the “core” and the rest of the processor circuitry is referred to as the “uncore.” - The foregoing aspects and many of the attendant advantages of this invention will become more readily appreciated as the same becomes better understood by reference to the following detailed description, when taken in conjunction with the accompanying drawings, wherein like reference numerals refer to like parts throughout the various views unless otherwise specified:
-
FIG. 1 is a schematic diagram of a conventional symmetric multi-processor platform architecture; -
FIG. 2 is a schematic diagram of a NUMA platform architecture, further depicting processing of a packet under a conventional approach; -
FIG. 3 is a schematic diagram of a NUMA platform architecture employing NUMA aware network interfaces, according to one embodiment; -
FIG. 4 is a schematic diagram illustrating further details of the NICs and Node system memories ofFIG. 3 , according to one embodiment; -
FIG. 4 a is a schematic diagram illustrating further details of NIC B ofFIG. 3 , according to one embodiment; -
FIG. 4 b is a schematic diagram illustrating one embodiment of a NUMA aware NIC employing two NIC-to-NIC interconnect interfaces; -
FIG. 5 is a flowchart illustrating operations and logic performed during handling of packets using the NUMA platform architectures ofFIGS. 3 , 4, and 4 a. -
FIG. 6 is a schematic diagram of a NUMA platform architecture employing four nodes including NUMA aware network interfaces, according to one embodiment; and -
FIG. 7 is a schematic diagram illustrating a ring interconnect architecture used to interconnect multiple NUMA aware network interfaces, according to one embodiment. - Embodiments of methods, apparatus, and computer platforms and architectures employing Non-uniform Memory Access (NUMA) aware network interfaces are described herein. In the following description, numerous specific details are set forth to provide a thorough understanding of embodiments of the invention. One skilled in the relevant art will recognize, however, that the invention can be practiced without one or more of the specific details, or with other methods, components, materials, etc. In other instances, well-known structures, materials, or operations are not shown or described in detail to avoid obscuring aspects of the invention.
- Reference throughout this specification to “one embodiment” or “an embodiment” means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the present invention. Thus, the appearances of the phrases “in one embodiment” or “in an embodiment” in various places throughout this specification are not necessarily all referring to the same embodiment. Furthermore, the particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.
-
FIG. 2 illustrates a Non-Uniform Memory Access (NUMA)multi-processor platform architecture 200 employing twoNUMA nodes - In addition to a processor 204, each node includes system memory 206 and a NIC 208. Each processor 204 includes a core portion including a plurality of processor cores 210, each including a local level 1 (L1) and level 2 (L2) cache. The remaining portion of the processor is referred to as the uncore, and includes various interconnect circuitry and interfaces for connecting various functional blocks on the processor in communication. For simplicity this interconnect circuitry is depicted as an interconnect 212; however, it will be understood that interconnect 212 may be representative of one or more interconnect structures, such as buses and single or multi-lane serial point-to-point or mesh interconnect structures.
- A portion of the uncore circuitry is configured to handle many of the North-bridge functions under the legacy architecture of
FIG. 1 , including memory control and PCIe interfaces for devices such as NICs. Corresponding exemplary functional blocks depicted in the processor uncores inFIG. 2 include a PCIe interface (I/F) 214, a last level cache (LL Cache) 215, a memory controller (MC) 216, and a socket-to-socket link interface (S-to-S I/F) 218. In addition to these illustrated blocks, each processor 204 would include many other functional blocks that are not shown for clarity. - Each of processors 204 is operatively coupled to a printed circuit board called
main board 220 via a socket, or otherwise coupled to the main board via a direct coupling technique, such as flip-chip bonding. In either case, it is common practice to refer to the processors themselves as sockets, and thus the usage of the terminology socket-to-socket links and interfaces for coupling processor nodes in communication.Main board 220 includes electrical wiring (i.e., traces and vias) to facilitate electrical connections corresponding to the physical structure of various interconnects depicted inFIG. 2 . These interconnects include PCIe interconnects 222 between PCIe interfaces 214 and NICs 208, interconnects 224 between memory controllers 216 and system memory 206, and a socket-to-socket link 226 coupled between socket-to-socket interfaces socket interfaces - Under a NUMA architecture, processors (and processor cores) are enabled to access different memory resources distributed across the platform. The memory resources may be considered local memory resources (i.e., memory resources on the same node as a processor or core) or non-local memory resources (i.e., memory resources on other nodes). For example, under the viewpoint of
node 202 a,system memory 206 a comprises a local memory resource, whilesystem memory 206 b comprises a non-local memory resource. Under another type of NUMA architecture (not depicted herein), non-local memory resources may also be shared between processors while not being associated with a particular processor or node. Since, for a given node, local memory resources are operatively coupled to a processor, while non-local memory resources are not, the access to the local memory resources relative to the non-local memory resources is not the same (i.e., the access is non-uniform). Moreover, it is preferable to employ local memory resources when available. - Under
platform architecture 200, software running on a processor core in one node may be allocated a memory address space in system memory nominally associated with another node. Accordingly, under some embodiments the address space for a portion of the memory resources for a platform may be managed as a single global pool, with unique addresses for each memory address location within the pool. Additionally, a portion of the address space may be considered local address space that employs local addressing (e.g., address space allocated to an operating system), while another portion may be addressed using global addressing. - In accordance with another aspect of NUMA architectures, network ports (as thus associated NICs) may be assigned to software application that are running on processors (or processor cores) that are on nodes that are different than the node associated with the assigned port. When the memory allocated to the software application is on the same node as the processor, the NIC receiving the packets and the destined memory resource in which the packet data is to be written are on separate nodes.
- As is well-known, NICs are configured to provide an interface with a computer network using a corresponding network protocol, such as the Ethernet protocol depicted in
FIG. 2 . Under a typical implementation, each NIC is associated with an operating system (OS) NIC (device) driver that is logically located in an OS kernel. The NIC driver is used as an abstracted interface between the operating system software and the NIC, which is a hardware device. For example, a NIC driver may provide access to registers on a NIC, provide a program interface to the NIC, etc. The NIC driver also facilitates handling and forwarding of data received via packets from the network to consumers of that data, such as a software application. For instance, under conventional usage packets are received at a NIC input port and buffered in an input buffer and then copied to a memory buffer in system memory that is allocated to the NIC driver by the operating system. Additionally processing of the packet data may also be performed by the NIC driver. - Under NUMA architectures, there may be instances where packets received via a computer network at a NIC on a first node needs to be written to system memory or a memory cache associated with another node, as discussed above. Such a situation is illustrated in
FIG. 2 , which further shows how data corresponding to apacket 228 that is received by a NIC at a first node but contains data that is to be written to system memory or cache on a second node is handled under a conventional approach. In the example illustrated inFIG. 2 , each of NICs A and B are connected via respective Ethernet links 230 to anetwork 232. For convenience,network 232 is shown as two separate clouds; however, it will be understood that these represent connections to the same network. Optionally, the depiction of separate clouds in the Figures herein may be interpreted as identifying connections to separate networks, regardless to whether they are depicted using the same reference number. - The handling of
packet 228 proceeds as follows, with corresponding operations depicted by encircled numbers inFIG. 2 . First, the packet is received fromnetwork 232 viaEthernet link 230 a at an input port ofNIC 208 a and is buffered (i.e., temporarily stored) in an input buffer. During a second operation, a packet data ofpacket 228 is then copied from the input buffer and written tosystem memory 206 a using a DMA (Direct Memory Access) write operation. This involves forwarding packet data corresponding to the packet fromNIC 208 a via PCIe link 222 a andPCIe interface 214 a tomemory controller 216 a via interconnect 212 a.Memory controller 216 a then writes the packet data tosystem memory 206 a, as depicted at anoperation 2 a. Optionally, the packet data is written toLL Cache 214 a, as depicted by anoperation 2 b. The use of a DMA write operation enables the memory write to be performed independent of an operating system. - As will be understood by one skilled in the networking arts, forwarding packet data from one storage location to another is commonly termed forwarding the packet, regardless of the number of copies of the packet data that are employed. Accordingly, forwarding a packet or forwarding packet data in the detailed description and the claims includes implied data copy operations, as applicable. In addition, forwarding a packet or forwarding packet data may include transferring a portion or portions of the data in the packet, the complete packet, and/or meta-data associated with the packet such as classification results, hash values, etc.
- Depending on the implementation, during a third operation a software entity running on one of
cores 210 a (e.g., a NIC driver) will either detect or be apprised of the arrival ofpacket 228 insystem memory 206 a orLL Cache 214 a and determine that the packet data is actually associated with (i.e., the consumer of the packet is) a consumer software application having a memory allocation including a portion of the memory address space ofsystem memory 206 b on node B. Or more simply, a determination will be made that the packet data is to be written to an address within an address range associated withsystem memory 206 b. Accordingly, during a forth operation the data forpacket 228 is copied fromsystem memory 206 a orLL Cache 214 a to a first buffer on socket-to-socket interface 218 a on node A and sent to socket-to-socket interface 218 b on node B overinterconnect link 226, where it is received in a second buffer. Depending on the implementation, the buffered copy ofpacket 228 is either then written tosystem memory 206 b orLL Cache 214 b during a fifth operation, as depicted byoperations system memory 214 b orLL Cache 214 b. - The foregoing conventional approach requires the involvement of several hardware components and software entities, increasing processor overhead and interconnect traffic, resulting in additional packet transfer latency and cache pollution. These problems are addressed by embodiments of the present invention, as now discussed. Under the embodiments disclosed herein, packets received at a NIC for a first node that are destined to be written to system memory or cache corresponding to a second node are detected at the NIC of the first node and transferred to the NIC for the second node over a high-speed NIC-to-NIC interconnect between the NICs. Once the packet is received at the second NIC, it is handled as if the packet was received from the network via the second NIC to begin with. Thus, this process is transparent to other components and software entities running on both the first and second node.
-
FIG. 3 shows a NUMAmulti-processor platform architecture 300 that is configured to implement the foregoing approach, according to one embodiment. Aspects ofplatform architecture 300 are similar toplatform architecture 200, where like-numbered components perform similar operations, while augmented and new components are shown in light gray. This includes modifications to NICs 308 a and 308 b (NIC A and NIC B) to make them NUMA aware, as well as new components comprising a NIC-to-NIC interconnect 302coupling NICs FIG. 3 further shows how data corresponding to apacket 228 that is received by a NIC at a first node (A) but contains data that is to be written to a memory resource on a second node (B) is handled underNUMA platform architecture 300. - Under the
FIG. 3 embodiment, handling ofpacket 228 proceeds as follows. As before, the packet is received atNIC 308 a fromnetwork 232 viaEthernet link 230 a and is buffered in an input buffer on the NIC during a first operation. During a second operation corresponding to an initial packet processing operation performed byNIC 308 a, the packet header and/or packet payload forpacket 228 is inspected, and applicable filter match input data is generated based on packet header field data and/or packet payload data, which is then employed by packet processing logic onNIC 308 a as an input lookup against filters and/or rules defined in node filter table 304 that is maintained insystem memory 206 a. As detailed below, the filters and/or rules in node filter table 304 enables the node of the memory resource to which applicable packet data is to be written to be identified. In this example, the memory resource node is node B, which has acorresponding NIC 308 b. Accordingly, during a third operation, data corresponding to a copy ofpacket 228 is forwarded from the input buffer (or another buffer) onNIC 308 a to an input buffer onNIC 308 b. The copy ofpacket 228 is then processed during a fourth operation as if it was received byNIC 308 b fromnetwork 232 viaEthernet 230 b, resulting in a copy ofpacket 228 being written using a DMA write operation to eithersystem memory 206 b (operation 4 a) orLL Cache 215 b (operation 4 b). - The packet handling technique illustrated for
NUMA platform architecture 300 enhances platform performance by eliminating the copying of the packet into system memory of the incorrect node (A) and then having to forward the packet data to its appropriate destination node (B) over the socket-to-socket interconnect 226. This reduces both the bandwidth consumption on the various interconnects that were employed under the conventionalNUMA platform architecture 200, as well as packet handling operations performed by an applicable software entity or entities on the incorrect node. -
FIG. 4 shows further details ofNICs system memories NUMA platform architecture 300, according to one embodiment. Each ofNICs input buffer 404 and filter table registers 406. Each NIC further includes packet processing logic 408, aDMA engine 410, a processor interconnect interface 412, and a NIC-to-NIC interconnect interface 414. In one embodiment,NICs - Node
A system memory 206 a includes various portions of memory address space that is allocated to store corresponding data. These include a portion of memory address space allocated to software applications A and B, and a node filter table 304 a. Similarly, nodeB system memory 206 b includes memory address space allocated to a NIC B device driver, a software application C, and a node filter table 304 b. Although not shown for clarity, a portion of the memory address space for each ofsystem memory -
FIG. 4 a depicts aNIC 308 b′ illustrating further details of a Node 308 inFIG. 4 , according to one embodiment.MAC 402 b′ is further depicted to include a secondFIFO input buffer 418 b to which a PHY 420 a of NIC-to-NIC interconnect interface 414 a is coupled.Packet processing logic 408 b includes apacket classifier 422 b and ahash engine 424 b. In addition,FIG. 4 a shows further details of optional nodefilter table cache 416 b. -
FIG. 4 b shows further details of aNIC 309 b, according to one embodiment. Generally,NIC 309 b is similar toNIC 308 b′ ofFIG. 4 a, but with the further addition of a second NIC-to-NIC interface 415 b including aPHY 421 b and aFIFO input buffer 419 b. Accordingly, under this configuration each NIC-to-NIC interconnect interface is operatively coupled to a respective FIFO input buffer. Extending this approach, a NIC may have a third NIC-to-NIC interface including a PHY that is operatively coupled to a third FIFO input buffer (not shown). As an optional approach, a NIC-to-NIC interconnect interface may be multiplexed either before or after a PHY, enabling a single NIC-to-NIC interconnect interface to be shared for multiple NIC-to-NIC interconnect links. -
FIG. 4 b also depicts anoperating system 432 including aNIC B driver 434, andsoftware applications NIC B driver 434 operates as an interface betweenoperating system 432 and the NIC associated with the node the operating system is running on, e.g., NIC B inFIG. 4 b. NIC driver 434 B may also be configured to update one or more of nodefilter table cache 416 b,packet processing logic 408 b, and filter table registers 406 b. - With reference to
FIGS. 4 , 4 a, 4 b, and flowchart 500 ofFIG. 5 , processing of apacket 228 proceeds as follows. In ablock 502 of flowchart 500, packet data including apacket header 426 andpayload 428 is received in the form of an analog modulated Ethernet signal at an input port 430 that is communicatively coupled toPHY 400 a ofNIC 308 a. The PHY is used to convert the Ethernet signal via which packet data is conveyed from an analog signal to a digital data stream, and then deserialize the data stream, forwarding the deserialized digital data as blocks of data to MAC block 402. MAC block 402 performs additional processing, including detection of the start and end of packets to extract packet data that is buffered inFIFO input buffer 404 a. - When the packet advances to the top of the FIFO queue,
packet header 426 and/orpacket payload 428 are inspected, and the packet is classified in ablock 504 by packet classifier 422. There are various schemes well known in the art that may be used for packet classification, such as a 5-tuple classification of an Internet Protocol (IP) packet header (source address, destination address, source port, destination port, and transport protocol), or classification based on one or more header fields. As another option, data corresponding topacket payload 428 may also be considered during packet classification. Under some protocols, packet classification data may be encoded in a packet header or the packet payload. In one embodiment, packet classification is implemented by performing a hash on one or more header fields via hash engine 424. For example, in the 5-tuple example a hash may be performed on a concatenation of the 5-tuple IP field values. More generally, hash engine 424 may also be used to generate a hash value based on data in any combination of packet header fields and packet payload data. - The result of packet classification is the generation of a filter match input value (i.e., filter match input data) that is used as a lookup input to a lookup table comprising node filter table 304 a or a packet flow table including node filter columns, as depicted in a
block 506. For example, rather than employ a separate node filter table, the corresponding data for the NIC column and/or Node column could be included in a packet flow table that is used to associate packets with particular header and/or payload data to corresponding packet flows. Since flow table filters and rules are typically derived via packet classification, a conventional packet flow table could be modified to include additional columns corresponding to the illustrated NIC and Node columns. - Returning to flowchart 500, following
block 506 the flowchart logic proceeds to adecision block 508, which is depicted in dashed outline to indicate this is optional. For embodiments that employ a node filter table cache, a lookup into nodefilter table cache 416 a is first made to determine whether there is a matching node filter table entry. The use of a cached node filter table (or a cached packet flow table including node filter columns) may be advantageous under various server implementations. For example, it is common to classify packets associated with a client-server connection session into corresponding flows based on a combination of header field values, such as a 5-tuple hash. In cases where there is a large amount of data being transferred for a given connection, such as an upload of a file to a server, a significant portion of packets arriving at the NIC used for the connection will be classified to the same flow while the upload is being performed (since the 5-tuple values for the received packets will be the same). Thus, by caching corresponding node filter table or packet flow table entries, a match may be found in local memory on the NIC as opposed to having to perform a look up against a node filter table or forwarding table in system memory. - If a match is found in the cache as depicted by
decision block 508, the flowchart logic proceeds todecision block 510. If not, the logic proceeds to adecision block 512 in which a look up against a node filter table or packet flow table in system memory (e.g., node filter table 304 a) is performed. (Although depicted in the drawings herein in system memory for simplicity, if is possible that data corresponding to a node filter table or packet flow table may actually be accessed from a processor memory cache under some embodiments.) If a match is found, the logic proceeds to anoptional block 514 in which an entry in the nodefilter table cache 416 a is added. When the cache is full, each added entry requires an eviction of a current entry, which may be selected using any well-known cache eviction policy, such as least recently used (LRU), least frequency used (LFU), or FIFO, for example. The logic then proceeds todecision block 510. - If the match lookup of node filter table 304 a (or a forwarding table including node filter table data) results in a miss, a new node filter entry including corresponding filter match value and/or rule is generated in a
block 516. This process may typically be handled by a NIC driver or other entity that is employed for managing the node filter table for the node. Generally, the new entry will include a match filter or rule and identification to which node the classified flow is assigned to. Under some implementations, a connection setup and teardown feature may be employed that enables applicable packet classification information to be determined before any packets are transferred. Accordingly, under this approach a new node filter table entry or packet flow table entry may be added during the connection setup operation, rather than in response to a lookup table miss. - In one embodiment, global copies of the node filter table are maintained across all of the nodes on a platform. Optionally, copies of node filter tables may be shared among neighboring nodes. Accordingly, in an optional block 518 a copy of the new entry is forwarded to other node(s), as applicable. The new entry may also be added to node
filter table cache 416 a if this is implemented and depending on the cache replacement policy. As before, the logic then proceeds todecision block 510. - At decision block 510 a determination is made to whether the node corresponding to the NIC that receives the packet from the network (e.g., Node A in this example) is the correct node for handling the packet. More particularly,
decision block 510 determines whether the memory resource to which the packet is to be written is associated with the current node handling the packet or another node. If the result is YES, the packet is handled in the conventional manner on the current node, with applicable packet data being written to an applicable destination address in system memory or LL Cache using a DMA write operation. - If the node receiving the packet is not the correct node, the logic proceeds to a
block 522 in which a determination of the NIC via which the correct node can be accessed is made, followed by forwarding a copy of the packet data from the first NIC to that (second) NIC via one or more NIC-to-NIC interconnect links between the first and second NICs. In the example illustrated inFIG. 4 , a copy of the packet data would be forwarded fromNIC 308 a toNIC 308 b via NIC-to-NIC Interconnect 302. As explained in further detail below, in situations where there are more than two platform nodes, forwarding a packet from a first NIC to a second NIC may traverse more than one NIC-to-NIC interconnect link under some embodiments. - As depicted by a
block 524, the packet is received at the NIC-to-NIC interconnect interface of the correct (second) node, and the packet is buffered in an input buffer. With reference toFIG. 4 a, the packet is received and processed byPHY 420 b and then buffered inFIFO input buffer 418 b. As an optional implementation, the packet could be buffered in the same buffer as the packets received at aninput port 430 b forNIC 308 b′ using a multiplexing scheme that multiplexes inputs fromPHY 400 b andPHY 420 b (not shown). In addition to the configuration shown, a NIC may have a PHY or PHYs coupled to multiple input ports, each with one or more input buffers and multiplexers (as applicable). Generally, one of these buffers could be shared with a NIC-to-NIC interconnect interface, or a separate input buffer or buffers may be used. - Once the packet is buffered in the applicable input buffer of the second NIC (e.g.,
FIFO input buffer 418 b), the logic returns to block 504, and further handling of the packet proceeds as if the packet was originally received at the correct NIC to begin with. In the example illustrated inFIGS. 4 and 4 a, the packet data is written to a memory location specified by a memory address associated with the NIC B driver via a DMA write operation facilitated byDMA engine 410 b and then the packet payload is copied to a memory address associated with the software consumer of the packet, depicted as software application C. Optionally, the packet payload may be copied directly to an applicable memory address allocated to the software consumer application, or to an applicable address in the LL Cache for the second Node (B). - In addition to performing packet classification using a hash function or the like and then using the resulting hash value as a lookup data input against a node filter lookup table, some protocols may facilitate identification of the memory address to which a packet is to be written directly. For example, for protocols such as Fiber Channel over Ethernet (FCOE) that support Direct Data Placement (DDP), there is sufficient data in the packet for the NIC to determine the desired final destination (i.e., memory address) for the packet. Accordingly, in some embodiments a node filter lookup table may include filter and/or rule data that considers a memory address that is determined during packet processing, with corresponding table entries identifying nodes and/or NICs that packet is to be forwarded to. For example, a filter column may include address range maps to corresponding nodes and/or NICs.
- In general, aspects of the embodiments disclosed herein may be implemented for NUMA platform architecture employing two or more nodes. An exemplary four-node
NUMA platform architecture 600 is shown inFIG. 6 , includingnodes respective processor 304, memory 206, and NIC 309. Although the socket-to-socket interconnect interfaces 218 onprocessors processors embodiment processors 304 a-d have similar configurations; and the use of mirroring the interfaces inFIG. 6 is used to make the interconnect routing simpler and easier to understand. Similarly, althoughnodes nodes - In
platform architecture 600, each ofprocessors 304 a-d is operatively coupled to amain board 602 via a respective socket (not shown) or otherwise mounted tomain board 602 via an appropriate bonding scheme, such as flip-chip bounding. Optionally, the components for a given node may be mounted or otherwise operatively coupled to a daughter board, which in turn is coupled via one or more connectors to a main board. In one embodiment,main board 602 includes wiring traces to facilitateinterconnects 226 xy between pairs of socket-to-socket interfaces 318, wherein x and y correspond to the respective socket nodes for the processors being linked in communication via the interconnect. For example, interconnect 226 ac is an interconnect between socket-to-socket interfaces 318 a and 318 c. In one embodiment, there are four socket-to-socket interconnects labeled 226 ac, 226 cd, 226 bd, and 226 ab. Under an optional configuration, there are additional socket-to-socket interconnects 226 ad and 226 bc. - As stated above,
platform architecture 600 includes four NICs 309 a-d, which are also labeled NIC A, NIC B, NIC C, and NIC D. In one embodiment, as depicted by solid connector lines, each of NICs 309 a-d is connected to two other NICs via a NIC-to-NIC interconnect 302 nm, where m and n identify the two NICs that are interconnected. These NIC-to-NIC interconnects includeinterconnects 302 ab, 302 ac, 302 cd, and 302 bd. Under an optional configuration, each NIC is interconnected to the other three NICs, as depicted by additional interconnects shown with dashed lines, labeled 302 ad and 302 bc. - In general, the physical structure of a NIC-to-NIC interconnect may comprise one or more wired or optical pathways between a pair of NICs (forming a pathway between to NIC-to-NIC interface PHYs). In one embodiment, wired pathways are provided by wire trances in
main board 602. In one embodiment employing an optical pathway, an optical cable is coupled to the NIC-to-NIC interfaces via appropriate optical connectors (not shown). As another option, a wire cable may be used for the wired pathway. - Generally, a wired NIC-to-NIC interconnect link may be implemented over a single path or multiple paths or “lanes” routed in parallel. For example, in one embodiment a PCIe link is used, such as a PCIe 1x, 2x, 4x, 8x, or 16x link, were the n in nx corresponds to the number of lanes. Other types of high-speed serial interconnects and associated protocols may also be used, including but not limited to Intel's QPI or Keizer Technology Interconnect® (KTI), or an Open Core Protocol (OCP) interconnect.
- In accordance with one embodiment of
platform architecture 600, NICs A-D may be interconnected to form a ring-type interconnect. This approach may be extended to link larger numbers of NICs, as shown in further detail inFIG. 7 , wherein the top portion of the Figure depicts a plurality of NUMA aware NICs 700 (also labeled NIC A, NIC B, etc.) interconnected via NIC-to-NIC interconnect links 302 x-y. The lower portion ofFIG. 7 shows details of aNIC 700, according to one embodiment.NIC 700 includes a first NIC-to-NIC interconnect interface 702 including aPHY 704 to which aninput buffer 706 and anoutput buffer 708 is coupled.NIC 700 also includes a similarly configured NIC-to-NIC interconnect interface 710 including aPHY 712 coupled to aninput buffer 714 and anoutput buffer 716. As further shown,input buffer 706 is connected tooutput buffer 716 andoutput buffer 708 is connected to inputbuffer 714. - In one embodiment, NIC-to-NIC interconnect interfaces 702 and 710 are configured to implement an interconnect protocol employing node-based addressing, where each
NIC 700 represents a ring stop node on the ring interconnect. Accordingly, packet data may be forwarded between anyNIC 700 to anyother NIC 700 via the ring interconnect. Moreover, in one embodiment the NIC-to-NIC interconnect interfaces 702 and 710 are configured to bypass the FIFO input buffers when packet data is forwarded to aNIC 700 that is an intermediate node along a path between a forwarding NIC node and NIC node that is targeted for receiving the forwarded packet data. For example, suppose a packet is to be forwarded from NIC A to NIC N−1 via NIC N. When the packet data arrives at NIC N, logic in NIC-to-NIC interconnect interface 702 inspects the destination node address for the packet data and determines 1) that NIC Nis not the destination NIC node, and 2) that the destination NIC node (NIC N−1) can be reached by forwarding the packet outbound from NIC-to-NIC interconnect interface 710. Accordingly, when the packet data is received at NIC N it is buffered ininput buffer 706 and then forwarded tooutput buffer 716, whereupon the packet data is temporarily stored until it can be send outbound NIC-to-NIC interconnect interface 710. In this manner, packet data can be forwarded between any NIC nodes on the ring without having to be processed by packet processing logic on the intermediate NIC nodes along the forwarding portion of the ring. - In addition to a ring-type configuration, the NIC-to-NIC interconnects may be configured to support other node configuration. For example, star-type node configuration and hierarchical node configuration may be implemented with NIC-to-NIC interconnects and corresponding forwarding logic on the NICs.
- The foregoing embodiment provide enhanced performance over the conventional NUMA approach under which the NICs are agnostic to being deployed in a NUMA platform architecture. Conversely, since the NICs are configured to be NUMA aware, they are enabled to forward packet data to the correct node for handling the data for situations where they are not associated with the correct node to begin with. This approach avoids additional processing operation and interconnect usage under the conventional approach, such as reducing QPI traffic employed needed for cache coherency. For Direct Data IO implementations, it also increases the effectiveness of DDIO since packet data may be directly written to the correct processor's last level cache.
- While the embodiments depicted herein are referred to as NUMA architectures, the teachings and principles may be applied to other non-NUMA architectures under which forwarding of packets between NICs via NIC-to-NIC interconnects are implemented. Generally, such architectures will include multiple nodes, each including one or more processors and/or processor cores including local caches. Under some architectures, system memory may be configured such that it is accessed in a distributed manner across nodes, wherein at least a portion of the memory is not associated with a particular node. Similarly, a last level cached or the like may be configured in a distributed manner across nodes.
- The NIC-to-NIC forwarding feature may be implemented in a similar manner under such non-NUMA architectures as described in the embodiments herein. Under these architectures NIC-to-NIC forwarding may be implemented such that the packet data may be accessed by the processor or processor core running the software application that is a consumer of the packet data. Accordingly, the forwarding table data is configured in consideration of the location of the processor or processor core associated with a software consumer (for a given packet).
- In addition to supporting connection to networks via wired Ethernet links, embodiments of the invention may be configured to support network connections via wireless links, such as via wireless connections over an IEEE 802.11-based (aka WiFi) wireless link, a WiMAX link, or various mobile telecommunication-based data links, e.g., GSM, CDMA, GPRS, W-CDMA, EDGE, CDMA2000, UMTS, HSUPA, HSDPA, EV-DO, LTE, etc. To support a wireless network link, a NIC may be configured to process corresponding radio frequency (RF) signal data. Optionally, RF signal data processing may be implemented via a separate component or components that are communicatively coupled to a NIC.
- Although some embodiments have been described in reference to particular implementations, other implementations are possible according to some embodiments. Additionally, the arrangement and/or order of elements or other features illustrated in the drawings and/or described herein need not be arranged in the particular way illustrated and described. Many other arrangements are possible according to some embodiments.
- In each system shown in a figure, the elements in some cases may each have a same reference number or a different reference number to suggest that the elements represented could be different and/or similar. However, an element may be flexible enough to have different implementations and work with some or all of the systems shown or described herein. The various elements shown in the figures may be the same or different. Which one is referred to as a first element and which is called a second element is arbitrary.
- In the description and claims, the terms “coupled,” operatively coupled,” and “connected,” along with their derivatives, may be used. It should be understood that these terms are not intended as synonyms for each other. Rather, in particular embodiments, “connected” may be used to indicate that two or more elements are in direct physical or electrical contact with each other. “Coupled” may mean that two or more elements are in direct physical or electrical contact. However, “coupled” may also mean that two or more elements are not in direct contact with each other, but yet still co-operate or interact with each other, while “operatively coupled” may mean that two or more elements are directly or indirectly connected when a component or device is operating.
- An embodiment is an implementation or example of the inventions. Reference in the specification to “an embodiment,” “one embodiment,” “some embodiments,” or “other embodiments” means that a particular feature, structure, or characteristic described in connection with the embodiments is included in at least some embodiments, but not necessarily all embodiments, of the inventions. The various appearances “an embodiment,” “one embodiment,” or “some embodiments” are not necessarily all referring to the same embodiments.
- Not all components, features, structures, characteristics, etc. described and illustrated herein need be included in a particular embodiment or embodiments. If the specification states a component, feature, structure, or characteristic “may”, “might”, “can” or “could” be included, for example, that particular component, feature, structure, or characteristic is not required to be included. If the specification or claim refers to “a” or “an” element, that does not mean there is only one of the element. If the specification or claims refer to “an additional” element, that does not preclude there being more than one of the additional element.
- Various aspects of the embodiments herein may be facilitated by corresponding software and/or firmware components and applications, such as software running on a server or firmware executed by an embedded processor on a network element. Thus, embodiments of this invention may be used as or to support a software program, software modules, firmware, and/or distributed software executed upon some form of processing core (such as the CPU of a computer, one or more cores of a multi-core processor), a virtual machine running on a processor or core or otherwise implemented or realized upon or within a machine-readable medium. A machine-readable medium includes any mechanism for storing or transmitting information in a form readable by a machine (e.g., a computer). For example, a machine-readable medium may include a read only memory (ROM); a random access memory (RAM); a magnetic disk storage media; an optical storage media; and a flash memory device, etc.
- The above description of illustrated embodiments of the invention, including what is described in the Abstract, is not intended to be exhaustive or to limit the invention to the precise forms disclosed. While specific embodiments of, and examples for, the invention are described herein for illustrative purposes, various equivalent modifications are possible within the scope of the invention, as those skilled in the relevant art will recognize.
- These modifications can be made to the invention in light of the above detailed description. The terms used in the following claims should not be construed to limit the invention to the specific embodiments disclosed in the specification and the drawings. Rather, the scope of the invention is to be determined entirely by the following claims, which are to be construed in accordance with established doctrines of claim interpretation.
Claims (30)
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US13/663,131 US9047417B2 (en) | 2012-10-29 | 2012-10-29 | NUMA aware network interface |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US13/663,131 US9047417B2 (en) | 2012-10-29 | 2012-10-29 | NUMA aware network interface |
Publications (2)
Publication Number | Publication Date |
---|---|
US20140122634A1 true US20140122634A1 (en) | 2014-05-01 |
US9047417B2 US9047417B2 (en) | 2015-06-02 |
Family
ID=50548465
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US13/663,131 Active 2033-02-23 US9047417B2 (en) | 2012-10-29 | 2012-10-29 | NUMA aware network interface |
Country Status (1)
Country | Link |
---|---|
US (1) | US9047417B2 (en) |
Cited By (21)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20140254594A1 (en) * | 2013-03-05 | 2014-09-11 | Iosif Gasparakis | In nic flow switching |
US20140369348A1 (en) * | 2013-06-17 | 2014-12-18 | Futurewei Technologies, Inc. | Enhanced Flow Entry Table Cache Replacement in a Software-Defined Networking Switch |
US20150249606A1 (en) * | 2014-02-28 | 2015-09-03 | Cisco Technology, Inc. | Smarter policy decisions based on metadata in data flows |
US20150347410A1 (en) * | 2014-06-03 | 2015-12-03 | Sap Ag | Cached Views |
US20160127493A1 (en) * | 2014-11-04 | 2016-05-05 | Qlogic Corporation | Caching methods and systems using a network interface card |
US20160285971A1 (en) * | 2014-12-05 | 2016-09-29 | Foundation For Research And Technology - Hellas (Forth) | Network Storage Protocol and Adaptive Batching Apparatuses, Methods, and Systems |
US20170131915A1 (en) * | 2014-05-09 | 2017-05-11 | Micron Technology, Inc. | Interconnect systems and methods using hybrid memory cube links to send packetized data over different endpoints of a data handling device |
US20170139744A1 (en) * | 2015-11-18 | 2017-05-18 | Freescale Semiconductor, Inc. | Systems and methods for frame presentation and modification in a networking environment |
CN107566302A (en) * | 2017-10-10 | 2018-01-09 | 新华三技术有限公司 | Message forwarding method and device |
CN107948072A (en) * | 2017-10-31 | 2018-04-20 | 新华三技术有限公司 | Message forwarding method and device |
US20190068698A1 (en) * | 2017-08-30 | 2019-02-28 | Intel Corporation | Technologies for providing efficient pooling for a hyper converged infrastructure |
US20200036636A1 (en) * | 2018-07-25 | 2020-01-30 | Vmware, Inc. | Selection of paired transmit queue |
US10686716B2 (en) | 2018-07-23 | 2020-06-16 | Vmware, Inc. | Dynamic processing of packets using multiple receive queue features |
US10708158B2 (en) * | 2015-04-10 | 2020-07-07 | Hewlett Packard Enterprise Development Lp | Network address of a computing device |
US10735341B2 (en) | 2018-04-26 | 2020-08-04 | Nicira, Inc. | Dynamic provisioning of multiple RSS engines |
US20210352048A1 (en) * | 2020-05-08 | 2021-11-11 | Gigamon Inc. | One-Armed Inline Decryption/Encryption Proxy Operating in Transparent Bridge Mode |
US20220121583A1 (en) * | 2020-10-19 | 2022-04-21 | Softiron Limited | Acceleration of Data Between a Network and Local I/O in a NUMA System |
US11394666B2 (en) * | 2017-12-18 | 2022-07-19 | Intel Corporation | Scalable communication with a packet processing unit |
WO2022265552A1 (en) * | 2021-06-14 | 2022-12-22 | Telefonaktiebolaget Lm Ericsson (Publ) | System, method, and apparatus for fine-grained control of input/output data placement in a memory subsystem |
US20230006930A1 (en) * | 2021-07-02 | 2023-01-05 | Realtek Semiconductor Corp. | Method and system for processing data flow with incomplete comparison process |
US11621915B2 (en) | 2020-07-07 | 2023-04-04 | Huawei Technologies Co., Ltd. | Packet forwarding method, route sending and receiving method, and apparatus |
Families Citing this family (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US10684973B2 (en) | 2013-08-30 | 2020-06-16 | Intel Corporation | NUMA node peripheral switch |
CN107534576B (en) * | 2015-06-27 | 2020-03-31 | 华为技术有限公司 | Link recovery method and network equipment |
CN106559838B (en) * | 2015-09-24 | 2019-12-06 | 大唐移动通信设备有限公司 | business processing optimization method and device |
US10681189B2 (en) * | 2017-05-18 | 2020-06-09 | At&T Intellectual Property I, L.P. | Terabit-scale network packet processing via flow-level parallelization |
Citations (15)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6295280B1 (en) * | 1997-03-05 | 2001-09-25 | Hyundai Electronics Industries Co., Ltd. | Method for network node recognition |
US6308282B1 (en) * | 1998-11-10 | 2001-10-23 | Honeywell International Inc. | Apparatus and methods for providing fault tolerance of networks and network interface cards |
US6662276B2 (en) * | 2000-12-29 | 2003-12-09 | Intel Corporation | Storing directory information for non uniform memory architecture systems using processor cache |
US20060206489A1 (en) * | 2005-03-11 | 2006-09-14 | International Business Machines Corporation | System and method for optimally configuring software systems for a NUMA platform |
US7639624B2 (en) * | 2004-07-23 | 2009-12-29 | Hewlett-Packard Development Company, L.P. | Method and system for monitoring network connectivity |
US20100020818A1 (en) * | 2008-07-24 | 2010-01-28 | International Business Machines Corporation | Sharing buffer space in link aggregation configurations |
US7693045B2 (en) * | 2004-12-29 | 2010-04-06 | Hewlett-Packard Development Company, L.P. | Verifying network connectivity |
US20100217949A1 (en) * | 2009-02-24 | 2010-08-26 | International Business Machines Corporation | Dynamic Logical Partition Management For NUMA Machines And Clusters |
US20110131328A1 (en) * | 2009-11-30 | 2011-06-02 | Neil Horman | Network buffer allocations based on consumption patterns |
US8134928B1 (en) * | 2005-12-15 | 2012-03-13 | Nvidia Corporation | Technique for identifying a failed network interface card within a team of network interface cards |
US20120072624A1 (en) * | 2010-09-17 | 2012-03-22 | Oracle America, Inc. | Numa i/o framework |
US20120265801A1 (en) * | 2011-04-13 | 2012-10-18 | Microsoft Corporation | Out of order assembling of data packets |
US20130055270A1 (en) * | 2011-08-26 | 2013-02-28 | Microsoft Corporation | Performance of multi-processor computer systems |
US20140029617A1 (en) * | 2012-07-27 | 2014-01-30 | Ren Wang | Packet processing approach to improve performance and energy efficiency for software routers |
US8694618B2 (en) * | 2011-04-13 | 2014-04-08 | Microsoft Corporation | Maximizing data transfer through multiple network devices |
Family Cites Families (82)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US3749845A (en) | 1971-08-27 | 1973-07-31 | Bell Telephone Labor Inc | Digital data communication system |
US5875288A (en) | 1990-02-05 | 1999-02-23 | Scitex Corporation Ltd. | Apparatus and techniques for computerized printing |
US5398245A (en) | 1991-10-04 | 1995-03-14 | Bay Networks, Inc. | Packet processing method and apparatus |
US5243699A (en) | 1991-12-06 | 1993-09-07 | Maspar Computer Corporation | Input/output system for parallel processing arrays |
DE69635880T2 (en) | 1995-09-18 | 2006-10-05 | Kabushiki Kaisha Toshiba, Kawasaki | System and method for the transmission of parcels, suitable for a large number of entrance gates |
US6021263A (en) | 1996-02-16 | 2000-02-01 | Lucent Technologies, Inc. | Management of ATM virtual circuits with resources reservation protocol |
US5781549A (en) | 1996-02-23 | 1998-07-14 | Allied Telesyn International Corp. | Method and apparatus for switching data packets in a data network |
US6434620B1 (en) | 1998-08-27 | 2002-08-13 | Alacritech, Inc. | TCP/IP offload network interface device |
US6658480B2 (en) | 1997-10-14 | 2003-12-02 | Alacritech, Inc. | Intelligent network interface system and method for accelerated protocol processing |
US6226680B1 (en) | 1997-10-14 | 2001-05-01 | Alacritech, Inc. | Intelligent network interface system method for protocol processing |
US5937169A (en) | 1997-10-29 | 1999-08-10 | 3Com Corporation | Offload of TCP segmentation to a smart adapter |
US6216167B1 (en) | 1997-10-31 | 2001-04-10 | Nortel Networks Limited | Efficient path based forwarding and multicast forwarding |
US6351474B1 (en) | 1998-01-14 | 2002-02-26 | Skystream Networks Inc. | Network distributed remultiplexer for video program bearing transport streams |
US6246683B1 (en) | 1998-05-01 | 2001-06-12 | 3Com Corporation | Receive processing with network protocol bypass |
US6157955A (en) | 1998-06-15 | 2000-12-05 | Intel Corporation | Packet processing system including a policy engine having a classification unit |
US6434115B1 (en) | 1998-07-02 | 2002-08-13 | Pluris, Inc. | System and method for switching packets in a network |
JP3077677B2 (en) | 1998-07-14 | 2000-08-14 | 日本電気株式会社 | Quality assurance node equipment |
US6389468B1 (en) | 1999-03-01 | 2002-05-14 | Sun Microsystems, Inc. | Method and apparatus for distributing network traffic processing on a multiprocessor computer |
US6453360B1 (en) | 1999-03-01 | 2002-09-17 | Sun Microsystems, Inc. | High performance network interface |
US6650640B1 (en) | 1999-03-01 | 2003-11-18 | Sun Microsystems, Inc. | Method and apparatus for managing a network flow in a high performance network interface |
US6483804B1 (en) | 1999-03-01 | 2002-11-19 | Sun Microsystems, Inc. | Method and apparatus for dynamic packet batching with a high performance network interface |
US6768992B1 (en) | 1999-05-17 | 2004-07-27 | Lynne G. Jolitz | Term addressable memory of an accelerator system and method |
US6728265B1 (en) | 1999-07-30 | 2004-04-27 | Intel Corporation | Controlling frame transmission |
US6597661B1 (en) | 1999-08-25 | 2003-07-22 | Watchguard Technologies, Inc. | Network packet classification |
US6633576B1 (en) | 1999-11-04 | 2003-10-14 | William Melaragni | Apparatus and method for interleaved packet storage |
US6564267B1 (en) | 1999-11-22 | 2003-05-13 | Intel Corporation | Network adapter with large frame transfer emulation |
US6683873B1 (en) | 1999-12-27 | 2004-01-27 | Cisco Technology, Inc. | Methods and apparatus for redirecting network traffic |
US6973040B1 (en) | 2000-03-13 | 2005-12-06 | Netzentry, Inc. | Method of maintaining lists of network characteristics |
US7050437B2 (en) | 2000-03-24 | 2006-05-23 | International Business Machines Corporation | Wire speed reassembly of data frames |
US6574195B2 (en) | 2000-04-19 | 2003-06-03 | Caspian Networks, Inc. | Micro-flow management |
US6778548B1 (en) | 2000-06-26 | 2004-08-17 | Intel Corporation | Device to receive, buffer, and transmit packets of data in a packet switching network |
US6715005B1 (en) | 2000-06-29 | 2004-03-30 | International Business Machines Corporation | Method and system for reducing latency in message passing systems |
US6687905B1 (en) | 2000-08-11 | 2004-02-03 | International Business Machines Corporation | Multiple port input/output job scheduling |
US6718326B2 (en) | 2000-08-17 | 2004-04-06 | Nippon Telegraph And Telephone Corporation | Packet classification search device and method |
US6708292B1 (en) | 2000-08-18 | 2004-03-16 | Network Associates, Inc. | System, method and software for protocol analyzer remote buffer management |
US6665495B1 (en) | 2000-10-27 | 2003-12-16 | Yotta Networks, Inc. | Non-blocking, scalable optical router architecture and method for routing optical traffic |
US7032035B2 (en) | 2000-12-08 | 2006-04-18 | Intel Corporation | Method and apparatus for improving transmission performance by caching frequently-used packet headers |
US6618793B2 (en) | 2000-12-18 | 2003-09-09 | Redback Networks, Inc. | Free memory manager scheme and cache |
US6665755B2 (en) | 2000-12-22 | 2003-12-16 | Nortel Networks Limited | External memory engine selectable pipeline architecture |
JP4427196B2 (en) | 2001-02-07 | 2010-03-03 | 株式会社日立製作所 | IP packet communication apparatus and redundant configuration switching method |
JP2002259352A (en) | 2001-03-01 | 2002-09-13 | Handotai Rikougaku Kenkyu Center:Kk | Multiprocessor system device |
US20020144004A1 (en) | 2001-03-29 | 2002-10-03 | Gaur Daniel R. | Driver having multiple deferred procedure calls for interrupt processing and method for interrupt processing |
EP1382165A2 (en) | 2001-04-13 | 2004-01-21 | MOTOROLA INC., A Corporation of the state of Delaware | Manipulating data streams in data stream processors |
US6816455B2 (en) | 2001-05-09 | 2004-11-09 | Telecom Italia S.P.A. | Dynamic packet filter utilizing session tracking |
US7164680B2 (en) | 2001-06-04 | 2007-01-16 | Koninklijke Philips Electronics N.V. | Scheme for supporting real-time packetization and retransmission in rate-based streaming applications |
US7590143B2 (en) | 2001-07-05 | 2009-09-15 | Qualcomm Incorporated | System and method for voice over IP |
US7305492B2 (en) | 2001-07-06 | 2007-12-04 | Juniper Networks, Inc. | Content service aggregation system |
ATE352150T1 (en) | 2001-08-30 | 2007-02-15 | Tellabs Operations Inc | SYSTEM AND METHOD FOR TRANSMITTING DATA USING A COMMON SWITCHING FIELD |
US7039061B2 (en) | 2001-09-25 | 2006-05-02 | Intel Corporation | Methods and apparatus for retaining packet order in systems utilizing multiple transmit queues |
JP4093741B2 (en) | 2001-10-03 | 2008-06-04 | シャープ株式会社 | External memory control device and data driven information processing device including the same |
US6754800B2 (en) | 2001-11-14 | 2004-06-22 | Sun Microsystems, Inc. | Methods and apparatus for implementing host-based object storage schemes |
US7227870B2 (en) | 2001-11-20 | 2007-06-05 | Broadcom Corporation | Systems including packet interfaces, switches, and packet DMA circuits for splitting and merging packet streams |
US20030108044A1 (en) | 2001-12-11 | 2003-06-12 | Roland Hendel | Stateless TCP/IP protocol |
US6633835B1 (en) | 2002-01-10 | 2003-10-14 | Networks Associates Technology, Inc. | Prioritized data capture, classification and filtering in a network monitoring environment |
US6957281B2 (en) | 2002-01-15 | 2005-10-18 | Intel Corporation | Ingress processing optimization via traffic classification and grouping |
US7080308B2 (en) | 2002-03-22 | 2006-07-18 | Intel Corporation | Method and apparatus to perform error control |
US7342894B2 (en) | 2002-04-03 | 2008-03-11 | Qualcomm Incorporated | System and method for transparent Mobile IP registration within PPP negotiation |
US7496689B2 (en) | 2002-04-22 | 2009-02-24 | Alacritech, Inc. | TCP/IP offload device |
US7373663B2 (en) | 2002-05-31 | 2008-05-13 | Alcatel Canada Inc. | Secret hashing for TCP SYN/FIN correspondence |
US7284272B2 (en) | 2002-05-31 | 2007-10-16 | Alcatel Canada Inc. | Secret hashing for TCP SYN/FIN correspondence |
US7944920B2 (en) | 2002-06-11 | 2011-05-17 | Pandya Ashish A | Data processing system using internet protocols and RDMA |
US20030231657A1 (en) | 2002-06-12 | 2003-12-18 | Kacheong Poon | System and method for a multi-data network layer transmit interface |
US7080379B2 (en) | 2002-06-20 | 2006-07-18 | International Business Machines Corporation | Multiprocessor load balancing system for prioritizing threads and assigning threads into one of a plurality of run queues based on a priority band and a current load of the run queue |
US7142540B2 (en) | 2002-07-18 | 2006-11-28 | Sun Microsystems, Inc. | Method and apparatus for zero-copy receive buffer management |
US7162740B2 (en) | 2002-07-22 | 2007-01-09 | General Instrument Corporation | Denial of service defense by proxy |
US6968358B2 (en) | 2002-07-25 | 2005-11-22 | International Business Machines Corporation | Method and apparatus for network communication card memory management |
US7308000B2 (en) | 2002-11-07 | 2007-12-11 | Sun Microsystems, Inc. | Methods and systems for efficient multi-packet data processing in a layered network protocol |
US7043494B1 (en) | 2003-01-28 | 2006-05-09 | Pmc-Sierra, Inc. | Fast, deterministic exact match look-ups in large tables |
US7835397B2 (en) | 2003-04-25 | 2010-11-16 | Alcatel-Lucent Usa Inc. | Frame processing |
US7219228B2 (en) | 2003-08-25 | 2007-05-15 | Lucent Technologies Inc. | Method and apparatus for defending against SYN packet bandwidth attacks on TCP servers |
US7586925B2 (en) | 2003-09-09 | 2009-09-08 | Sonus Networks, Inc. | Data adaptation protocol |
US7188198B2 (en) | 2003-09-11 | 2007-03-06 | International Business Machines Corporation | Method for implementing dynamic virtual lane buffer reconfiguration |
US6981074B2 (en) | 2003-10-14 | 2005-12-27 | Broadcom Corporation | Descriptor-based load balancing |
US20050249228A1 (en) | 2004-05-05 | 2005-11-10 | Linden Cornett | Techniques for providing scalable receive queues |
US20060004933A1 (en) | 2004-06-30 | 2006-01-05 | Sujoy Sen | Network interface controller signaling of connection event |
US20060031474A1 (en) | 2004-07-19 | 2006-02-09 | Linden Cornett | Maintaining reachability measures |
US8098676B2 (en) | 2004-08-12 | 2012-01-17 | Intel Corporation | Techniques to utilize queues for network interface devices |
US20060064508A1 (en) | 2004-09-17 | 2006-03-23 | Ramesh Panwar | Method and system to store and retrieve message packet data in a communications network |
US7620046B2 (en) | 2004-09-30 | 2009-11-17 | Intel Corporation | Dynamically assigning packet flows |
US7620071B2 (en) | 2004-11-16 | 2009-11-17 | Intel Corporation | Packet coalescing |
US7404040B2 (en) | 2004-12-30 | 2008-07-22 | Intel Corporation | Packet data placement in a processor cache |
US8311059B2 (en) | 2005-09-07 | 2012-11-13 | Emulex Design & Manufacturing Corporation | Receive coalescing and automatic acknowledge in network interface controller |
-
2012
- 2012-10-29 US US13/663,131 patent/US9047417B2/en active Active
Patent Citations (16)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6295280B1 (en) * | 1997-03-05 | 2001-09-25 | Hyundai Electronics Industries Co., Ltd. | Method for network node recognition |
US6308282B1 (en) * | 1998-11-10 | 2001-10-23 | Honeywell International Inc. | Apparatus and methods for providing fault tolerance of networks and network interface cards |
US6662276B2 (en) * | 2000-12-29 | 2003-12-09 | Intel Corporation | Storing directory information for non uniform memory architecture systems using processor cache |
US7639624B2 (en) * | 2004-07-23 | 2009-12-29 | Hewlett-Packard Development Company, L.P. | Method and system for monitoring network connectivity |
US7693045B2 (en) * | 2004-12-29 | 2010-04-06 | Hewlett-Packard Development Company, L.P. | Verifying network connectivity |
US20060206489A1 (en) * | 2005-03-11 | 2006-09-14 | International Business Machines Corporation | System and method for optimally configuring software systems for a NUMA platform |
US8134928B1 (en) * | 2005-12-15 | 2012-03-13 | Nvidia Corporation | Technique for identifying a failed network interface card within a team of network interface cards |
US20100020818A1 (en) * | 2008-07-24 | 2010-01-28 | International Business Machines Corporation | Sharing buffer space in link aggregation configurations |
US20100217949A1 (en) * | 2009-02-24 | 2010-08-26 | International Business Machines Corporation | Dynamic Logical Partition Management For NUMA Machines And Clusters |
US20110131328A1 (en) * | 2009-11-30 | 2011-06-02 | Neil Horman | Network buffer allocations based on consumption patterns |
US8438284B2 (en) * | 2009-11-30 | 2013-05-07 | Red Hat, Inc. | Network buffer allocations based on consumption patterns |
US20120072624A1 (en) * | 2010-09-17 | 2012-03-22 | Oracle America, Inc. | Numa i/o framework |
US20120265801A1 (en) * | 2011-04-13 | 2012-10-18 | Microsoft Corporation | Out of order assembling of data packets |
US8694618B2 (en) * | 2011-04-13 | 2014-04-08 | Microsoft Corporation | Maximizing data transfer through multiple network devices |
US20130055270A1 (en) * | 2011-08-26 | 2013-02-28 | Microsoft Corporation | Performance of multi-processor computer systems |
US20140029617A1 (en) * | 2012-07-27 | 2014-01-30 | Ren Wang | Packet processing approach to improve performance and energy efficiency for software routers |
Cited By (45)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20140254594A1 (en) * | 2013-03-05 | 2014-09-11 | Iosif Gasparakis | In nic flow switching |
US10693781B2 (en) | 2013-03-05 | 2020-06-23 | Intel Corporation | In NIC flow switching |
US9178815B2 (en) * | 2013-03-05 | 2015-11-03 | Intel Corporation | NIC flow switching |
US11102117B2 (en) | 2013-03-05 | 2021-08-24 | Intel Corporation | In NIC flow switching |
US20140369348A1 (en) * | 2013-06-17 | 2014-12-18 | Futurewei Technologies, Inc. | Enhanced Flow Entry Table Cache Replacement in a Software-Defined Networking Switch |
US9160650B2 (en) * | 2013-06-17 | 2015-10-13 | Futurewei Technologies, Inc. | Enhanced flow entry table cache replacement in a software-defined networking switch |
US9282040B2 (en) | 2014-02-28 | 2016-03-08 | Cisco Technology, Inc. | Smarter policy decisions based on metadata in data flows |
US11483243B2 (en) | 2014-02-28 | 2022-10-25 | Cisco Technology, Inc. | Smarter policy decisions based on metadata in data flows |
US20150249606A1 (en) * | 2014-02-28 | 2015-09-03 | Cisco Technology, Inc. | Smarter policy decisions based on metadata in data flows |
US10320676B2 (en) * | 2014-02-28 | 2019-06-11 | Cisco Technology, Inc. | Smarter policy decisions based on metadata in data flows |
US10126947B2 (en) * | 2014-05-09 | 2018-11-13 | Micron Technology, Inc. | Interconnect systems and methods using hybrid memory cube links to send packetized data over different endpoints of a data handling device |
US20170131915A1 (en) * | 2014-05-09 | 2017-05-11 | Micron Technology, Inc. | Interconnect systems and methods using hybrid memory cube links to send packetized data over different endpoints of a data handling device |
CN111190553A (en) * | 2014-05-09 | 2020-05-22 | 美光科技公司 | Interconnect system and method using hybrid memory cube links |
US11947798B2 (en) | 2014-05-09 | 2024-04-02 | Micron Technology, Inc. | Packet routing between memory devices and related apparatuses, methods, and memory systems |
US11132127B2 (en) * | 2014-05-09 | 2021-09-28 | Micron Technology, Inc. | Interconnect systems and methods using memory links to send packetized data between different data handling devices of different memory domains |
US10061808B2 (en) * | 2014-06-03 | 2018-08-28 | Sap Se | Cached views |
US20150347410A1 (en) * | 2014-06-03 | 2015-12-03 | Sap Ag | Cached Views |
US20160127493A1 (en) * | 2014-11-04 | 2016-05-05 | Qlogic Corporation | Caching methods and systems using a network interface card |
US10216666B2 (en) * | 2014-11-04 | 2019-02-26 | Cavium, Llc | Caching methods and systems using a network interface card |
US9934177B2 (en) | 2014-11-04 | 2018-04-03 | Cavium, Inc. | Methods and systems for accessing storage using a network interface card |
US20160285971A1 (en) * | 2014-12-05 | 2016-09-29 | Foundation For Research And Technology - Hellas (Forth) | Network Storage Protocol and Adaptive Batching Apparatuses, Methods, and Systems |
US10721302B2 (en) * | 2014-12-05 | 2020-07-21 | Foundation for Research and Technology—Hellas (FORTH) | Network storage protocol and adaptive batching apparatuses, methods, and systems |
US10708158B2 (en) * | 2015-04-10 | 2020-07-07 | Hewlett Packard Enterprise Development Lp | Network address of a computing device |
US10379899B2 (en) * | 2015-11-18 | 2019-08-13 | Nxp Usa, Inc. | Systems and methods for frame presentation and modification in a networking environment |
US20170139744A1 (en) * | 2015-11-18 | 2017-05-18 | Freescale Semiconductor, Inc. | Systems and methods for frame presentation and modification in a networking environment |
US11748172B2 (en) * | 2017-08-30 | 2023-09-05 | Intel Corporation | Technologies for providing efficient pooling for a hyper converged infrastructure |
US20190068698A1 (en) * | 2017-08-30 | 2019-02-28 | Intel Corporation | Technologies for providing efficient pooling for a hyper converged infrastructure |
CN107566302A (en) * | 2017-10-10 | 2018-01-09 | 新华三技术有限公司 | Message forwarding method and device |
CN107948072A (en) * | 2017-10-31 | 2018-04-20 | 新华三技术有限公司 | Message forwarding method and device |
US11394666B2 (en) * | 2017-12-18 | 2022-07-19 | Intel Corporation | Scalable communication with a packet processing unit |
US10735341B2 (en) | 2018-04-26 | 2020-08-04 | Nicira, Inc. | Dynamic provisioning of multiple RSS engines |
US11356381B2 (en) | 2018-07-23 | 2022-06-07 | Vmware, Inc. | Dynamic processing of packets using multiple receive queue features |
US10686716B2 (en) | 2018-07-23 | 2020-06-16 | Vmware, Inc. | Dynamic processing of packets using multiple receive queue features |
US20200036636A1 (en) * | 2018-07-25 | 2020-01-30 | Vmware, Inc. | Selection of paired transmit queue |
US11025546B2 (en) * | 2018-07-25 | 2021-06-01 | Vmware, Inc. | Selection of paired transmit queue |
US11848869B2 (en) | 2018-07-25 | 2023-12-19 | Vmware, Inc. | Selection of paired transmit queue |
US20210352048A1 (en) * | 2020-05-08 | 2021-11-11 | Gigamon Inc. | One-Armed Inline Decryption/Encryption Proxy Operating in Transparent Bridge Mode |
US11509631B2 (en) * | 2020-05-08 | 2022-11-22 | Gigamon Inc. | One-armed inline decryption/encryption proxy operating in transparent bridge mode |
US20230128082A1 (en) * | 2020-05-08 | 2023-04-27 | Gigamon Inc. | One-Armed Inline Decryption/Encryption Proxy Operating in Transparent Bridge Mode |
US11621915B2 (en) | 2020-07-07 | 2023-04-04 | Huawei Technologies Co., Ltd. | Packet forwarding method, route sending and receiving method, and apparatus |
US11537539B2 (en) * | 2020-10-19 | 2022-12-27 | Softiron Limited | Acceleration of data between a network and local I/O in a NUMA system |
US20220121583A1 (en) * | 2020-10-19 | 2022-04-21 | Softiron Limited | Acceleration of Data Between a Network and Local I/O in a NUMA System |
WO2022265552A1 (en) * | 2021-06-14 | 2022-12-22 | Telefonaktiebolaget Lm Ericsson (Publ) | System, method, and apparatus for fine-grained control of input/output data placement in a memory subsystem |
US20230006930A1 (en) * | 2021-07-02 | 2023-01-05 | Realtek Semiconductor Corp. | Method and system for processing data flow with incomplete comparison process |
US11765088B2 (en) * | 2021-07-02 | 2023-09-19 | Realtek Semiconductor Corp. | Method and system for processing data flow with incomplete comparison process |
Also Published As
Publication number | Publication date |
---|---|
US9047417B2 (en) | 2015-06-02 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US9047417B2 (en) | NUMA aware network interface | |
US11960429B2 (en) | Many-to-many PCIE switch | |
EP3931706B1 (en) | Adaptive address translation caches | |
US11102117B2 (en) | In NIC flow switching | |
US11507528B2 (en) | Pooled memory address translation | |
US11657015B2 (en) | Multiple uplink port devices | |
US11929927B2 (en) | Network interface for data transport in heterogeneous computing environments | |
US11238203B2 (en) | Systems and methods for accessing storage-as-memory | |
CN107111576B (en) | Issued interrupt architecture | |
US9208110B2 (en) | Raw memory transaction support | |
US9984017B2 (en) | Intelligent network fabric to connect multiple computer nodes with one or more SR-IOV devices | |
CN112631959A (en) | High bandwidth link layer for coherent messages | |
Sharma et al. | An introduction to the compute express link (cxl) interconnect | |
US11487695B1 (en) | Scalable peer to peer data routing for servers |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: INTEL CORPORATION, CALIFORNIA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:CONNOR, PATRICK;PAVLAS, CHRIS;KAPPLER, ELIZABETH M.;AND OTHERS;SIGNING DATES FROM 20121025 TO 20121029;REEL/FRAME:029211/0558 |
|
FEPP | Fee payment procedure |
Free format text: PAYOR NUMBER ASSIGNED (ORIGINAL EVENT CODE: ASPN); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY |
|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |
|
MAFP | Maintenance fee payment |
Free format text: PAYMENT OF MAINTENANCE FEE, 4TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1551); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY Year of fee payment: 4 |
|
AS | Assignment |
Owner name: TAHOE RESEARCH, LTD., IRELAND Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:INTEL CORPORATION;REEL/FRAME:061175/0176 Effective date: 20220718 |
|
MAFP | Maintenance fee payment |
Free format text: PAYMENT OF MAINTENANCE FEE, 8TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1552); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY Year of fee payment: 8 |