WO2016175837A1

WO2016175837A1 - Configuration of a peripheral component interconnect express link

Info

Publication number: WO2016175837A1
Application number: PCT/US2015/028543
Authority: WO
Inventors: Stephen K. Gee; Sahba Etaati
Original assignee: Hewlett Packard Enterprise Development Lp
Priority date: 2015-04-30
Filing date: 2015-04-30
Publication date: 2016-11-03

Abstract

In some examples, the techniques of the present application herein describes a computer system that partitions a peripheral component interconnect express (PCIe) link based on a personality of a PCIe device. The computer system includes a PCIe link. A partial link configuration can be established when available lanes of the PCIe link are not in use. The computer system also includes a plug-in PCIe device. The PCIe link can be partitioned based on a personality of the plug-in PCIe device.

Description

CONFIGURATION OF A PERIPHERAL COMPONENT INTERCONNECT

EXPRESS LINK

BACKGROUND

[0001 ] Hardware components of modern computer systems can communicate using a variety of different connections and means. Peripheral component interconnect express (PCIe) provides a mechanism to connect PCIe devices to a printed circuit board (PCB).

BRIEF DESCRIPTION OF THE DRAWINGS

[0002] Certain examples are described in the following detailed description and in reference to the drawings, in which:

[0003] Fig. 1 is a block diagram of an example computer system to partition a PCIe link based on a personality communicated by a plug-in PCIe device;

[0004] Fig. 2 is a block diagram of an example of a motherboard;

[0005] Fig. 3 is a process flow diagram illustrating an example of a method for configuring lanes of a PCIe link;

[0006] Fig. 4 is a process flow diagram of an example method for hot plugging a PCIe device in a computer system and for run-time configuration; and

[0007] Fig. 5 is an example block diagram showing a non-transitory, computer- readable media that holds code that enables a processor to configure a PCIe add-in slot and partition a PCIe link.

DETAILED DESCRIPTION

[0008] The primary components of a computer system are coupled to a

motherboard, or printed circuit board (PCB). Examples of main components include a central processing unit (CPU) and memory. The motherboard can also include connectors or slots for peripheral devices. Common hardware standards and

specifications exists for peripheral component interconnect express (PCIe), which includes different lane count options for various slot widths. [0009] The techniques disclosed herein describe configuring the lanes of a PCIe link at a PCIe add-in slot based on the characteristics of a plug-in PCIe device. The PCIe add-in slot can be for a computer system that can include a server or multiple servers. Industry standards for PCIe slot width include link lane counts of x4, x8, and x16, for example. For example, a host bus adapter (HBA) is a plug-in PCIe device that can be inserted into a PCIe add-in slot. An HBA may provide multiple external ports for coupling other devices to the computer system. The external ports can include PCIe ports, USB ports, or other types of ports. For example, an HBA is commonly configured with four external ports, which may each be considered an end point. In examples, a PCIe link of a fixed width has lanes partitioned to accommodate the multiple external ports of a plug-in PCIe device, such as an HBA.

[0010] The partitioning occurs in a PCIe interconnect stack prior to the slot, for example, in a PCIe interconnect stack in a PCIe switch or in the interconnect stack of a processor or root structure itself. The PCIe interconnect stack is a layered protocol stack implemented in hardware. The layers of a PCIe interconnect stack include a transaction layer, a data link layer, and a physical layer. The partitioning may occur in the physical layer of a PCIe interconnect stack coupled to the lines feeding the PCIe slot. The partitioning is based on a personality of the plug-in PCIe device, for example, instructions sent from the plug-in PCIe device to the PCIe stack that report back the number of endpoints on the plug-in PCIe device, information about power consumption, and the like. A partial link configuration is established when available lanes of the PCIe link are not in use.

[0011 ] Fig. 1 is a block diagram of an example computer system 100 that may partition a link 102 from a PCIe switch 104 to a PCIe add-in slot 106 based on a personality communicated by a plug-in PCIe device 108. In addition to providing the number of endpoints 1 10 located on the plug-in PCIe device 108, the personality can describe how the lanes of the link 102 may be partitioned between the endpoints 1 10. For example, an HBA may be configured to link to a high bandwidth device, such as a storage attached network (SAN), through one endpoint 1 10, while the other endpoints 1 10 are linked to low bandwidth devices, such as a USB hub. The personality may instruct the allocation of 8 lanes to the endpoint 1 10 coupled to the SAN while allocating 4 lanes to two other endpoints 1 10 and leaving one endpoint disabled. In some examples, a carrier, signaling pin, or interposer board can be used to provide the personality or status of the plug-in PCIe device 108 to the relevant PCIe interconnect stack.

[0012] The computer system 100 may include a server computer or desktop computer, among others. The computer system 100 may include a processor 1 12 that is adapted to execute stored instructions. The processor 1 12 can be a single core processor, a multi- core processor, a computing cluster, or any number of other configurations.

[0013] The processor 1 12 may be connected through a system link 1 14 to the PCIe switch 104. Any number of other components may be coupled to the processor 1 12 through other links. For example, the processor 1 12 may also be linked to a memory 1 16 via a memory link 1 18. The memory link 1 18 may be a PCIe link, or a faster link, such as a quick path interconnect (QPI®) from Intel. The memory 1 16 can include static random- access memory (SRAM), dynamic random-access memory (DRAM), resistive random- access memory (RRAM), phase-change random-access memory (PRAM), memristor, non-volatile memory, electrically erasable programmable read-only memory (EEPROM), or any other suitable memory systems. The memory 1 16 can be configured to send, receive, and store data, including, for example, data related to a personality received from a plug-in PCIe device 108.

[0014] The memory 1 16 can include data, such as firmware images, system configuration information and device configuration information. The memory 1 16 can include code to direct the operation of the system. For example, a link configuration module 120, can instruct the processor 1 12 to configure a link 102 from a PCIe switch 104 to a plug-in PCIe device 108 based on a personality communicated by the plug-in PCIe device 108. In some examples, the configuration may be performed by a link configuration module located in the PCIe switch 104 itself.

[0015] The processor 1 12, or the PCIe switch 104, can access the data stored in the link configuration module 120 and send the data to the PCIe switch 104. The PCIe switch 104 can then be configured to partition the link 102 depending on the personality of the plug-in PCIe device 108. The link 102 can be partitioned into a partial

configuration and may also be retrained to accommodate a lane configuration according to the personality. The link configuration module 120 may provide a dynamic training technique to test the plug-in PCIe device 108 inserted into the PCIe add-in slot 106, and negotiate the width parameters based on the personality.

[0016] Other units may use the partitioning of the link 102. For example, the plug-in PCIe device 108 may include a storage device, an IOC for a user interface device, or a graphics processing device, among others. A user interface device may include a touchpad or a touchscreen, a keyboard, or a pointing device, among others.

[0017] The processor 1 12 may also be linked to a number of other units, either through the PCIe switch 104 or through a direct interconnect. For example the processor 1 12 may have a video link 122 to a display interface 124 adapted to connect the computer system 100 to a display device 126. The video link 122 may be a PCIe interconnection, or may be another type of interconnect. The display device 126 may include a display screen that is a built-in component of the computer system 100. The display device 126 may also include a computer monitor, television, or projector, among others, that is externally connected to the computer system 100.

[0018] Additionally, the processor 1 12 may also be linked through the PCIe switch 104 through another link 128 to a network interface controller (NIC) 130. The NIC 130 may be couple the computer system 100 to a network 132. The network 132 may be a local area network (Ethernet LAN), or a wireless (WiFi) network, among others.

[0019] It is to be understood that the block diagram of Fig. 1 is not intended to indicate that the computer system 100 is to include all of the components shown in Fig. 1 . Rather, the computer system 100 can include fewer or additional components not illustrated in Fig. 1 . In one example, the computer system 100 may include additional processors, memory controller devices, network interfaces, microcontrollers, microcontroller memory, etc. In some examples, the computer system 100 may not include the PCIe switch 104. In this example, the processor 1 12, or root complex including the processor 1 12, may be coupled to the PCIe add-in slot 106 through link 1 14, and may handle the partitioning through a PCIe interconnect stack located within the processor 1 12 or root complex

[0020] Fig. 2 is a block diagram of an example of a motherboard 200. The motherboard 200 is a printed circuit board (PCB), for example. A plurality of components can be electrically coupled to the motherboard 200. The examples of components include a basic input/output system (BIOS) chip 202, a backup battery 204, peripheral connectors 206, such as universal serial bus connectors (USBs), for example, one or more PCIe add-in slots 208, and memory 210, among other

components.

[0021 ] The motherboard 200 can also include a central processing unit (CPU) 212 and a CPU fan and heat sink 214. The CPU 212 can include a single processor or a plurality of processors. The CPU 212 can include any suitable type of processor or microprocessor. The CPU 212 can be coupled to the various system components via a cable, system links, or by any suitable means.

[0022] A plug-in PCIe device 216 can be inserted into the PCIe add-in slot 208. In some examples, the plug-in PCIe device 216 can be plugged into a carrier card, which can be an HBA, for example, and inserted into the PCIe add-in slot 208. In some examples, no carrier card is used and the plug-in PCIe device 216 is inserted directly into the PCIe add-in slot 208.

[0023] An interconnect or port on a PCIe switch 218 between the CPU 212 and the PCIe add-in slot 208 can be configured to implement a partitioning scheme based on a personality communicated by the plug-in PCIe device 216. For example, as described herein, the personality can describe how many endpoints exist on the plug-in PCIe device 216, how much power the plug-in PCIe device 216 utilizes, and how the available lanes may be partitioned between the endpoints. The motherboard 200 can support multiple plug-in PCIe devices 216 to meet product specifications, and allow concurrent access across multiple endpoints. Further, in some examples the PCIe switch 218 may not be present and an interconnect, or port, on the CPU 212 or a root complex may be directly configured to partition a link to a slot.

[0024] Partitioning of the link into lanes for each endpoint can be performed through an iterative training procedure to determine how available lanes of the link are divided. When the available link is discovered, for example, an 8 lane link can be virtually divided into two four lane links, or a 16 lane link can be divided into two 8 lane links, and so on in accordance with the number of endpoints and other parameters communicated by the personality on the plug-in PCIe device. In examples, the PCIe add-in slot 208 includes signals to allow link mode setting from the plug-in PCIe device 216 to the motherboard 200. Thus, a partial link configuration, or partitioning, can be established, and if some available lanes of the link are not in use, link training can be reinitiated to complete the partitioning of available lanes according to the personality.

[0025] It is to be understood that the illustration of Fig. 2 is not intended to indicate that the motherboard 200 is to include all of the components shown in Fig. 2 in every case. Further, any number of additional components can be included within the motherboard 200, depending on the details of the specific implementation. For example, in some implementations the motherboard 200 can omit the PCIe switch chip. Distribution of the connection to multiple endpoints is thus accomplished by the techniques described herein without using a hierarchy of switching circuits, such as PCIe switch chips located at either the plug-in PCIe device 216 or the motherboard 200. Thus, the current techniques can provide lower component and product cost, and a relatively compact product that is reliable, while consuming less power, generating less heat, and incorporating flexibility to accept standard industry hardware where

appropriate.

[0026] Fig. 3 is a process flow diagram illustrating an example of a method 300 for configuring lanes of a PCIe link. The method 300 partitions lanes of a PCIe link as a result of configuring the PCIe add-in slot. The PCIe link can be implemented throughout a computer system, such as computer system 100 of Fig. 1 , for example. The method 300 may be implemented by hardware components, such as, for example, processor 1 12, with instruction from memory 1 16, for example.

[0027] The method 300 begins at block 302, where a personality of a plug-in PCIe device is recognized by an interconnect linked to a PCIe add-in slot, for example, in a PCIe switch chip, a processor, or a root complex on a motherboard. The personality provides data that includes a number of PCIe endpoints on the plug-in PCIe device, as well as other parameters. In some examples, the plug-in PCIe device can be a common HBA with a single endpoint, for example, using a single lane of a PCIe link. In some examples, the plug-in PCIe device can be a custom-designed card with any number of PCIe endpoints, up to the limits of link subdivision. [0028] At block 304, the PCIe add-in slot can identify a first lane of the link and perform a link training operation on the lane. Once the link is established, the plug-in PCIe device has a read status that registers within the interconnect and determines the actual width capability of the plug-in PCIe device.

[0029] The method 300 continues at block 306, where a link is configured, e.g., partitioned, based on the personality communicated to a PCIe switch from the plug-in PCIe device. In some examples, no PCIe switch is used on the motherboard to partition the PCIe link, e.g., for a link directly from the processor or root complex. As used herein, a root complex connects the processor and memory subsystem to other PCIe devices, such as PCIe switches or PCIe slots. Further, the partitioning of lanes in the link on the motherboard eliminates the use of extra I/O chips on the plug-in PCIe device to implement the method 300. After the personality is recognized, the link to the PCIe add-in slot is configured to partition the lanes of the PCIe link at the add-in slot according to the number of endpoints on the plug-in PCIe device. The link can be disabled or re-enabled, in some cases, and the link can also be retrained to

accommodate a maximum slot width.

[0030] At block 308, a PCIe link is partitioned based on the personality

communicated from the plug-in PCIe device. In some examples, the PCIe link is partitioned between the number of endpoints by using logical signaling. Lanes of the PCIe link can be partitioned according to the detected personality of the plug-in PCIe device. For example, a 8 lane link can be split into two 4 lane links, or a 16 lane link into two 8 lane links, and so on in accordance with the number of endpoints and other parameters communicated by the personality. The method 300 can be implemented to partition any number of lanes in a link at any width allowed under PCIe standards.

[0031] In some examples, the personality can be communicated and the link partitioned using a processor and software that queries the plug-in PCIe device and programs the link to the PCIe add-in slot. A partial link configuration can be achieved at this point in the method 300, and then when additional lanes of the PCIe link are available, the method 300 can partition the lanes. Further, it is possible for a plug-in PCIe device with many endpoints to exceed the host capabilities. For example, the motherboard may be power or thermally limited in some implementations. Thus, in some examples, power demand can also be provided as part of the personality. In these example, the plug-in PCIe device may enable the number of endpoints that can be supported.

[0032] The method 300 can be performed in multiple use cases. In an example use case, a plug-in PCIe device can already be present in a PCIe add-in slot at startup of the computer system. In some examples, the board management controller (BMC) on the motherboard receives communication from the plug-in PCIe device, and the basic input/output system (BIOS) firmware uses the personality to program the PCIe link to accommodate a number of endpoints on the plug-in PCIe device. Alternatively, in examples, the BIOS can directly access the plug-in PCIe device personality information. In another example use case, an add-in slot can be empty at the startup of the computer system and a plug-in PCIe device can be hot-plugged into the PCIe add-in slot. In this case, the operating system can determine the hardware present on the plug-in PCIe device, and use the personality from the plug-in PCIe device to partition the lanes of the PCIe link between endpoints of the plug-in PCIe device. The plug-in PCIe device can be started when the configuration of the link to the add-in slot is complete based on the number of endpoints present on the plug-in PCIe device.

[0033] It is to be understood that the process flow diagram of Fig. 3 is not intended to indicate that the method 300 is to include all of the blocks shown in Fig. 3 in every case. Further, any number of additional blocks can be included within the method 300, depending on the details of the specific implementation.

[0034] Fig. 4 is a process flow diagram of an example method 400 for hot plugging a PCIe device in a computer system and for run-time configuration. The method 400 can allow a single-endpoint plug-in PCIe device to be replaced with a multiple-endpoint plug-in PCIe device and vice versa. This may increase the flexibility of the system implementation. The hot-plug sequence can begin at block 402, when a user indicates a need to replace the plug-in PCIe device. The user can make such an indication via push button, software, latch actuation, or by other appropriate means.

[0035] At block 404, application software on memory of the computer system can be used to deconfigure the plug-in PCIe device that is to be replaced. Clocks, power, and electrical signals are turned off, and light emitting diodes (LEDs), electromechanical locks, etc. are disabled or enabled in accordance with PCIe specification or system defined specification, for example.

[0036] At block 406, the user removes the deconfigured PCIe device and adds a new plug-in PCIe device. The insertion of the new plug-in PCIe device can be confirmed complete through a presence detection, a user action, a push-button activation, or a latch closing, for example.

[0037] At block 408, the new plug-in PCIe device can be configured. LEDs, electromechanical locks, and the like can be turned on or off in accordance with PCIe specification or other system-defined specification. Clocks, power, and electrical signals for the plug-in PCIe device, for example, can be enabled in accordance with standard PCIe specifications.

[0038] At block 410, the personality of the plug-in PCIe device can be detected using any number of methods. For example, the board management controller (BMC) on the computer system motherboard can receive a communication from the plug-in PCIe device, and the basic input/output system (BIOS) firmware uses the personality to partition the PCIe link to accommodate a number of endpoints on the plug-in PCIe device. Alternatively, in some examples, the BIOS can access the personality of the plug-in PCIe device. In some examples, a carrier, signaling pin, or interposer board can be used to provide the personality or status of the plug-in PCIe device. If a PCIe switch is providing the link to the slot on the motherboard, the PCIe switch may use the personality to partition the lanes.

[0039] At block 412, lanes of the PCIe link can be partitioned according to the detected personality of the plug-in PCIe device. The method 400 can be implemented to partition any number of lanes in a link at any allowed PCIe lane width. The method 400 can be used to allow a partition of the PCIe link for the plug-in PCIe device, whether or not the PCIe device includes hardware such as a PCIe switch. In examples, partitioning can occur at a PCIe switch upstream of the PCIe add-in slot.

[0040] It is to be understood that the process flow diagram of Fig. 4 is not intended to indicate that the method 400 is to include all of the blocks shown in Fig. 4 in every case. Further, any number of additional blocks can be included within the method 400, depending on the details of the specific implementation. [0041 ] Fig. 5 is an example block diagram showing a non-transitory, computer- readable media 500 that holds code that enables a processor 502 to configure a PCIe add-in slot and partition a link to an add-in slot. The computer-readable media 500 may be accessed by the processor 502 over a link 504. In some examples, the system link 504 is a PCIe link. The code may direct the processor 502 to perform the steps of the current method as described with respect to Fig. 3 or Fig. 4, for example.

[0042] The non-transitory, computer-readable media 500 may include a recognition module 506 that is configured to recognize a personality of a plug-in PCIe device. The plug-in PCIe device can be inserted into a PCIe add-in slot to

communication through a PCIe link. In some examples, the add-in slot can identify LaneO, and perform link training at a single lane width. The PCIe status registers within the endpoint may then be read to determine the actual width capacity of the plug-in PCIe device.

[0043] A connection module 508 can configure a link to PCIe add-in slot based on the based on the personality of the plug-in PCIe device. A partial link configuration can be established and link training can be performed again so the width capability of the plug-in PCIe device is satisfied, as well as for potential additional PCIe devices. Thus, the configuration may be iterative. As an example, a link may be established for one endpoint at a time, until all the endpoints are satisfied, or the system decides it has enabled all that can be supported. The connection can be established between lanes of the PCIe link and the plug-in PCIe device.

[0044] A partition module 510 can be configured to partition the lanes of the PCIe link to the add-in slot. For example, as discussed herein, the PCIe link can be partitioned based on the number of endpoints present on the plug-in PCIe device. In this way, the lanes of a PCIe add-in slot can be logically split without use of a chipset switch or extra input/output controllers. Thus, a customizable connection can be established between a plug-in PCIe device and a PCIe add-on slot, based on the available lanes in the PCIe link and the parameters of the PCIe device.

[0045] The block diagram of Fig. 5 is not intended to indicate that the computer- readable media 500 is to include all of the components or modules shown in Fig. 5. Further, any number of additional components may be included within the computer- readable media 500, depending on the details of the specific implementation of partitioning a PCIe link described herein.

[0046] While the present techniques may be susceptible to various modifications and alternative forms, the exemplary examples discussed above have been shown only by way of example. It is to be understood that the technique is not intended to be limited to the particular examples disclosed herein. Indeed, the present techniques include all alternatives, modifications, and equivalents falling within the true spirit and scope of the appended claims.

Claims

CLAIMS What is claimed is:

1 . A computer system for configuration of a peripheral component interconnect express (PCIe) link, comprising:

a PCIe link, wherein a partial link configuration is established when

available lanes of the PCIe link are not in use; and

a plug-in PCIe device, wherein the available lanes of the PCIe link are partitioned based on a personality of the plug-in PCIe device.

2. The computer system of claim 1 , wherein the personality of the plug-in PCIe device includes a number of endpoints for the plug-in PCIe device, and wherein the personality is communicated to a motherboard.

3. The computer system of claim 2, wherein the PCIe link based on the number of endpoints communicated to the motherboard.

4. The computer system of claim 1 , wherein the plug-in PCIe device is hot- plugged into a PCIe add-in slot.

5. The computer system of claim 1 , further comprising a memory to store data related to the personality.

6. The computer system of claim 1 , further comprising a basic input/output system (BIOS), wherein the personality is directly accessed by the BIOS.

7. The computer system of claim 1 , further comprising a board management controller (BMC), wherein the personality is accessed by the BMC and communicated to a BIOS of the computer system.

8. A method for configuration of a peripheral component interconnect express (PCIe) link on a computer system, comprising:

recognizing a personality of a plug-in PCIe device;

configuring lanes of a PCIe link to a PCIe add-in slot based on the

personality communicated from the plug-in PCIe device;

establishing a partial link configuration when available lanes of the PCIe link are not in use; and

partitioning the PCIe link based on the personality.

9. The method of claim 8, wherein the personality includes a number of endpoints associated with the plug-in PCIe device.

10. The method of claim 8, wherein the lanes of the PCIe link are configured through a basic input/output system (BIOS) of the computer system.

1 1. The method of claim 8, wherein the lanes of the PCIe link are configured through an operating system (OS) of the computer system.

12. The method of claim 8, wherein the plug-in PCIe device is present and connected to the PCIe add-in slot when the computer system is booted up, and wherein the PCIe link is partitioned at run time of the computer system.

13. A tangible, non-transitory, computer-readable storage medium, comprising code for configuration of a peripheral component interconnect express link that, when executed, causes a processor to:

recognize a personality of a plug-in peripheral component interconnect express (PCIe) device; configure lanes of a PCIe link to a PCIe add-in slot based on the personality communicated from the plug-in PCIe device;

establish a partial link configuration when available lanes of the PCIe link are not in use; and

partition the PCIe link based on the personality.

14. The tangible, non-transitory, computer-readable storage medium of claim 13, wherein the personality of the plug-in PCIe device includes a number of endpoints on the plug-in PCIe device.

15. The tangible, non-transitory, computer-readable storage medium of claim 14, wherein lanes of the PCIe link are partitioned to accommodate the number of endpoints communicated by the personality.