Concurrent testing of PCI express devices on a server platform

• US 10,114,658 B2
• Filed: 05/23/2016
• Issued: 10/30/2018
• Est. Priority Date: 05/23/2016
• Status: Active Grant

• Current Assignee: Baidu USA LLC (Baidu Incorporated)
• Original Assignee: Baida USA LLC
• Inventors: Huang, Davy, Elango, Krishna, Zhou, Xu
• Primary Examiner(s): Bonzo, Bryce P
• Assistant Examiner(s): Arcos, Jeison C
• Application Number:
  US15/162,529
• Publication Number:
  US 20170337069A1
• Time in Patent Office: 890 Days
• CPC Class Codes: G06F 11/221 to test buses, lines or interfaces, e.g. stuck-at or open line faults
• CPC Class Codes: G06F 11/2221 to test input/output devices or peripheral units
• CPC Class Codes: G06F 13/4022 using switching circuits, e.g. switching matrix, connection or expansion network G06F13/4009 takes precedence
• CPC Class Codes: G06F 13/4282 on a serial bus, e.g. I2C bus, SPI bus on daisy chain buses G06F13/4247
• CPC Class Codes: G06F 9/4411 Configuring for operating with peripheral devices; Loading of device drivers

Abstract

A method for testing peripheral component interconnect express (PCIe) devices is provided. The method implemented at a PCIe testing system detects that one or more PCIe devices have been inserted into one or more PCIe buses of a data processing system. In response to the detection, the PCIe testing system scans all PCIe buses of the data processing system to discover the one or more PCIe devices. For each of the PCIe devices discovered, the PCIe testing system repairs and retrains a PCIe link associated with the PCIe device, without rebooting the data processing system. The PCIe testing system loads a device driver instance for the PCIe device to be hosted by an operating system. The PCIe testing system then executes a test routine to concurrently test the one or more PCIe devices via the respective device driver instances.

Claims

1. A computer-implemented method for testing peripheral component interconnect express (PCIe) devices, the method comprising:

detecting that a plurality of PCIe devices have been inserted into one or more PCIe buses of a data processing system;

in response to the detection, scanning all PCIe buses of the data processing system to discover the plurality of PCIe devices;

for each of the PCIe devices discovered,

repairing and retraining a PCIe link associated with the PCIe device, without rebooting the data processing system, and

loading a device driver instance for the PCIe device to be hosted by an operating system;

executing a test routine to concurrently test the plurality of PCIe devices via respective device driver instances;

in response to a signal indicating that the execution of the test routine has been completed, unloading the device driver instances of the PCIe devices; and

communicating with the operating system to remove the Me devices from a namespace of the operating system, without rebooting the data processing system.

2. The method of claim 1, wherein loading a device driver instance comprises assigning a logical device identifier (ID) representing the PCIe device to the device driver instance, wherein the logical device ID is constructed based on a combination of one or more of a bus number, a device number, or a function number of the PCIe device.

3. The method of claim 2, wherein the logical device ID is utilized by the device driver instance to identify a physical PCIe slot in which the PCIe device is inserted, such that the plurality of PCIe devices can be concurrently identified and tested.

4. The method of claim 1, further comprising displaying a message to prompt a user that the PCIe devices can be physically removed from corresponding PCIe slots.

5. A computer-implemented method for testing peripheral component interconnect express (PCIe) devices, the method comprising:

detecting that a plurality of PCIe devices have been inserted into one or more PCIe buses of a data processing system;

in response to the detection, scanning all Pete buses of the data processing system to discover the plurality of PCIe devices;

for each of the PCIe devices discovered,

repairing and retraining a PCIe link associated with the PCIe device, without rebooting the data processing system, wherein repairing and retraining a PCIe link associated with the PCIe device comprises;

setting a first clock configuration bit in a first PCIe link control register of the PCIe device, and

setting a second clock configuration bit in a second PCIe link control register of a root complex device coupled to the PCIe device to recover a PCIe reference clock configuration, and

loading a device driver instance for the PCIe device to be hosted by an operating system; and

executing a test routine to concurrently test the plurality of PCIe devices via respective device driver instances.

6. The method of claim 5, further comprising setting a retrain link bit in the second PCIe link control register of the root complex device to trigger a PCIe link retraining process.

7. The method of claim 6, further comprising:

recovering settings of a PCIe device control register of the PCIe device; and

clearing any error in a PCIe configuration space reported during PCIe bus scanning.

8. The method of claim 7, further comprising copying memory address settings from the root complex device to the PCIe configuration space of the PCIe device.

9. A non-transitory machine-readable medium having instructions stored therein, which when executed by a processor, cause the processor to perform operations for testing PCIe devices, the operations comprising:

detecting that a plurality of PCIe devices have been inserted into one or more PCIe buses of a data processing system;

in response to the detection, scanning all PCIe buses of the data processing system to discover the plurality of PCIe devices;

for each of the PCIe devices discovered,

repairing and retraining a PCIe link associated with the PCIe device, without rebooting the data processing system, and

loading a device driver instance for the PCIe device to be hosted by an operating system;

executing a test routine to concurrently test the plurality of PCIe devices via respective device driver instances;

in response to a signal indicating that the execution of the test routine has been completed, unloading the device driver instances of the PCIe devices; and

communicating with the operating system to remove the PCIe devices from a namespace of the operating system, without rebooting the data processing system.

10. The medium of claim 9, wherein loading a device driver instance comprises assigning a logical device identifier (ID) representing the PCIe device to the device driver instance, wherein the logical device ID is constructed based on a combination of one or more of a bus number, a device number, or a function number of the PCIe device.

11. The medium of claim 10, wherein the logical device ID is utilized by the device driver instance to identify a physical PCIe slot in which the PCIe device is inserted, such that the plurality of PCIe devices can be concurrently identified and tested.

12. The medium of claim 9, wherein the operations further comprise displaying a message to prompt a user that the PCIe devices can be physically removed from corresponding PCIe slots.

13. A non-transitory machine-readable medium having instructions stored therein, which when executed b a processor, cause the processor to perform operations for testing PCIe devices, the operations comprising:

detecting that a plurality of PCIe devices have been inserted into one or more PCIe buses of a data processing system;

in response to the detection, scanning all PCIe buses of the data processing s stem to discover the plurality of PCIe devices;

for each of the PCIe devices discovered,

repairing and retraining a PCIe link associated with the PCIe device, without rebooting the data processing system, wherein repairing and retraining a PCIe link associated with the PCIe device comprises;

setting a first clock configuration bit in a first PCIe link control register of the PCIe device, and

setting a second clock configuration bit in a second PCIe link control register of a root complex device coupled to the PCIe device to recover a PCIe reference clock configuration, and

loading a device driver instance for the PCIe device to be hosted by an operating system; and

executing a test routine to concurrently test the plurality of PCIe devices via respective device driver instances.

14. The medium of claim 13, wherein the operations further comprise setting a retrain link bit in the second PCIe link control register of the root complex device to trigger a PCIe link retraining process.

15. The medium of claim 14, wherein the operations further comprise:

recovering settings of a PCIe device control register of the PCIe device; and

clearing any error in a PCIe configuration space reported during PCIe bus scanning.

16. The medium of claim 15, wherein the operations further comprise copying memory address settings from the root complex device to the PCIe configuration space of the PCIe device.

17. A system, comprising:

a processor; and

a memory coupled to the processor for storing instructions, which when executed from the memory, cause the processor to perform operations for testing PCIe devices, the operations including;

detecting that a plurality of PCIe devices have been inserted into one or more PCIe buses of a data processing system;

in response to the detection, scanning all PCIe buses of the data processing system to discover the plurality of PCIe devices;

for each of the PCIe devices discovered,

repairing and retraining a PCIe link associated with the PCIe device, without rebooting the data processing system, and

loading a device driver instance for the PCIe device to be hosted by an operating system;

executing a test routine to concurrently test the plurality of PCIe devices via respective device driver instances,

in response to a signal indicating that the execution of the test routine has been completed, unloading the device driver instances of the PCIe devices; and

communicating with the operating system to remove the PCIe devices from a namespace of the operating system, without rebooting the data processing system.

18. The system of claim 17, wherein loading a device driver instance comprises assigning a logical device identifier (ID) representing the PCIe device to the device driver instance, wherein the logical device ID is constructed based on a combination of one or more of a bus number, a device number, or a function number of the PCIe device.

19. The system of claim 18, wherein the logical device ID is utilized by the device driver instance to identify a physical PCIe slot in which the PCIe device is inserted, such that the plurality of PCIe devices can be concurrently identified and tested.

20. The system of claim 17, wherein the operations further comprise displaying a message to prompt a user that the PCIe devices can be physically removed from corresponding PCIe slots.

21. A system, comprising:

a processor; and

a memory coupled to the processor for storing instructions, which when executed from the memory, cause the processor to perform operations for testing PCIe devices, the operations including;

detecting that a plurality of PCIe devices have been inserted into one or more PCIe buses of a data processing system;

in response to the detection, scanning all PCIe buses of the data processing system to discover the plurality of PCIe devices;

for each of the PCIe devices discovered,

repairing and retraining a PCIe link associated with the PCIe device, without rebooting the data processing system, wherein repairing and retraining a PCIe link associated with the PCIe device comprises;

setting a first clock configuration bit in a first PCIe link control register of the PCIe device, and

setting a second clock configuration bit in a second PCIe link control register of a root complex device coupled to the PCIe device to recover a PCIe reference clock configuration, and

loading a device driver instance for the PCIe device to be hosted by an operating system; and

executing a test routine to concurrently test the plurality of PCIe devices via respective device driver instances.

22. The system of claim 21, wherein the operations further comprise setting a retrain link bit in the second PCIe link control register of the root complex device to trigger a PCIe link retraining process.

23. The system of claim 22, wherein the operations further comprise:

recovering settings of a PCIe device control register of the PCIe device; and

clearing any error in a PCIe configuration space reported during PCIe bus scanning.

24. The system of claim 23, wherein the operations further comprise copying memory address settings from the root complex device to the PCIe configuration space of the PCIe device.