Method and apparatus for implementing PCI DMA speculative prefetching in a message passing queue oriented bus system

• US 20040015622A1
• Filed: 07/09/2003
• Published: 01/22/2004
• Est. Priority Date: 11/16/2000
• Status: Active Grant

• Current Assignee: Sun Microsystems Incorporated (Oracle Corporation)
• Original Assignee: Sun Microsystems Incorporated (Oracle Corporation)
• Inventors: Avery, James M.
• Granted Patent: US 6,813,653 B2
• Time in Patent Office: Days
• US Class Current: 710/22
• CPC Class Codes: G06F 13/28 using burst mode transfer, e.g. direct memory access DMA, cycle steal G06F13/32 takes precedence

Abstract

Speculative prefetching during DMA reads in a message-passing, queue-oriented bus system is controlled by creating a special data structure, called a “DMA scoreboard”, for each work queue entry associated with a DMA read. The DMA scoreboard tracks the completion of DMA writes and reads by monitoring acknowledgements received from DMA writes and data tags received from DMA read responses. The DMA scoreboard also contains a section that indicates the current PCI address, and size and number of prefetches to be performed. After a DMA read has completed, the PCI current address is incremented to obtain a new PCI address for the first prefetch request. A new work queue entry is then created from the information in the DMA scoreboard to perform the prefetch. If the amount of data to be fetched exceeds the maximum amount of data that can be retrieved by a single read request, when the read request has been completed, the address stored in the DMA scoreboard is again incremented to create another address and another work queue entry is created. Operation continues in this manner until the number of prefetches specified in the DMA scoreboard have been performed.

Claims

1. Apparatus for performing speculative prefetching for a PCI DMA read request in a PCI—

InfiniBand bridge system, the apparatus comprising;

an update mechanism responsive to data returning from an initial InfiniBand RDMA request issued to satisfy the PCI DMA read request for computing a next address from the read address of the initial InfiniBand RDMA request; and

a repeat mechanism that generates a new RDMA read request using the next address to perform a speculative prefetch.

2. The apparatus of claim 1 wherein the next address is computed from information defining a prefetch request.

3. The apparatus of claim 2 wherein the information defining a prefetch request comprises a prefetch request number and a prefetch request size.

4. The apparatus of claim 3 wherein the update mechanism computes the next address by adding the prefetch request size to the read address of the initial InfiniBand RDMA request.

5. The apparatus of claim 3 wherein the repeat mechanism generates further RDMA prefetch read requests until an amount of data equal to the prefetch request size has been retrieved.

6. The apparatus of claim 1 further comprising a data structure that stores a data tag identifying the new RDMA read request.

7. The apparatus of claim 6 wherein the data structure is referenced in the new RDMA read request.

8. The apparatus of claim 1 wherein the repeat mechanism generates the new RDMA read request on a work queue used to generate the initial RDMA read request.

9. The apparatus of claim 1 wherein the data returning from an RDMA read request comprises a plurality of data packets.

10. The apparatus of claim 1 wherein the update mechanism comprises an address map that computes the read address of the initial InfiniBand RDMA request from a PCI address.

11. The apparatus of claim 10 wherein the address map includes an R-key that that is associated with an area in a memory from which DMA data is retrieved and a pointer to a work queue that generates the initial InfiniBand RDMA request.

12. The apparatus of claim 11 wherein the repeat mechanism generates a new read address for an RDMA read request by combining the PCI address with the R-Key and the next address.

13. A method for performing speculative prefetching for a PCI DMA read request in a PCI—

InfiniBand bridge system, the method comprising;

(a) in response to data returning from an initial InfiniBand RDMA request issued to satisfy the PCI DMA read request, computing a next address from the read address of the initial InfiniBand RDMA request; and

(b) generating a new RDMA read request using the next address to perform a speculative prefetch.

14. The method of claim 13 wherein step (b) comprises computing the next address from information defining a prefetch request.

15. The method of claim 14 wherein the information defining a prefetch request comprises a prefetch request number and a prefetch request size.

16. The method of claim 15 wherein step (a) comprises computing the next address by adding the prefetch request size to the read address of the initial InfiniBand RDMA request.

17. The method of claim 15 wherein step (b) comprises generating further RDMA prefetch read requests until an amount of data equal to the prefetch request size has been retrieved.

18. The method of claim 13 further comprising (c) storing a data tag identifying the new RDMA read request in a data structure.

19. The method of claim 18 wherein the data structure is referenced in the new RDMA read request.

20. The method of claim 13 wherein step (b) comprises generating the new RDMA read request on a work queue used to generate the initial RDMA read request.

21. The method of claim 13 wherein the data returning from an RDMA read request comprises a plurality of data packets.

22. The method of claim 13 wherein the step (a) comprises using an address map to compute the read address of the initial InfiniBand RDMA request from a PCI address.

23. The method of claim 22 wherein the address map includes an R-key that that is associated with an area in a memory from which DMA data is retrieved and a pointer to a work queue that generates the initial InfiniBand RDMA request.

24. The method of claim 23 wherein step (b) comprises generating a new read address for an RDMA read request by combining the PCI address with the R-Key and the next address.

25. Apparatus for performing speculative prefetching for a PCI DMA read request in a PCI—

InfiniBand bridge system, the apparatus comprising;

means responsive to data returning from an initial InfiniBand RDMA request issued to satisfy the PCI DMA read request for computing a next address from the read address of the initial InfiniBand RDMA request; and

means for generating a new RDMA read request using the next address to perform a speculative prefetch.

26. The apparatus of claim 25 wherein the means for computing the next address comprises means for computing the next address from information defining a prefetch request.

27. The apparatus of claim 26 wherein the information defining a prefetch request comprises a prefetch request number and a prefetch request size.

28. The apparatus of claim 27 wherein the means for computing the next address comprise means for adding the prefetch request size to the read address of the initial InfiniBand RDMA request.

29. The apparatus of claim 27 wherein the means for generating the new RDMA read request comprises means for generating further RDMA prefetch read requests until an amount of data equal to the prefetch request size has been retrieved.

30. The apparatus of claim 25 wherein the means for generating the new RDMA read request comprises means for generating the new RDMA read request on a work queue used to generate the initial RDMA read request.

31. A method for performing DMA read speculative prefetches in a message-passing, queue-oriented bus system having a memory and a DMA mechanism that generates a DMA read request to retrieve data, via the bus system, from the memory and receives a response for each DMA read request, the method comprising:

(a) using a DMA scoreboard data structure to store information concerning a current DMA request, the information including the current read address and a data tag identifying the current request;

(b) updating the DMA scoreboard data structure when a response is received that corresponds to the stored data tag; and

(c) generating a new DMA read request using the information in the updated DMA scoreboard data structure.

32. The method of claim 31 wherein step (a) comprises using the DMA scoreboard to store the size of a prefetch request.

33. The method of claim 32 wherein step (a) comprises incrementing the current address by adding the prefetch request size to the current address.

34. The method of claim 33 wherein step (a) comprises using the DMA scoreboard to store a data tag identifying the prefetch read request.

35. The method of claim 31 wherein step (b) comprises generating a new DMA read request on a work queue used to generate the initial DMA read request.

36. The method of claim 35 wherein step (a) comprises referencing the DMA scoreboard data structure in the new DMA read request.

37. The method of claim 31 wherein step (b) comprises generating further DMA prefetch read requests until an amount of data equal to the prefetch size has been retrieved.

38. The method of claim 31 wherein the response to a DMA read request comprises a plurality of data packets.

39. The method of claim 38 wherein the current address is a PCI address.

40. The method of claim 39 wherein step (b) comprises generating a new DMA read request by combining the PCI address with an R-Key that is associated with an area in the memory.