SYSTEMS AND METHODS IMPLEMENTING NON-SHARED PAGE TABLES FOR SHARING MEMORY RESOURCES MANAGED BY A MAIN OPERATING SYSTEM WITH ACCELERATOR DEVICES

• US 20110161619A1
• Filed: 12/29/2009
• Published: 06/30/2011
• Est. Priority Date: 12/29/2009
• Status: Active Grant

• Current Assignee: Advanced Micro Devices, Inc.
• Original Assignee: Advanced Micro Devices, Inc.
• Inventors: KAMINSKI, Patryk, LOWERY, Keith, BOLEYN, Erich, WOLLER, Thomas
• Granted Patent: US 8,719,543 B2
• Time in Patent Office: Days
• US Class Current: 711/207
• CPC Class Codes: G06F 12/1027 using associative or pseudo-associative address translation means, e.g. translation look-aside buffer [TLB]
• CPC Class Codes: G06F 12/1081 for peripheral access to main memory, e.g. direct memory access [DMA]
• CPC Class Codes: G06F 2212/302 In image processor or graphics adapter
• CPC Class Codes: G06F 2212/682 Multiprocessor TLB consistency
• CPC Class Codes: G06F 2212/683 Invalidation
• CPC Class Codes: G06F 9/5016 the resource being the memory

Abstract

Systems and methods are provided that utilize non-shared page tables to allow an accelerator device to share physical memory of a computer system that is managed by and operates under control of an operating system. The computer system can include a multi-core central processor unit. The accelerator device can be, for example, an isolated core processor device of the multi-core central processor unit that is sequestered for use independently of the operating system, or an external device that is communicatively coupled to the computer system.

Claims

1. A method for allowing an accelerator device to share physical memory of a computer system that is managed by and operates under control of an operating system of the computer system, the method comprising:

creating a plurality of operating system (OS) page tables for memory management for the accelerator device; and

creating another set of non-shared page tables that are independent of the operating system and are to be used exclusively by the accelerator device to share the physical memory, wherein each of the non-shared page tables correspond with one of the OS page tables.

2. A method according to claim 1, wherein the non-shared page tables are created at a driver, and further comprising:

providing the addresses of the non-shared page tables from the driver to the accelerator device.

3. A method according to claim 1, further comprising:

monitoring for page fault notifications generated by the accelerator device at a driver; and

handling page fault notifications received from the accelerator device at the driver.

4. A method according to claim 3, wherein the physical memory is divided into a plurality of memory pages, wherein each of the OS page tables are associated with corresponding ones of the memory pages in the physical memory, and wherein each of the non-shared page tables are associated with corresponding memory pages in the physical memory.

5. A method according to claim 4, wherein the operating system creates the operating system (OS) page tables when the operating system creates a process for the accelerator device, and wherein each of the non-shared page tables includes a plurality of page table entries that are used to store mappings of virtual memory addresses to physical memory addresses in the physical memory, and wherein the step of handling page fault notifications received from the accelerator device at the driver comprises:

determining, at the driver, a memory address space and virtual memory location of the process that contains a virtual memory address specified in a request for access to the physical memory that caused a page fault notification from the accelerator device;

determining, at the driver, whether the request from the accelerator device for access to physical memory that triggered the page fault is a valid request;

sending an error signal to the accelerator device from the driver when the request from the accelerator device for access to physical memory is determined to be invalid to indicate that the page fault could not be successfully handled;

pinning, at the driver when the request from the accelerator device for access to physical memory is determined to be valid, a limited amount of memory pages of the physical memory for use by the accelerator device to prevent the process from releasing limited amount of memory pages of the physical memory;

updating, at the driver, the non-shared page table for the memory pages being used by the accelerator device by adding new page table entries in the non-shared page table or editing existing page table entries in the non-shared page table; and

when the shared page table is updated, notifying the accelerator device that the page fault has been successfully handled and that the accelerator device is permitted to resume processing; and

further comprising;

using the updated page table entries from the non-shared page table at the accelerator device when processing resumes to perform virtual address translation.

6. A method according to claim 3, wherein the step of handling page fault notifications received from the accelerator device at the driver further comprises:

determining whether the non-shared page table includes page table entries for the corresponding virtual memory addresses with valid permissions to access the virtual memory addresses corresponding to the faulting memory location(s);

determining, when the non-shared page table does not include page table entries for the corresponding virtual memory addresses with valid permissions, whether the request from the accelerator device for access to physical memory that triggered the page fault is a valid request;

using the non-shared page table to lookup virtual memory address to physical memory address translations when the non-shared page table does include page table entries for the corresponding virtual memory addresses with valid permissions;

providing the page table entries from the driver to the accelerator device; and

notifying the accelerator device that the accelerator device is permitted to resume processing.

7. A method according to claim 6, further comprising:

using the page table entries from the driver at the accelerator device when processing resumes to perform virtual address translation and resumes processing.

8. A method according to claim 3, wherein the step of handling page fault notifications received from the accelerator device at the driver further comprises:

determining whether the page fault was caused by a missing page table entry;

when the driver determines that the page fault was caused by the missing page table entry, determining whether the accelerator device has permission to access the memory location at the specified virtual memory address; and

when the driver determines that the accelerator device has permission to access the memory location at the specified virtual memory address, updating the non-shared page table by adding new page table entries or editing existing page table entries in the non-shared page table for the memory pages being used by the accelerator device.

9. A method according to claim 8, wherein the step of handling page fault notifications received from the accelerator device at the driver further comprises:

when the driver determines that the page fault was not caused by a missing page table entry, determining whether the page fault was caused because the accelerator device did not have valid permission to access the memory;

when the driver determines that the accelerator device has caused the page fault because the accelerator device did not have valid permission to access the memory, determining whether the process that owns the virtual memory address space should have the permission to access the memory address and location; and

when the driver determines that the process that owns the virtual memory address space should have the permission to access the memory address and location, updating the corresponding page table entry to indicate the new permission/access rights and give necessary permission to the process to access the memory address.

10. A method according to claim 8, wherein the step of handling page fault notifications received from the accelerator device at the driver further comprises:

sending an error signal to the accelerator device to indicate that the page fault could not be successfully handled when the driver determines that;

(1) the accelerator device does not have permission to access the memory location at the specified virtual memory address, (2) the page fault was not caused because the accelerator device did not have valid permission to access the memory, or (3) the process that owns the virtual memory address space should not have the permission to access the memory address and location.

11. A method according to claim 3, wherein the physical memory is divided into a plurality of memory pages, wherein each of the OS page tables are associated with corresponding ones of the memory pages in the physical memory, wherein each of the non-shared page tables are associated with corresponding memory pages in the physical memory, and further comprising:

determining, at the driver, whether the process has released a block of memory back to the operating system;

determining, when the module determines that the process has released the block of memory back to the operating system, a memory address space and location of the block of memory;

determining, at the driver, whether the non-shared page table includes a page table entry corresponding to the block of memory being released;

identifying each memory page that corresponds to the block of memory being released when the non-shared page table includes a page table entry corresponding to the block of memory being released; and

invalidating each page table entry in the non-shared page table corresponding to the block of memory being released to ensure that the corresponding memory blocks and corresponding memory pages are no longer used by the accelerator device.

12. A method according to claim 11, wherein the accelerator device includes a translation lookaside buffer (TLB) cache of recently used page table translation entries, and further comprising:

issuing a TLB flush signal to the accelerator device to flush the translation lookaside buffer (TLB) cache of any page table entries that were invalidated;

waiting for a confirmation signal from the accelerator device confirming that the accelerator device has completed the TLB flush operation and removed the page table entries; and

unpinning physical memory pages corresponding to the block of memory being released by the process so that the physical memory pages are no longer be accessible to the accelerator device; and

allowing a process call to release the block of memory to resume and complete.

13. A method according to claim 3, wherein the physical memory is divided into a plurality of memory pages, wherein each of the OS page tables are associated with corresponding ones of the memory pages in the physical memory, wherein each of the non-shared page tables are associated with corresponding memory pages in the physical memory, and further comprising:

determining, at the driver, whether the process has terminated;

when the process has terminated, identifying an address space of the process that includes memory pages that are to be unpinned; and

invalidating page table entries corresponding to the memory pages that are to be unpinned.

14. A method according to claim 13, wherein the accelerator device includes a translation lookaside buffer (TLB) cache of recently used page table translation entries, and further comprising:

issuing a TLB flush signal to the accelerator device to flush the translation lookaside buffer (TLB) cache of any page table entries that were invalidated;

waiting for a confirmation signal from the accelerator device confirming that the accelerator device has completed the TLB flush operation and removed the page table entries;

unpinning physical memory pages corresponding to the block of memory being released by the process so that the physical memory pages are no longer accessible to the accelerator device; and

allowing a process call to release the block of memory to resume and complete.

15. A method according to claim 3, wherein the computer system further comprises:

a multi-core central processor unit comprising a plurality of core processor devices, wherein virtual addresses assigned to the process are used by at least one of the core processor devices, and wherein each of the core processor devices are associated with one or more of the OS page tables, and

wherein the driver directly uses non-shared page tables for virtual memory address translation to translate virtual memory addresses assigned to the process and used by a core processor device to physical memory addresses in the physical memory when the accelerator device needs to access a block of memory in a virtual memory address space (VMAS) assigned the process.

26. A system according to claim 13, wherein the shared physical memory is divided into a plurality of memory pages, wherein each of the OS page tables are associated with corresponding ones of the memory pages in the shared physical memory, wherein each of the non-shared page tables are associated with corresponding memory pages in the shared physical memory, wherein the accelerator device includes a translation lookaside buffer (TLB) cache of recently used page table translation entries, and wherein the driver is further designed to:

continuously monitor the system to determine whether the process has released a block of memory back to the OS;

determine a memory address space and location of the block of memory when the module determines that the process has released the block of memory back to the operating system;

determine whether the non-shared page table includes a page table entry corresponding to the block of memory being released;

identify each memory page that corresponds to the block of memory being released when the non-shared page table includes a page table entry corresponding to the block of memory being released, and invalidate each page table entry in the non-shared page table corresponding to the block of memory being released to ensure that the corresponding memory blocks and corresponding memory pages are no longer used by the accelerator device;

issue a TLB flush signal to the accelerator device to flush the translation lookaside buffer (TLB) cache of any page table entries that were invalidated, and wait for a confirmation signal from the accelerator device confirming that the accelerator device has completed the TLB flush operation and removed the page table entries; and

unpin physical memory pages corresponding to the block of memory being released by the process so that the physical memory pages are no longer be accessible to the accelerator device, and allow a process call to release the block of memory to resume and complete.

27. A system according to claim 13, wherein the shared physical memory is divided into a plurality of memory pages, wherein each of the OS page tables are associated with corresponding ones of the memory pages in the shared physical memory, wherein each of the non-shared page tables are associated with corresponding memory pages in the shared physical memory, wherein the driver comprises a process termination detection module designed to:

continuously monitor the system to determine whether the process has terminated or closed a last open driver handle;

identify an address space of the process that includes memory pages that are to be unpinned;

invalidate page table entries corresponding to the memory pages that are to be unpinned;

issue a TLB flush signal to the accelerator device to flush the translation lookaside buffer (TLB) cache of any page table entries that were invalidated, and wait for a confirmation signal from the accelerator device confirming that the accelerator device has completed the TLB flush operation and removed the page table entries; and

unpin physical memory pages corresponding to the block of memory being released by the process so that the physical memory pages are no longer accessible to the accelerator device, and allow a process call to release the block of memory to resume and complete.

31. A system according to claim 13, wherein the computer system further comprises:

a multi-core central processor unit comprising a plurality of core processor devices, wherein virtual addresses assigned to the process are used by at least one of the core processor devices, and wherein each of the core processor devices are associated with one or more of the OS page tables, and

wherein the driver directly uses non-shared page tables for virtual memory address translation to translate virtual memory addresses assigned to the process and used by a core processor device to physical memory addresses in the shared physical memory when the accelerator device needs to access a block of memory in a virtual memory address space (VMAS) assigned the process.

32. A system according to claim 13, wherein the driver is a kernel mode device driver that runs in protected mode and has unrestricted access to the shared physical memory and the operating system.

33. A system according to claim 13, wherein the computer system further comprises:

a multi-core central processor unit comprising a plurality of core processor devices, wherein virtual addresses assigned to the process are used by at least one of the core processor devices, and wherein each of the core processor devices are associated with one or more of the OS page tables, and

wherein the accelerator device comprises either;

an isolated core processor device that is sequestered for use independently of the operating system;

or

an external device that is communicatively coupled to the computer system, wherein the external device comprises a specialized processor that performs tasks independently of the multi-core central processor unit and does not directly execute operating system code.

34. A system according to claim 33, wherein the external device is selected from the group consisting of:

a Graphics Processing Unit (GPU);

an embedded central processor unit (CPU);

an advanced reduced instruction set computer (RISC) central processor unit (CPU);

an encryption/decryption device;

a compression device; and

a network accelerator device.

16. A system, comprising:

an accelerator device;

a computer system communicatively coupled to the accelerator device, the computer system comprising;

a main operating system (OS) designed to create a plurality of OS page tables for memory management for the accelerator device; and

shared physical memory that is managed by and operates under control of the main OS; and

a driver for the accelerator device that maintains another set of non-shared page tables that are independent of the main OS and are to be used exclusively by the accelerator device to share the shared physical memory, wherein each of the non-shared page tables correspond with one of the OS page tables.

17. A system according to claim 16, wherein the driver provides the addresses of the non-shared page tables to the accelerator device.

18. A system according to claim 16, wherein the shared physical memory is divided into a plurality of memory pages, wherein each of the OS page tables are associated with corresponding ones of the memory pages in the shared physical memory, wherein each of the non-shared page tables are associated with corresponding memory pages in the shared physical memory, and wherein the driver comprises a memory management unit (MMU) that provides a memory management function for the accelerator device, the MMU comprising:

a page fault notification module designed to continuously monitor the system for page fault notifications generated by the accelerator device; and

a page fault handler module for handling page fault notifications received from the accelerator device.

19. A system according to claim 18, wherein the operating system creates the operating system (OS) page tables when the operating system creates a process for the accelerator device, wherein each of the non-shared page tables includes a plurality of page table entries that are used to store mappings of virtual memory addresses to physical memory addresses in the physical memory, and wherein the page fault handler module is designed to:

determine a memory address space and virtual memory location of the process that contains a virtual memory address specified in a request for access to the shared physical memory, wherein the request for access to the shared physical memory caused a page fault notification from the accelerator device;

determine whether the request from the accelerator device for access to shared physical memory that triggered the page fault is a valid request;

pin a limited amount of memory pages of the shared physical memory at the specific address of the shared physical memory for use by the accelerator device to prevent the process from releasing limited amount of memory pages of the shared physical memory when the request from the accelerator device for access to shared physical memory is determined to be valid; and

update the non-shared page table for the memory pages being used by the accelerator device by adding new page table entries in the non-shared page table or editing existing page table entries in the non-shared page table.

20. A system according to claim 19, when the shared page table is updated, wherein the page fault handler module is further designed to:

notify the accelerator device that the page fault has been successfully handled and that the accelerator device is permitted to resume processing, and wherein the accelerator device uses the updated page table entries from the non-shared page table when processing resumes to perform virtual address translation.

21. A system according to claim 19, wherein the driver is configured to send an error signal to the accelerator device to indicate that the page fault could not be successfully handled when the request for access to shared physical memory is determined to be invalid.

22. A system according to claim 18, wherein the page fault handler module is further designed to:

determine whether the non-shared page table includes page table entries for the corresponding virtual memory addresses with valid permissions to access the virtual memory addresses corresponding to the faulting memory location(s), wherein the request for access to the shared physical memory caused a page fault notification from the accelerator device;

determine whether the request from the accelerator device for access to shared physical memory that triggered the page fault is a valid request when the non-shared page table does not include page table entries for the corresponding virtual memory addresses with valid permissions to access the virtual memory addresses corresponding to the faulting memory location(s);

use the non-shared page table to lookup the virtual memory address to physical memory address translations when the non-shared page table does include page table entries for the corresponding virtual memory addresses with valid permissions to access the virtual memory addresses corresponding to the faulting memory location(s);

provide the page table entries to the accelerator device and implicitly notify the accelerator device that the accelerator device is permitted to resume processing and execution of the work unit, and wherein the accelerator device uses the page table entries from the driver to perform virtual address translation and resumes processing.

23. A system according to claim 18, wherein the page fault handler module is designed to:

determine whether the page fault was caused by a missing page table entry;

determine whether the accelerator device has permission to access the memory location at the specified virtual memory address when the driver determines that the page fault was caused by the missing page table entry; and

update the non-shared page table by adding new page table entries or editing existing page table entries in the non-shared page table for the memory pages being used by the accelerator device when the driver determines that the accelerator device has permission to access the memory location at the specified virtual memory address.

24. A system according to claim 23, wherein the page fault handler module is further designed to:

determine whether the page fault was caused because the accelerator device did not have valid permission to access the memory when the driver determines that the page fault was not caused by a missing page table entry;

determine whether the process that owns the virtual memory address space should have the permission to access the memory address and location when the driver determines that the accelerator device has caused the page fault because the accelerator device did not have valid permission to access the memory; and

update the corresponding page table entry to indicate the new permission/access rights and give necessary permission to the process to access the memory address when the driver determines that the process that owns the virtual memory address space should have the permission to access the memory address and location.

25. A system according to claim 24, wherein the page fault handler module is further designed to:

send an error signal to the accelerator device to indicate that the page fault could not be successfully handled when the driver determines that;

(1) the accelerator device does not have permission to access the memory location at the specified virtual memory address, (2) the page fault was not caused because the accelerator device did not have valid permission to access the memory, or (3) the process that owns the virtual memory address space should not have the permission to access the memory address and location.

28. A system according to claim 18, wherein the driver is further designed to:

create and maintain a global list of pinned memory pages that are shared by all processes that interact with the accelerator device and a global counter for the pinned memory pages that are shared by all processes that interact with the accelerator device;

create and maintain a local list of pinned memory pages for each process and local counter for the pinned memory pages for each process;

determine, when the driver determines that it needs to pin a memory page, whether the local counter has reached a maximum number of pinned memory pages for that process; and

determine, when the driver determines that the local counter has reached the limit, the oldest pinned memory page in the local list, remove the oldest pinned memory page from the global list and decrement the global counter, remove the oldest pinned memory page from the local list and decrement the local counter for the process that owns the oldest pinned memory page; and

invalidate the corresponding page table entry in the page table, issue a TLB flush to the accelerator device, unpin the oldest pinned memory page, pin a new memory page, increment the global counter, and increment the local counter for the process that owns the new memory page.

29. A system according to claim 28, wherein the driver is further designed to:

determine, when the driver determines that the local counter has not yet reached the limit, whether the global counter has reached a maximum number of pinned memory pages shared by all processes that interact with the accelerator device;

pin the memory page when the driver determines that the global counter has not reached the maximum number of pinned memory pages shared by all processes that interact with the accelerator device, increment the global counter, and increment the local counter for the process that owns the memory page; and

determine, when the driver determines that the global counter has reached a maximum number of pinned memory pages shared by all processes that interact with the accelerator device, the oldest pinned memory page in the global list, remove the oldest pinned memory page from the global list and decrement the global counter, remove the oldest pinned memory page from the local list and decrement the local counter for the process that owns the oldest pinned memory page, invalidate the corresponding page table entry in the page table, issue a TLB flush to the accelerator device, unpin the oldest pinned memory page, pin a new memory page, increment the global counter, and increment the local counter for the process that owns the memory page.

30. A system according to claim 28, when the driver determines that a memory page needs to be unpinned before pinning an additional memory page, wherein the driver is further designed to remove a memory page from the global list, decrement the global counter, remove the memory page from the local list, decrement the local counter for the process that owns the memory page, and unpin the memory page.

35. An Operating System (OS) for a computer system, comprising:

a memory management unit (MMU) designed to manage and control physical memory of a computer system, the MMU comprising;

a plurality of operating system (OS) page tables, wherein each of the OS page tables correspond to a non-shared page table of a set of non-shared page tables that are to be used exclusively by the accelerator device along with corresponding OS page tables to share the physical memory, wherein the non-shared page tables are independent of the operating system.

36. An accelerator device designed to share physical memory of a computer system that is managed by and operates under control of an operating system of the computer system, wherein the accelerator device shares the physical memory by using a set of non-shared page tables in conjunction with a plurality of operating system (OS) page tables for memory management for the accelerator device, wherein each of the non-shared page tables correspond with one of the OS page tables, and wherein the non-shared page tables are independent of the operating system and used exclusively by the accelerator device to share the physical memory.