Address translation mechanism for shared memory based inter-domain communication

• US 8,825,984 B1
• Filed: 10/13/2008
• Issued: 09/02/2014
• Est. Priority Date: 10/13/2008
• Status: Active Grant

• Current Assignee: NetApp, Inc.
• Original Assignee: NetApp, Inc.
• Inventors: Srinivasan, Kiran, Radhakrishnan, Prashanth
• Primary Examiner(s): Bradley, Matthew
• Assistant Examiner(s): GOLDSCHMIDT, CRAIG S
• Application Number:
  US12/250,153
• Time in Patent Office: 2,150 Days
• Field of Search: 711/206
• US Class Current: 711/206
• CPC Class Codes: G06F 12/1009 using page tables, e.g. page table structures
• CPC Class Codes: G06F 12/1036 for multiple virtual address spaces, e.g. segmentation G06F12/1045 takes precedence
• CPC Class Codes: G06F 12/109 for multiple virtual address spaces, e.g. segmentation G06F12/1036 takes precedence
• CPC Class Codes: G06F 2212/151 Emulated environment, e.g. virtual machine
• CPC Class Codes: G06F 2212/656 Address space sharing

Abstract

A technique for “zero copy” transitive communication of data between virtual address domains maintains a translation table hierarchy for each domain. The hierarchy of each domain includes a portion corresponding to every other domain in the system, where the portion for any particular domain begins at the same offset in the virtual address space of every domain. For each domain, there is a source hierarchy used only by the domain itself, which provides read/write access to the addresses in that domain; and a target hierarchy which provides read-only access to that domain, for use only when another domain is the target of IDC from that domain. Only one instance of the target hierarchy of each domain is provided, for all other domains as targets of IDC from that domain. For further space savings the source and target translation table hierarchies can be combined at all but the top hierarchy level.

Claims

1. A method, comprising:

creating a first address translation table hierarchy for a first virtual address domain of a plurality of virtual address domains of a processing system;

creating a second address translation table hierarchy for the first virtual address domain, for use when a second virtual address domain is a target of data communication from the first virtual address domain, the second address translation table hierarchy incorporating part of the first address translation table hierarchy; and

using the second address translation table hierarchy to communicate data between the first and second virtual address domains,

wherein the first address translation table hierarchy provides read-write permission to access a corresponding memory space, and the second address translation table hierarchy provides read-only permission to access the corresponding memory space;

wherein the first address translation hierarchy includes a first to level, and first lower levels include at least a page directory level and a page table level, the page directory level including page directories comprising pointers to page tables of the page table level, the page tables comprising pointers to memory pages;

wherein the first top level includes an access permission indicator for the corresponding memory space, the access permission indicator governing access to the corresponding memory space regardless of permission indicators of the first lower levels;

wherein the second address translation hierarchy includes a second to level that has at least one directory entry pointing to directories of the first lower levels of the first address translation hierarchy, the second address translation hierarchy including levels of the first address translation hierarchy below the first to level;

wherein the second to level of the second address translation hierarchy includes an access permission indicator that is set differently from the access permission indicator in the first to level of the first translation hierarchy.

2. A method as recited in claim 1, wherein said creating a first address translation table hierarchy and said creating a second address translation table hierarchy are done for each virtual address domain of the plurality of virtual address domains.

3. A method as recited in claim 1, wherein each of the virtual address domains corresponds to a separate one of a plurality of virtual machines.

4. A method comprising:

operating a plurality of processing entities in a processing system, each of the processing entities having a separate virtual address domain of a plurality of virtual address domains; and

maintaining a translation table hierarchy for each of the plurality of virtual address domains, for mapping between virtual addresses and physical addresses, wherein the translation table hierarchy of each domain includes a portion corresponding to every other domain of the plurality of virtual address domains, and wherein the portion corresponding to any given domain begins at the same offset in the virtual address space of each domain of the plurality of virtual address domains;

wherein the translation table hierarchy for each of the domains includes;

a source address translation table hierarchy for said domain that provides read-write permission to access a corresponding memory space, for use only by the corresponding processing entity; and

a separate target address translation table hierarchy for said domain that provides read-only permission to access the corresponding memory space, for use by the processing entity of any other domain which is a target of communication from said processing entity;

wherein the source address translation hierarchy includes a first to level, and first lower levels include at least a page directory level and a page table level, the page directory level including page directories comprising pointers to page tables of the page table level, the page tables comprising pointers to memory pages;

wherein the first top level includes an access permission indicator for the corresponding memory space, the access permission indicator governing access to the corresponding memory space regardless of permission indicators of the first lower levels;

wherein the target address translation hierarchy includes a second to level that has at least one directory entry pointing to directories of the first lower levels of the source address translation hierarchy, the target address translation hierarchy including levels of the source address translation hierarchy below the first to level;

wherein the second to level of the target address translation hierarchy includes an access permission indicator that is set differently from the access permission indicator in the first to level of the source translation hierarchy.

5. A method as recited in claim 4, wherein each of the processing entities is a virtual machine.

6. A method comprising:

creating a plurality of virtual machines in a processing system, the plurality of virtual machines including a first virtual machine and a second virtual machine;

creating a plurality of address translation table hierarchies for use in performing virtual address to physical address translation during communications between the first and second virtual machines, including

creating a first address translation table hierarchy and a second address translation table hierarchy for the first virtual machine, for mapping between virtual addresses and physical addresses, wherein a portion of the first address translation table hierarchy is incorporated into the second address translation table hierarchy; and

communicating data from the first virtual machine to the second virtual machine without copying the data, wherein said communicating includes

determining a first virtual address of the data in a domain of the first virtual machine,

modifying the first virtual address according to an offset of the second translation table hierarchy in a domain of the second virtual machine, to form a modified virtual address,

passing the modified virtual address to the second virtual machine, and

using the second address translation table hierarchy of the first virtual machine to map the modified virtual address to a physical address;

wherein;

the first address translation table hierarchy of the first virtual machine corresponds to the virtual address space of the first virtual machine with read-write permission; and

the second address translation table hierarchy of the first virtual machine corresponds to the virtual address space of the first virtual machine with read-only permission;

wherein the first address translation hierarchy includes a first to level, and first lower levels include at least a page directory level and a page table level, the page directory level including page directories comprising pointers to page tables of the page table level, the page tables comprising pointers to memory pages;

wherein the first top level includes an access permission indicator for the corresponding address space, the access permission indicator governing access to the corresponding address space regardless of permission indicators of the first lower levels;

wherein the second address translation hierarchy includes a second to level that has at least one directory entry pointing to directories of the first lower levels of the first address translation hierarchy, the second address translation hierarchy including all levels of the first address translation hierarchy below the first to level;

wherein the second to level of the second address translation hierarchy includes an access permission indicator that is set differently from the access permission indicator in the first to level of the first translation hierarchy.

7. A method as recited in claim 6, wherein the processing system has a 64-bit address space, and wherein said portion of the first address translation table hierarchy which is incorporated into the second address translation table hierarchy includes all levels of the first address translation table hierarchy below a page global directory level.

8. A method as recited in claim 6, wherein each of the first and second virtual machines each correspond to first and second domains, respectively, of a pseudo-global virtual address space.

9. A processing system comprising:

a processor;

a plurality of processing entities, each of the processing entities having a separate one of a plurality of virtual address domains; and

a storage medium storing code which, when executed by the processor, causes the processing system to perform operations including

creating a source address translation table hierarchy corresponding to a virtual address space of a first virtual address domain, for use by the first virtual address domain,

creating a target address translation table hierarchy corresponding to a virtual address space of the first virtual address domain, the target address translation table hierarchy for use by a second virtual address domain when the second virtual address domain is a target of communication from the first virtual address domain, the target address translation table hierarchy incorporating part of the source address translation table hierarchy; and

using the target address translation table hierarchy to communicate data between the first and second virtual address domains;

wherein;

the source address translation table hierarchy of the first virtual address domain provides read-write permission to access a memory space of the first virtual address domain; and

the target address translation table hierarchy of the first virtual address domain provides read-only permission to access the memory space of the first virtual address domain;

wherein the source address translation hierarchy includes a first to level, and first lower levels including at least a page directory level and a page table level, the page directory level including page directories comprising pointers to page tables of the page table level, the page tables comprising pointers to memory pages;

wherein the first to level includes an access permission indicator for the corresponding memory space, the access permission indicator governing access to the corresponding memory space regardless of permission indicators of the first lower levels;

wherein the target address translation hierarchy includes a second to level that has at least one directory entry pointing to directories of the first lower levels of the source address translation hierarchy, so that the target address translation hierarchy including all levels of the source address translation hierarchy below the first to level;

wherein the second to level of the target address translation hierarchy includes an access permission indicator that is set differently from the access permission indicator in the first to level of the source translation hierarchy.

10. A processing system as recited in claim 9, wherein the storage medium further stores code which, when executed by the processor, causes the processing system to perform operations including

creating a source address translation table hierarchy corresponding to the virtual address space of the second virtual address domain, for use by the second virtual address domain,

creating a target address translation table hierarchy corresponding to the virtual address space of the second virtual address domain, the target address translation table hierarchy for use by the first virtual address domain when the first virtual address domain is a target of communication from the second virtual address domain, the target address translation table hierarchy of the second virtual address domain incorporating part of the source address translation table hierarchy of the second virtual address domain.

11. A processing system as recited in claim 9, wherein at least two of the plurality of processing entities are virtual machines.

12. A processing system as recited in claim 9, wherein at least two of the plurality of processing entities are configured collectively to perform network storage services in response to a request of a remote storage client.

13. A network storage server comprising:

a processor;

a network interface through which to receive a data access request from a client;

a storage interface through which to access a mass storage facility in response to the data access request;

a plurality of processing entities, each having a separate one of a plurality of virtual address domains, collectively to implement network storage functions of the network storage server, each of the virtual address domains corresponding to a separate portion of a pseudo-global virtual address space;

for each said virtual address domain,

a source address translation table hierarchy for use by the corresponding processing entity, and

a target address translation table hierarchy for use by another processing entity which is a communication target, the target address translation table hierarchy incorporating part of the source address translation table hierarchy; and

code which when executed by the processor causes the target address translation table hierarchy of one of the domains to be used to translate a virtual address for communication of data between domains;

wherein;

the source address translation table hierarchy of each domain provides the corresponding processing entity with read-write permission to access a memory space of the domain; and

the target address translation table hierarchy of each domain provides read-only permission to access the memory space of the domain;

wherein the source address translation hierarchy includes a first to level, and first lower levels include at least a page directory level and a page table level, the page directory level including page directories comprising pointers to page tables of the page table level, the page tables comprising pointers to memory pages;

wherein the top level includes an access permission indicator for the corresponding memory space, the access permission indicator governing access to the corresponding memory space regardless of permission indicators of the first lower levels;

wherein the target address translation hierarchy includes a second to level that has at least one directory entry pointing to directories of the first lower levels of the source address translation hierarchy, so that the target address translation hierarchy including all levels of the source address translation hierarchy below the first to level;

wherein the second to level of the target address translation hierarchy includes an access permission indicator that is set differently from the access permission indicator in the first to level of the source translation hierarchy.

14. A network storage server as recited in claim 13, wherein each of the plurality of processing entities is a virtual machine.

15. A network storage server as recited in claim 14, wherein:

a first one of the virtual machines implements a file system layer of the network storage server;

a second one of the virtual machines implements a network access layer of the network storage server; and

a third one of the virtual machines implements a storage access layer of the network storage server.