Virtual machine spanning multiple computers

• US 7,596,654 B1
• Filed: 01/26/2006
• Issued: 09/29/2009
• Est. Priority Date: 01/26/2006
• Status: Active Grant

• Current Assignee: Veritas Technologies, LLC (Whitehouse Group Ltd.)
• Original Assignee: Symantec Operating Corporation (Gen Digital Inc.)
• Inventors: Wong, Kai C.
• Primary Examiner(s): Ellis; Kevin L
• Assistant Examiner(s): OTTO, ALAN
• Application Number:
  US11/340,227
• Time in Patent Office: 1,342 Days
• Field of Search: None
• US Class Current: 711/6
• CPC Class Codes: G06F 12/0813 with a network or matrix configuration
• CPC Class Codes: G06F 12/0815 Cache consistency protocols
• CPC Class Codes: G06F 12/1027 using associative or pseudo-associative address translation means, e.g. translation look-aside buffer [TLB]
• CPC Class Codes: G06F 2212/2542 Non-uniform memory access [NUMA] architecture

Abstract

In one embodiment, a virtual NUMA system may be formed from multiple computer systems coupled to a network such as InfiniBand, Ethernet, etc. Each computer includes one or more software modules which present the resources of the computers as a virtual NUMA machine. A single instance of a guest operating system executes on the virtual NUMA machine. The guest operating system is designed to execute on a NUMA system and executes without modification on the virtual machine. The memory model of the virtual NUMA machine includes a single writer, multiple reader memory model.

Claims

1. A system comprising:

a plurality of computers coupled to a network, wherein each of the plurality of computers comprises one or more software modules which, when executed on the computer, present resources of the plurality of computers as a virtual machine on which a single instance of a guest operating system executes, during use;

wherein the virtual machine is a non-uniform memory access (NUMA) system implementing a single writer, multiple reader memory model;

wherein the guest operating system is designed to execute on a NUMA system and executes without modification on the virtual machine;

wherein each of the plurality of computers comprises a memory addressable via a local memory address space, and wherein a guest physical address space used by the guest operating system is divided into a plurality of ranges, and wherein a range of the plurality of ranges is mapped to a first portion of the memory in a computer of the plurality of computers, and wherein a corresponding portion of the guest physical address space is mapped to a first portion of the local memory address space that is assigned to the first portion of the memory;

wherein each memory comprises additional storage not mapped to any range of the plurality of ranges, and wherein the software modules use at least some of the additional storage to cache copies of data corresponding to guest physical addresses in other ones of the plurality of ranges, and wherein a second portion of the local memory address space is assigned to the additional storage, wherein the second portion is separate from the first portion of the local memory address space; and

wherein an access to a first guest physical address that is in one of the other ones of the plurality of ranges and that is not cached in the additional storage results in a page fault, and wherein a page fault handler module, which is one of the software modules, is executed in response to the page fault and determines that the page fault occurred because the first guest physical address is not cached in the additional storage, and wherein the page fault handler is configured to issue a request to acquire data corresponding to the first guest physical address from another one of the plurality of computers.

2. The system as recited in claim 1 wherein the software modules, when executed, map the guest physical addresses of the data cached in the additional storage to local memory address space addresses assigned to the additional storage, wherein the mapping from guest physical address to local memory address space addresses is stored in translation tables used by the computer during use.

3. The system as recited in claim 1 wherein the software modules, when executed, use at least some of the additional storage to store metadata corresponding to one or more ranges of the plurality of ranges, the one or more ranges mapped to the computer, and wherein coherency of the data corresponding to the respective range is maintained using the metadata.

4. The system as recited in claim 3 wherein the metadata identifies an owner of each block in the one or more ranges.

5. The system as recited in claim 4 wherein the block is a page.

6. The system as recited in claim 3 wherein the metadata indicates if one or more other computers have cached readable copies of blocks.

7. The system as recited in claim 3 wherein the coherency scheme includes weak ordering of memory accesses.

8. The system as recited in claim 1 wherein the plurality of computers comprise a configuration file describing the NUMA system, and wherein the software modules are configured to use the configuration file during boot to configure the plurality of computer systems as the virtual machine.

9. A computer accessible storage medium storing a plurality of instructions which, when executed on a first computer of a plurality of computers coupled to a network, present resources of the plurality of computers as a virtual machine on which a single instance of a guest operating system executes, during use;

wherein the virtual machine is a non-uniform memory access (NUMA) system implementing a single writer, multiple reader memory model; and

wherein the guest operating system is designed to execute on a NUMA system and executes without modification on the virtual machine, and wherein each of the plurality of computers comprises a memory addressable via a local memory address space, and wherein a guest physical address space used by the guest operating system is divided into a plurality of ranges, and the plurality of instructions, when executed, map a range of the plurality of ranges to a first portion of the memory in the first computer, and wherein a corresponding portion of the guest physical address space is mapped to a first portion of the local memory address space that is assigned to the first portion of the memory; and

wherein the memory in the first computer comprises additional storage not mapped to any range of the plurality of ranges, and wherein the plurality of instructions, when executed, use at least some of the additional storage to cache copies of data corresponding to guest physical addresses in other ones of the plurality of ranges, and wherein a second portion of the local memory address space is assigned to the additional storage, wherein the second portion is separate from the first portion of the local memory address space, and wherein an access to a first guest physical address that is in one of the other ones of the plurality of ranges and that is not cached in the additional storage results in a page fault, and wherein the plurality of instructions include a page fault handler module that is executed in response to the page fault and determines that the page fault occurred because the first guest physical address is not cached in the additional storage, and wherein the page fault handler is configured to issue a request to acquire data corresponding to the first guest physical address from another one of the plurality of computers.

10. The computer accessible storage medium as recited in claim 9 wherein the plurality of instructions, when executed, map the guest physical addresses of the data cached in the additional storage to local memory address space addresses in the additional storage, wherein the mapping of the guest physical addresses to the local memory address space address is stored in translation tables used by the computer during use.

11. The computer accessible storage medium as recited in claim 9 wherein the plurality of instructions, when executed, use at least some of the additional storage to store metadata corresponding to one or more ranges of the plurality of ranges that are mapped to the computer, and maintain coherency of the data corresponding to the one or more ranges using the metadata.

12. A computer comprising a memory addressable via a local memory address space, execution resources configured to execute instructions, during use, and hardware to couple to a network, wherein the execution resources, during use, execute one or more software modules that configure the computer with other computers coupled to the network to form a virtual machine on which a single instance of a guest operating system executes, during use, wherein the virtual machine is a non-uniform memory access (NUMA) system implementing a single writer, multiple reader memory model, and wherein the guest operating system is designed to execute on a NUMA system and executes without modification on the virtual machine;

and wherein at least one range of a guest physical address space used by the guest operating system is mapped to a first portion of the memory, and wherein a corresponding portion of the guest physical address space is mapped to a first portion of the local memory address space that is assigned to the first portion of the memory, and wherein the memory comprises additional storage not mapped to the range, and wherein the software modules use at least some of the additional storage to cache copies of data corresponding to guest physical addresses mapped to other computers, and wherein a second portion of the local memory address space is assigned to the additional storage, wherein the second portion is separate from the first portion of the local memory address space;

wherein an access to a first guest physical address that is in one of the other ones of the plurality of ranges and that is not cached in the additional storage results in a page fault, and wherein a page fault handler module, which is one of the software modules, is executed in response to the page fault and determines that the page fault occurred because the first guest physical address is not cached in the additional storage, and wherein the page fault handler is configured to issue a request to acquire data corresponding to the first guest physical address from another one of the computers.

13. The computer as recited in claim 12 wherein the software modules, when executed, map the guest physical addresses of the data cached in the additional storage to local memory address space addresses assigned to the additional storage, the mapping stored in translation tables used by the respective computer during use.

14. The computer as recited in claim 12 wherein the software modules, when executed, use at least some of the additional storage to store metadata corresponding to the respective range, and maintain coherency of the data corresponding to the respective range using the metadata.

15. In a system comprising a plurality of computers coupled to a network, a method comprising:

presenting resources of the plurality of computers as a virtual machine on which a single instance of a guest operating system executes, during use;

wherein the virtual machine is a non-uniform memory access (NUMA) system implementing a single writer, multiple reader memory model;

wherein the guest operating system is designed to execute on a NUMA system;

executing the guest operating system without modification on the virtual machine;

wherein each of the plurality of computers comprises a memory addressable via a local memory address space, and wherein a guest physical address space used by the guest operating system is divided into a plurality of ranges, and wherein a range of the plurality of ranges is mapped to a first portion of the memory in a computer of the plurality of computers, and wherein a corresponding portion of the guest physical address space is mapped to a first portion of the local memory address space that is assigned to the first portion of the memory; and

wherein each memory comprises additional storage not mapped to any range of the plurality of ranges, and wherein the method further comprises using at least some of the additional storage to cache copies of data corresponding to guest physical addresses in other ones of the plurality of ranges, and wherein a second portion of the local memory address space is assigned to the additional storage, wherein the second portion is separate from the first portion of the local memory address space;

executing an access to a first guest physical address that is in one of the other ones of the plurality of ranges and that is not cached in the additional storage;

detecting a page fault for the access;

executing a page fault handler module in response to the page fault, the page fault handler module determining that the page fault occurred because the first guest physical address is not cached in the additional storage and issuing a request to acquire data corresponding to the first guest physical address from another one of the plurality of computers.

16. The method as recited in claim 15 further comprising mapping the guest physical addresses of the data cached in the additional storage to local memory address space addresses assigned to the additional storage, the mapping stored in translation tables used by the computer during use.

17. The method as recited in claim 15 further comprising using at least some of the additional storage to store metadata corresponding to one or more ranges of the plurality of ranges, the one or more ranges mapped to the computer, and wherein coherency of the data corresponding to the respective range is maintained using the metadata.

18. The method as recited in claim 17 wherein the metadata identifies an owner of each block in the one or more ranges.

19. The method as recited in claim 18 wherein the metadata indicates if one or more other computers have cached readable copies of blocks.

20. The method as recited in claim 15 wherein the plurality of computers comprise a configuration file describing the NUMA system, and wherein the method further comprises using the configuration file during boot to configure the plurality of computer systems as the virtual machine.