Systems and methods for integrating compute resources in a storage area network

• US 8,943,284 B2
• Filed: 03/14/2013
• Issued: 01/27/2015
• Est. Priority Date: 03/14/2013
• Status: Active Grant

• Current Assignee: Joyent, Inc. (Samsung Electronics Co. Ltd.)
• Original Assignee: Joyent, Inc. (Samsung Electronics Co. Ltd.)
• Inventors: Hoffman, Jason Alexander
• Primary Examiner(s): DOAN, DUC T
• Application Number:
  US13/829,250
• Publication Number:
  US 20140281304A1
• Time in Patent Office: 684 Days
• Field of Search: None
• US Class Current: 711/162
• CPC Class Codes: G06F 11/3006 where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems multiprogramming arrangements G06F9/46; allocation of resources G06F9/50
• CPC Class Codes: G06F 11/3034 where the computing system component is a storage system, e.g. DASD based or network based digital input from or digital output to record carriers G06F3/06; digital recording or reproducing G11B20/18; for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097
• CPC Class Codes: G06F 11/3055 Monitoring arrangements for monitoring the status of the computing system or of the computing system component, e.g. monitoring if the computing system is on, off, available, not available error or fault processing without redundancy G06F11/0703; error detection or correction by redundancy in data representation G06F11/08; error detection or correction by redundancy in operation G06F11/14; error detection or correction by redundancy in hardware G06F11/16
• CPC Class Codes: G06F 2009/45562 Creating, deleting, cloning virtual machine instances
• CPC Class Codes: G06F 2009/45583 Memory management, e.g. access or allocation
• CPC Class Codes: G06F 2201/815 Virtual middleware or OS functionality using virtual machines to implement generic software techniques for error detection or fault masking G06F11/1484
• CPC Class Codes: G06F 3/065 Replication mechanisms
• CPC Class Codes: G06F 9/45554 Instruction set architectures of guest OS and hypervisor or native processor differ, e.g. Bochs or VirtualPC on PowerPC MacOS

Abstract

A data processing and storage system includes a compute module for running at least one virtual machine for processing guest data. State data on the at least one virtual machine is collected. The system also includes a storage module communicating with the compute module and storing the guest data. The storage module accesses the state data for controlling storage operations. A cloud storage/compute system is provided that includes a storage module for storing guest data for a virtual machine and operating based on a clock. The cloud storage/compute system also includes a compute module coupled to the storage module for performing operations on the guest data for the virtual machine and operating based on the clock. A method is provided that includes collecting state data on a virtual machine that processes guest data, and controlling storage operations relating to the guest data based on the state data.

Claims

1. A storage area network system, comprising:

an object store comprising a plurality of tenants;

a compute module for;

assigning to each of the plurality of tenants a virtual machine,

instantiating the virtual machine for each of the plurality of tenants directly on the object store;

running at least one virtual machine that is used to process guest data in the object store using at least one compute function that is executed directly on the object store in such a way that data is not moved from the object store, each of the compute operations having a timestamp generated by a clock;

collecting context data about each of the virtual machines of the plurality of tenants

a storage module for;

communicating with the compute module and executing storage operations for storing the guest data, the storage module accessing the context data for controlling storage operations, each of the storage operations comprising a timestamp generated by the clock, each of the compute operations and the storage operations comprising timestamps respectively; and

a context memory for;

storing context data generated by the virtual machine, the context memory being coupled to device drivers of the storage module, the context data being used to dynamically modify the device drivers, the context data comprising a correlation of the storage operations and the compute operations using the timestamps of the storage operations and the timestamps of the compute operations, wherein the timestamps associated with the storage operations are correlated with the timestamps associated with the compute operations, in the context memory.

2. The system of claim 1, wherein the context data is used to dynamically modify control software of the storage module.

3. The system of claim 2, wherein the context data is used to manage input/output throttling of the guest data with respect to the storage module.

4. The system of claim 1, wherein the context data is accessible by a system administrator for at least one of determining and modifying resource usage related to the guest data.

5. The system of claim 1, further comprising a clock accessible by the compute module and the storage module for managing operations.

6. The system of claim 1, further comprising a debug module adapted to access the context data and provide a stack trace for the compute module and the storage module.

7. The system of claim 1, wherein the storage module stores a read-only copy of a virtual machine operating system for instantiating new instances of virtual machines in the compute module, the storage module further storing modifications to the virtual machine operating system in a delta file on an image disk, separately from the read-only copy of the virtual machine operating system, the delta file being associated with the read-only copy of the virtual machine operating system by a pointer file.

8. An object store, comprising:

a storage module for executing storage operations comprising storing guest data for a virtual machine and operating based on a clock, each of the storage operations comprising a timestamp, context data about the virtual machine being collected, wherein the context data comprises an internal application identifier and an internal application code path for the virtual machine;

a compute module coupled to the storage module for performing compute operations on the guest data for the virtual machine and operating based on the clock each of the storage operations comprising a timestamp respectively; and

a context memory for storing context data generated by the virtual machine, the context memory being coupled to device drivers of the storage module, the context data being used to dynamically modify the device drivers, the context data comprising a correlation of the storage operations and the compute operations using the timestamps of the storage operations and the timestamps of the compute operations, wherein the timestamps associated with the storage operations are correlated with the timestamps associated with the compute operations, in the context memory.

9. The object store of claim 8, further comprising a cloud system administrator module accessing storage module operations data and compute module operations data for managing operations.

10. The object store of claim 9, further comprising a debug module accessing the storage module operations data and the compute module operations data for providing a stack trace.

11. The object store of claim 8, wherein the context data is used to at least one of dynamically modify control software of the storage module and manage input/output throttling of the guest data with respect to the storage module.

12. A method comprising:

collecting context data on each of the virtual machines of the plurality of tenants, the context data comprising a process identifier and a username, a compute module that executes compute operations comprising instantiating each of the virtual machines directly on the object store without moving guest data from the object store, the context data comprising a correlation of storage operations and the compute operations using timestamps of the storage operations and timestamps of the compute operations, wherein the timestamps associated with the storage operations are correlated with the timestamps associated with the compute operations, in the context memory;

controlling storage operations for the object store relating to the guest data based on the context data, each of the compute operations and the storage operations comprising timestamps respectively;

generating correlated data that comprises correlating the compute operations of the compute module and the storage operations of the storage module using timestamps; and

outputting the correlated data to an administrator terminal.

13. The method of claim 12, further comprising:

communicating by at least one virtual machine the guest data to a storage module; and

storing the guest data in the storage module.

14. The method of claim 13, further comprising:

storing in the storage module a read-only copy of a virtual machine operating system for instantiating new instances of virtual machines; and

storing in the storage module modifications to the virtual machine in an instance image file.

15. The method of claim 14, wherein the context data is used to:

dynamically modify control software of the storage module;

modify device drivers of the storage module;

manage input/output throttling of the guest data with respect to the storage module;

determine resource usage related to the guest data;

modify resource usage related to the guest data;

debug at least one of the virtual machine and the storage module; and

provide a stack trace.

16. The method of claim 12, further comprising managing processing operations and the storage operations based on a clock accessible by the virtual machine and the storage module.

17. The method according to claim 12, wherein the context data further comprises a process identifier.

18. The method according to claim 12, wherein the context data further comprises a username.

19. The method according to claim 12, wherein the context data further comprises a memory identifier.

20. The method according to claim 12, wherein the context data further comprises an internal application identifier.

21. A system for processing and storing data, comprising:

a compute module running at least one virtual machine that is used to process guest data in an object store using at least one compute function that is executed directly on the object store in such a way that data is not moved from the object store, the compute module accessing a clock for managing compute operations, context data on the at least one virtual machine being collected;

a storage module communicating with the compute module and storing the guest data, the storage module accessing the context data that is used to dynamically modify control software of the storage module and to manage input/output throttling of the guest data with respect to the storage module, the storage module storing a read-only copy of a virtual machine operating system for instantiating new instances of virtual machines in the compute module, the storage module accessing the clock for managing storage operations;

a context module for correlating the compute operations of the compute module and the storage operations of the storage module using timestamps, each of the compute operations and the storage operations comprising timestamps respectively; and

a system administrator module accessing the context data for determining resource usage related to the guest data and outputting the correlated data to an administrator terminal.