Data management tier coupling primary storage and secondary storage

• US 9,672,165 B1
• Filed: 05/21/2014
• Issued: 06/06/2017
• Est. Priority Date: 05/21/2014
• Status: Active Grant

• Current Assignee: Veritas Technologies, LLC (Whitehouse Group Ltd.)
• Original Assignee: Veritas Technologies, LLC (Whitehouse Group Ltd.)
• Inventors: Satish, Venkeepuram, Pendharkar, Niranjan
• Primary Examiner(s): SNYDER, STEVEN G
• Application Number:
  US14/284,070
• Time in Patent Office: 1,112 Days
• Field of Search: None
• CPC Class Codes: G06F 11/0709 in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
• CPC Class Codes: G06F 11/1456 Hardware arrangements for backup
• CPC Class Codes: G06F 11/1464 for networked environments
• CPC Class Codes: G06F 11/1469 Backup restoration techniques
• CPC Class Codes: G06F 11/2074 Asynchronous techniques
• CPC Class Codes: G06F 11/3006 where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems multiprogramming arrangements G06F9/46; allocation of resources G06F9/50
• CPC Class Codes: G06F 11/3034 where the computing system component is a storage system, e.g. DASD based or network based digital input from or digital output to record carriers G06F3/06; digital recording or reproducing G11B20/18; for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097
• CPC Class Codes: G06F 11/3476 Data logging G06F11/14, G06F11/2205 take precedence
• CPC Class Codes: G06F 13/10 Program control for peripheral devices G06F13/14 - G06F13/42 take precedence
• CPC Class Codes: G06F 16/00 Information retrieval; Database structures therefor; File system structures therefor
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
• CPC Class Codes: G06F 2201/815 Virtual
• CPC Class Codes: G06F 2201/855 Details of asynchronous mirroring using a journal to transfer not-yet-mirrored changes
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]

Abstract

A method of storing data in a storage system is provided. The method includes writing data, from a virtual machine in a compute node, to a primary storage coupled to the virtual machine and sending a first copy of the data from the compute node to a data node. The method includes writing the first copy of the data from the compute node to an intermediate storage coupled as direct attached storage to the data node, and writing a second copy of data from the intermediate storage to a secondary storage, wherein at least one method operation is performed by a processor. A storage system is also provided.

Claims

1. A method of storing data in a storage system, comprising:

writing data, from a virtual machine in a compute node, to a temporary storage in the compute node and coupled to the virtual machine and in parallel with the writing data to the temporary storage, reflecting the write onto another compute node;

copying data from the temporary storage to a primary storage coupled to the virtual machine;

reading data from the temporary storage;

sending the data that is read from the temporary storage in the compute node to a data node;

writing the data that is read from the temporary storage in the compute node to an intermediate storage coupled as direct attached storage to the data node; and

writing a second copy of data from the intermediate storage to a secondary storage; and

reusing space in the temporary storage, responsive to the sending the data from the temporary storage to the data node.

2. The method of claim 1, further comprising:

reading data from the intermediate storage, as recovered data; and

writing the recovered data from the intermediate storage to the primary storage.

3. The method of claim 1, further comprising:

reading data from the secondary storage, as recovered data;

writing the recovered data to the intermediate storage; and

writing the recovered data to the compute node.

4. The method of claim 1, further comprising:

writing data to a temporary storage coupled to the compute node, wherein the writing data to the primary storage includes copying data from the temporary storage to a virtual disk that is local to the virtual machine, the virtual disk configured in the primary storage.

5. The method of claim 1, further comprising:

writing data to a temporary storage; and

performing a synchronous write to another compute node, of the data written to the temporary storage.

6. The method of claim 1, further comprising:

writing data to a temporary storage, wherein sending a copy of the data from the compute node to a data node includes sending a copy of the data from the temporary storage to the data node for storage in the intermediate storage; and

freeing up space in the temporary storage in response to an acknowledgment from the data node of writing the copy of the data from the temporary storage to the intermediate storage.

7. The method of claim 1, wherein a periodicity of the writing data from the virtual machine to the primary storage is shorter than a periodicity of the sending a copy of the data from the compute node to the data node.

8. A method of operating a storage system having primary and secondary storage, the method comprising:

writing data to a writeback log coupled to a compute node, the compute node having a virtual machine and in parallel with the writing data to the writeback log, reflecting the write onto another compute node;

copying data that is read from the writeback log to a primary storage coupled as direct attached storage to the compute node; and

copying data that is read from the writeback log to a data node, the data node storing the data copied from the writeback log in an intermediate storage coupled as direct attached storage to the data node, wherein the intermediate storage is distinct from a secondary storage employed to backup data from the intermediate storage; and

reusing space in the writeback log, responsive to copying data from the writeback log to the data node.

9. The method of claim 8, wherein the primary storage includes a configuration as a virtual disk, and copying data from the writeback log to the primary storage includes copying data from the writeback log to the virtual disk.

10. The method of claim 8, wherein:

the copying data from the writeback log to the primary storage occurs more often than the copying data from the writeback log to the data node.

11. The method of claim 8, further comprising:

tracking the writing data to the writeback log, the copying data from the writeback log to the primary storage, and the copying data from the writeback log to the data node, via application of a pointer; and

reusing space in the writeback log, responsive to a completion of the copying data from the writeback log to the data node.

12. The method of claim 8, further comprising:

recovering data to the compute node from one of;

the writeback log, the intermediate storage, storage on a reflected node, or the secondary storage.

13. A storage system, comprising:

a compute node, having a temporary storage and a primary storage that are local to the compute node, the compute node configured to implement a virtual machine;

a data node, having an intermediate storage local to the data node, the data node configured to couple to a secondary storage;

the compute node configured to;

write data to the temporary storage, copy data that is read from the temporary storage to the primary storage, and copy data that is read from the temporary storage to the data node;

a further compute node, having a further temporary storage;

the compute node configured to perform a reflect write to the further compute node in parallel with the write data to the temporary storage;

the compute node configured to reuse space in the temporary storage, responsive to copying data from the temporary storage to the data node;

the data node configured to write the data that is read from the temporary storage to the intermediate storage; and

the data node configured to copy data from the intermediate storage to the secondary storage, in a backup operation.

14. The storage system of claim 13, further comprising:

the temporary storage including one of;

a circular file, a circular buffer, or a circular cache.

15. The storage system of claim 13, further comprising:

a first network, configured to couple the data node and a further data node to the compute node and a further compute node;

a second network configured to couple a control node of the compute node, a further control node of a further compute node, the data node, and the further data node to a management node;

a third network configured to couple the compute node and the further compute node; and

a fourth network configured to couple the data node, the further data node and a network having further storage.

16. The storage system of claim 13, further comprising:

the compute node having a control node configured to provision virtual machines in the compute node; and

the compute node having a hypervisor, configured to manage the virtual machines.

17. The storage system of claim 13, further comprising:

a further compute node;

the compute node configured to perform, as a reflected write, a synchronous write to the further compute node in parallel with the compute node write of the data to the temporary storage.

18. The storage system of claim 13, further comprising:

the compute node configured to free up space in the temporary storage, responsive to completion of the copy data from the temporary storage to the data node and completion of a reflected write.

19. The storage system of claim 13, further comprising:

the compute node configured to recover data from the temporary storage; and

the data node configured to recover data from the intermediate storage to the compute node, and configured to recover data from the secondary storage to the compute node.

20. A storage system, comprising:

a compute node, configured to implement a virtual machine;

a writeback log coupled to the compute node;

a primary storage as direct attached storage to the compute node;

a data node, implemented on a computing device, the data node configured to couple to a secondary storage;

an intermediate storage as direct attached storage to the data node;

the compute node configured to write data to the writeback log and the primary storage, and to send a copy of data that is read from the writeback log to the data node;

a further compute node, having a further writeback log;

the compute node configured to perform a reflect write to the further compute node in parallel with the write data to the writeback log;

the compute node configured to reuse space in the writeback log, responsive to the send a copy of data from the writeback log to the data node and an acknowledgment from the further compute node as to the reflect write;

the data node configured to write the copy of data that is read from the writeback log to the intermediate storage; and

the data node configured to write the copy of data from the intermediate storage to the secondary storage.

21. The storage system of claim 20, further comprising:

the writeback log including a pointer and one of;

a circular file, a circular buffer, or a circular cache.

22. The storage system of claim 20, further comprising:

one or more networks configured to couple compute nodes, including the compute node, data nodes, including the data node, further storage, a service, and a management node.

23. The storage system of claim 20, wherein the compute node configured to write data to the writeback log in the primary storage includes the compute node configured to write data to the writeback log and to copy data from the writeback log into a virtual disk configured in the primary storage.

24. The storage system of claim 20, further comprising:

a flash memory coupled to the compute node, wherein the writeback log is implemented in the flash memory.

25. The storage system of claim 20, further comprising:

the data node configured to support off-hosting of one of;

virus scan, data analytics, or backup, independent of the compute node.

26. The storage system of claim 20, further comprising:

the intermediate storage configured as a plurality of virtual disks, with at least one of the plurality of virtual disks available for analytics or warehousing applications executing on one of the compute node or a further compute node.