Batch mode object creation in an elastic cloud data storage environment

• US 10,268,417 B1
• Filed: 10/24/2017
• Issued: 04/23/2019
• Est. Priority Date: 10/24/2017
• Status: Active Grant

• Current Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Original Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Inventors: Danilov, Mikhail, O'Connell, Mark A.
• Primary Examiner(s): Tsui, Daniel D
• Application Number:
  US15/792,745
• Time in Patent Office: 546 Days
• Field of Search: None
• CPC Class Codes: G06F 21/6218 to a system of files or objects, e.g. local or distributed file system or database
• CPC Class Codes: G06F 2212/1016 Performance improvement
• CPC Class Codes: G06F 2221/2143 Clearing memory, e.g. to prevent the data from being stolen
• CPC Class Codes: G06F 3/0617 in relation to availability
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/0626 Reducing size or complexity of storage systems
• CPC Class Codes: G06F 3/064 Management of blocks
• CPC Class Codes: G06F 3/065 Replication mechanisms
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]

Abstract

Batch mode object creation in an elastic cloud data storage environment is presented herein. A batch mode data chunk component is configured to: facilitate a creation of a batch of objects corresponding to an application; store first objects of the batch of objects in a group of data storage chunks; and based on a determination that a data storage chunk of the group of data storage chunks contains at least a defined amount of data, seal the data storage chunk to obtain a sealed data storage chunk; and a data protection component configured to: perform erasure encoding on the sealed data storage chunk, and replicate the sealed data storage chunk on a data recovery storage device.

Claims

1. A system, comprising:

a batch mode data chunk component configured to;

facilitate a creation of a batch of objects corresponding to an application;

store first objects of the batch of objects in a group of data storage chunks; and

based on a determination that a data storage chunk of the group of data storage chunks contains at least a defined amount of data, seal the data storage chunk to obtain a sealed data storage chunk; and

a data protection component configured to;

perform erasure encoding on the sealed data storage chunk, and

replicate the sealed data storage chunk on a data recovery storage device.

2. The system of claim 1, wherein the batch mode data chunk component is further configured to:

receive, from the application, information representing a number of objects comprising the first objects that have been included in the batch of objects, and further representing a data size of the batch of objects.

3. The system of claim 2, wherein the batch mode data chunk component is further configured to:

based on the information, create the group of data storage chunks.

4. The system of claim 1, wherein the data storage chunk is a first data storage chunk, wherein the determination is a first determination, and wherein the batch mode data chunk component is further configured to:

based on the first determination that the first data storage chunk contains the defined amount of data, and based on a second determination that second objects of the batch of objects have not been stored in the group of data storage chunks,

create a second data storage chunk of the group of data storage chunks, and

store a portion of the second objects in the second data storage chunk.

5. The system of claim 4, wherein the sealed data storage chunk is a first sealed data storage chunk, and wherein the batch mode data chunk component is further configured to:

based on a third determination that the second data storage chunk contains the defined amount of data, seal the second data storage chunk to obtain a second sealed data storage chunk.

6. The system of claim 5, wherein the data protection component is further configured to:

perform erasure encoding on the second sealed data storage chunk; and

replicate the second sealed data storage chunk on the data recovery storage device.

7. The system of claim 6, wherein the data protection component is further configured to:

send an acknowledgment directed to the application representing that the first sealed data storage chunk and the second sealed data storage chunk have been replicated on the data recovery storage device.

8. The system of claim 1, wherein the determination is a first determination, and wherein the data protection component is further configured to:

based on a second determination that the application has completed the creation of the batch of objects, and that second objects of the batch of objects have not been replicated, sending information directed to the application representing the second objects have not been replicated.

9. The system of claim 8, wherein the data storage chunk is a first data storage chunk, wherein the sealed data storage chunk is a first sealed data storage chunk, and wherein the batch mode data chunk component is further configured to:

receive the second objects from the application;

create a second data storage chunk of the group of data storage chunks;

store a portion of the second objects in the second data storage chunk; and

based on a third determination that the second data storage chunk contains the defined amount of data, seal the second data storage chunk to obtain a second sealed data storage chunk.

10. The system of claim 9, wherein the data protection component is further configured to:

perform erasure encoding on the second sealed data storage chunk, and

replicate the second sealed data storage chunk on the data recovery storage device.

11. A method, comprising:

in response to facilitating, by a system comprising a processor, generation of a group of data objects of an application of a first device, storing, by the system, a portion of the group of data objects in a group of data chunks; and

in response to a data chunk of the group of data chunks being determined to satisfy a defined condition with respect to an amount of data that has been included in the data chunk,

sealing the data chunk to obtain a sealed data chunk,

performing erasure encoding on the sealed data chunk, and

replicating the sealed data chunk on a second device of geographically distributed data storage devices.

12. The method of claim 11, wherein the facilitating comprises:

creating, by the system via the application, the group of data objects.

13. The method of claim 11, wherein the storing comprises:

receiving, from the application, a data stream comprising the portion of the group of data objects; and

storing the data stream in the group of data chunks.

14. The method of claim 11, wherein the storing comprises:

receiving, from the application, information representing at least one of;

a number of data objects of the application that have been included in the group of data objects, respective sizes of the data objects that have been included in the group of data objects, or a data size of the group of data objects; and

based on the information, generating the group of data chunks.

15. The method of claim 11, wherein the data chunk is a first data chunk, wherein the portion is a first portion, and wherein the storing comprises:

generating the first data chunk;

storing the first portion in the first data chunk; and

in response to the sealing of the first data chunk,

generating a second data chunk of the group of data chunks, and

storing a second portion of the group of data objects in the second data chunk.

16. The method of claim 15, further comprising:

sealing, by the system, the second data chunk to obtain another sealed data chunk; and

replicating, by the system, the other sealed data chunk on the second device.

17. The method of claim 11, further comprising:

sending, by the system, status information directed to the application, wherein the status information represents data objects of the group of data objects that have not been replicated.

18. A machine-readable storage medium comprising instructions that, in response to execution, cause a system comprising a processor to perform operations, comprising:

facilitating a compilation, via an application, of objects of the application into a batch of objects;

in response to receiving, from the application, a stream of data comprising the batch of objects, storing a first portion of the batch of objects in data storage chunks; and

in response to determining that a data storage chunk of the data storage chunks has been filled with a defined amount of data,

sealing the data storage chunk resulting in a sealed data storage chunk,

performing erasure coding on the sealed data storage chunk, and

replicating the sealed data storage chunk to a data storage device.

19. The machine-readable storage medium of claim 18, wherein the operations further comprise:

in response to receiving, from the application, information representing at least one of a number of the objects that have been included in the batch of objects, respective sizes of the objects that have been included in the batch of objects, or a size of the batch of objects, generating the data storage chunks based on the information.

20. The machine-readable storage medium of claim 18, wherein the operations further comprise:

sending information directed to the application specifying a second portion of the batch of objects that have not been replicated on the data storage device.