Dispersed storage with partial data object storage and methods for use therewith

• US 9,558,071 B2
• Filed: 07/28/2014
• Issued: 01/31/2017
• Est. Priority Date: 01/28/2010
• Status: Active Grant

• Current Assignee: Pure Storage, Inc.
• Original Assignee: International Business Machines Corporation
• Inventors: Baptist, Andrew, Volvovski, Ilya
• Primary Examiner(s): SCHELL, JOSEPH O
• Application Number:
  US14/444,183
• Publication Number:
  US 20140337394A1
• Time in Patent Office: 918 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 11/1008 in individual solid state devices G06F11/1004 takes precedence
• CPC Class Codes: G06F 11/1076 Parity data used in redundant arrays of independent storages, e.g. in RAID systems
• CPC Class Codes: G06F 11/1448 Management of the data involved in backup or backup restore
• CPC Class Codes: G06F 16/182 Distributed file systems
• CPC Class Codes: G06F 16/184 implemented as replicated file system
• CPC Class Codes: G06F 16/1844 Management specifically adapted to replicated file systems
• CPC Class Codes: G06F 2201/805 Real-time
• CPC Class Codes: G06F 2201/84 Using snapshots, i.e. a logical point-in-time copy of the data
• CPC Class Codes: G06F 2211/1028 Distributed, i.e. distributed RAID systems with parity
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

A first section of the data object is received that contains less than a full amount of the data object. A processing module of a distributed storage (DS) processing unit determines that communication of the data object has been interrupted and whether to store the first section of the data object. The first section of the data object is stored in a DS network when determined to store the first section of the data object.

Claims

1. A method comprises:

receiving, at a processing module of a distributed storage (DS) processing unit, a store data object request;

receiving, at the processing module of the distributed storage (DS) processing unit, a first section of a data object that is less than and not equal to a full amount of the data object;

determining, via the processing module of the distributed storage (DS) processing unit, that communication of the data object has been interrupted;

determining, via the processing module of the distributed storage (DS) processing unit, whether to store the first section of the data object; and

storing the first section of the data object in a DS network when determined to store the first section of the data object.

2. The method of claim 1 further comprising:

generating a directory file associated with the data object that includes a plurality of entries associated with the first section, wherein the plurality of entries associated with the first section include a file or directory name field associated with the first section, a size associated with the first section and a data offset associated with the first section.

3. The method of claim 2 further comprising:

(a) receiving, at the processing module of the distributed storage (DS) processing unit, a second section of the data object that contains an additional amount of the data object;

(b) storing the second section of the data object in the DS network; and

(c) updating the directory file associated with the data object to include a plurality of entries associated with the second section, wherein the plurality of entries associated with the second section includes a file or directory name field associated with the second section, a size associated with the second section and a data offset associated with the second section.

4. The method of claim 3 further comprising:

repeating steps (a)-(c) until the full amount of the data object has been received and stored in the DS network.

5. The method of claim 3 further comprising:

determining a next available DS address to store the second section of the data object; and

wherein the second section of the data object is stored in the DS network in accordance with the next available DS address.

6. The method of claim 1 wherein the store data object request includes one or more of:

a user ID, a data object name, or a data size of the data object.

7. The method of claim 1 wherein determining whether to store the first section of the data object is based on determining that the communication of the data object has been interrupted and further based on one or more of:

an amount of data received in the first section, a data size of the data object, a time indicator, or a maximum size per section indicator.

8. A dispersed storage (DS) processing unit comprises:

at least one module that includes processing circuitry, when operable within a computing device, that causes the computing device to;

receive a store data object request;

receive a first section of a data object that is less than and not equal to a full amount of the data object;

determine that communication of the data object has been interrupted;

determine whether to store the first section of the data object; and

store the first section of the data object in a DS network when determined to store the first section of the data object.

9. The DS processing unit of claim 8 wherein the at least one module that includes the processing circuitry, when operable within the computing device, further causes the computing device to:

generate a directory file associated with the data object that includes a plurality of entries associated with the first section, wherein the plurality of entries associated with the first section include a file or directory name field associated with the first section, a size associated with the first section and a data offset associated with the first section.

10. The DS processing unit of claim 9 wherein the at least one module that includes the processing circuitry, when operable within the computing device, further causes the computing device to:

(a) receive a second section of the data object that contains an additional amount of the data object;

(b) store the second section of the data object in the DS network; and

(c) update the directory file associated with the data object to include a plurality of entries associated with the second section, wherein the plurality of entries associated with the second section includes a file or directory name field associated with the second section, a size associated with the second section and a data offset associated with the second section.

11. The DS processing unit of claim 10 wherein the at least one module that includes the processing circuitry, when operable within the computing device, further causes the computing device to:

repeat steps (a)-(c) until the full amount of the data object has been received and stored in the DS network.

12. The DS processing unit of claim 10 wherein the at least one module that includes the processing circuitry, when operable within the computing device, further causes the computing device to:

determine a next available DS address to store the second section of the data object; and

wherein the second section of the data object is stored in the DS network in accordance with the next available DS address.

13. The DS processing unit of claim 8 wherein the store data object request includes one or more of:

a user ID, a data object name, or a data size of the data object.

14. The DS processing unit of claim 8 wherein determining whether to store the first section of the data object is based on determining that the communication of the data object has been interrupted and further based on one or more of:

an amount of data received in the first section, a data size of the data object, a time indicator, or a maximum size per section indicator.

15. A non-transitory computer readable storage medium comprises:

at least one non-transitory memory section that stores operational instructions that, when executed by one or more processing modules that includes processing circuitry of one or more computing devices of a dispersed storage network (DSN), causes the one or more computing devices to;

receive a store data object request;

receive a first section of a data object that is less than and not equal to a full amount of the data object;

determine that communication of the data object has been interrupted;

determine whether to store the first section of the data object; and

store the first section of the data object in a DS network when determined to store the first section of the data object.

16. The non-transitory computer readable storage medium of claim 15 wherein the operational instructions, when executed by the one or more processing modules that includes the processing circuitry of the one or more computing devices of the dispersed storage network (DSN), further causes the one or more computing devices to:

generate a directory file associated with the data object that includes a plurality of entries associated with the first section, wherein the plurality of entries associated with the first section include a file or directory name field associated with the first section, a size associated with the first section and a data offset associated with the first section.

17. The non-transitory computer readable storage medium of claim 16 wherein the operational instructions, when executed by the one or more processing modules that includes the processing circuitry of the one or more computing devices of a dispersed storage network (DSN), further causes the one or more computing devices to:

(a) receive a second section of the data object that contains an additional amount of the data object;

(b) store the second section of the data object in the DS network; and

(c) update the directory file associated with the data object to include a plurality of entries associated with the second section, wherein the plurality of entries associated with the second section includes a file or directory name field associated with the second section, a size associated with the second section and a data offset associated with the second section.

18. The non-transitory computer readable storage medium of claim 17 wherein the operational instructions, when executed by the one or more processing modules that includes the processing circuitry of the one or more computing devices of the dispersed storage network (DSN), further causes the one or more computing devices to:

repeat steps (a)-(c) until the full amount of the data object has been received and stored in the DS network.

19. The non-transitory computer readable storage medium of claim 17 wherein the operational instructions, when executed by the one or more processing modules that includes the processing circuitry of the one or more computing devices of the dispersed storage network (DSN), further causes the one or more computing devices to:

determine a next available DS address to store the second section of the data object; and

wherein the second section of the data object is stored in the DS network in accordance with the next available DS address.

20. The non-transitory computer readable storage medium of claim 15 wherein determining whether to store the first section of the data object is based on determining that the communication of the data object has been interrupted and further based on one or more of:

an amount of data received in the first section, a data size of the data object, a time indicator, or a maximum size per section indicator.