Large object parallel writing

• US 10,114,697 B2
• Filed: 11/21/2016
• Issued: 10/30/2018
• Est. Priority Date: 06/25/2012
• Status: Expired due to Fees

• Current Assignee: Pure Storage, Inc.
• Original Assignee: International Business Machines Corporation
• Inventors: Gray, Adam M., Leggette, Wesley B., Resch, Jason K.
• Primary Examiner(s): Ko, Chae M
• Application Number:
  US15/357,814
• Publication Number:
  US 20170068597A1
• Time in Patent Office: 708 Days
• Field of Search: 714 622
• CPC Class Codes: G06F 11/008 Reliability or availability analysis
• CPC Class Codes: G06F 11/1076 Parity data used in redundant arrays of independent storages, e.g. in RAID systems
• CPC Class Codes: G06F 2211/109 Sector level checksum or ECC, i.e. sector or stripe level checksum or ECC in addition to the RAID parity calculation
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/064 Management of blocks
• CPC Class Codes: G06F 3/0644 Management of space entities, e.g. partitions, extents, pools
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: H03M 13/05 using block codes, i.e. a predetermined number of check bits joined to a predetermined number of information bits H03M13/2906 takes precedence
• CPC Class Codes: H03M 13/1515 Reed-Solomon codes
• CPC Class Codes: H03M 13/3761 using code combining, i.e. using combining of codeword portions which may have been transmitted separately, e.g. Digital Fountain codes, Raptor codes or Luby Transform [LT] codes
• CPC Class Codes: H03M 13/611 Specific encoding aspects, e.g. encoding by means of decoding

Abstract

A method includes partitioning data into first and second partitions and dispersed storage (DS) error encoding the first and second partition into first and second plurality of sets of encoded data slices (EDSs). The method further includes generating first SAT regarding storage of the first plurality of sets of EDSs and second SAT regarding storage of the second plurality of sets of EDSs. The method further includes DS error encoding the first and the second SAT to produce first and second sets of SAT slices, sending the first plurality of sets of EDSs and the first set of SAT slices to the first set of storage units, and sending the second plurality of sets of EDSs and the second set of SAT slices to the second set of storage units. The method further includes generating a third SAT regarding storage of the first and second sets of SAT slices.

Claims

1. A method comprises:

partitioning, by a computing device of a dispersed storage network (DSN), a data object into a first partition and a second partition;

dispersed storage error encoding, by the computing device, the first partition into a first plurality of sets of encoded data slices and the second partition into a second plurality of sets of encoded data slices;

generating, by the computing device, a first segment allocation table (SAT) regarding storage of the first plurality of sets of encoded data slices in a first set of storage units of the DSN and a second SAT regarding storage of the second plurality of sets of encoded data slices in a second set of storage units of the DSN;

dispersed storage error encoding, by the computing device, the first SAT to produce a first set of SAT slices and the second SAT to produce a second set of SAT slices;

sending, by the computing device, the first plurality of sets of encoded data slices and the first set of SAT slices to the first set of storage units;

sending, by the computing device, the second plurality of sets of encoded data slices and the second set of SAT slices to the second set of storage units; and

generating, by the computing device, a third SAT regarding storage of the first and second sets of SAT slices in the first and second set of storage units.

2. The method of claim 1 further comprises:

determining, by the computing device, to partition the data object based on one of;

a data object attribute;

a partitioning scheme lookup; and

receiving the partitioning scheme.

3. The method of claim 1, wherein the first SAT comprises:

a start segment vault source name entry indicating a vault source name associated with storage of a first set of encoded data slices of the first plurality of sets of encoded data slices, wherein a first data segment of one or more data segments of the first partition is dispersed error encoded into the first set of encoded data slices;

a segment size entry indicating a number of bytes of each data segment of the one or more data segments of the first partition; and

a total length entry indicating a number of bytes of all of the data segments of the one or more data segments of the first partition.

4. The method of claim 1, wherein the second SAT comprises:

a start segment vault source name entry indicating a vault source name associated with storage of a first set of encoded data slices of the second plurality of sets of encoded data slices, wherein a first data segment of one or more data segments of the second partition is dispersed error encoded into the first set of encoded data slices;

a segment size entry indicating a number of bytes of each data segment of the one or more data segments of the second partition; and

a total length entry indicating a number of bytes of all of the data segments of the one or more data segments of the second partition.

5. The method of claim 1, wherein the third SAT comprises:

a first data entry region including;

a start segment vault source name entry indicating a vault source name associated with storage of a first set of encoded data slices of the second plurality of sets of encoded data slices, wherein a first data segment of one or more data segments of the second partition is dispersed error encoded into the first set of encoded data slices;

a segment size entry indicating a number of bytes of each data segment of the one or more data segments of the second partition; and

a total length entry indicating a number of bytes of all of the data segments of the one or more data segments of the second partition; and

a second data entry region including;

a start segment vault source name entry indicating a vault source name associated with storage of a first set of encoded data slices of the second plurality of sets of encoded data slices, wherein a first data segment of one or more data segments of the second partition is dispersed error encoded into the first set of encoded data slices;

a segment size entry indicating a number of bytes of each data segment of the one or more data segments of the second partition; and

a total length entry indicating a number of bytes of all of the data segments of the one or more data segments of the second partition.

6. The method of claim 1 further comprises:

dispersed storage error encoding, by the computing device, the third SAT to produce a third set of SAT slices; and

sending, by the computing device, the third set of SAT slices to the first set of storage units or the second set of storage units.

7. The method of claim 1 further comprises:

updating, by the computing device, a directory with a first SAT vault source name, a second SAT vault source name, and a third SAT vault source name.

8. The method of claim 1 further comprises:

further partitioning, by the computing device, the data object into a third partition;

dispersed storage error encoding, by the computing device, the third partition into a third plurality of sets of encoded data slices;

generating, by the computing device, a fourth SAT regarding storage of the third plurality of sets of encoded data slices in a third set of storage units of the DSN;

dispersed storage error encoding, by the computing device, the fourth SAT to produce a fourth set of SAT slices;

sending, by the computing device, the third plurality of sets of encoded data slices and the fourth set of SAT slices to the third set of storage units; and

generating, by the computing device, a fifth SAT regarding storage of the first, second, and third sets of SAT slices in the first, second, and third set of storage units.

9. The method of claim 1 further comprises:

dispersed storage error encoding the first partition into the first plurality of sets of encoded data slices and the dispersed storage error encoding of the second partition into the second plurality of sets of encoded data slices substantially concurrently.

10. A computing device of a dispersed storage network (DSN), the computing device comprises:

an interface;

memory; and

a processing module operably coupled to the memory and the interface, wherein the processing module is operable to;

partition a data object into a first partition and a second partition;

dispersed storage error encode the first partition into a first plurality of sets of encoded data slices and the second partition into a second plurality of sets of encoded data slices;

generate a first segment allocation table (SAT) regarding storage of the first plurality of sets of encoded data slices in a first set of storage units of the DSN and a second SAT regarding storage of the second plurality of sets of encoded data slices in a second set of storage units of the DSN;

dispersed storage error encode the first SAT to produce a first set of SAT slices and the second SAT to produce a second set of SAT slices;

send the first plurality of sets of encoded data slices and the first set of SAT slices to the first set of storage units;

send the second plurality of sets of encoded data slices and the second set of SAT slices to the second set of storage units; and

generate a third SAT regarding storage of the first and second sets of SAT slices in the first and second set of storage units.

11. The computing device of claim 10, wherein the processing module is operable to:

determine to partition the data object based on one of;

a data object attribute;

a partitioning scheme lookup; and

receiving the partitioning scheme.

12. The computing device of claim 10, wherein the first SAT comprises:

a start segment vault source name entry indicating a vault source name associated with storage of a first set of encoded data slices of the first plurality of sets of encoded data slices, wherein a first data segment of one or more data segments of the first partition is dispersed error encoded into the first set of encoded data slices;

a segment size entry indicating a number of bytes of each data segment of the one or more data segments of the first partition; and

a total length entry indicating a number of bytes of all of the data segments of the one or more data segments of the first partition.

13. The computing device of claim 10, wherein the second SAT comprises:

a start segment vault source name entry indicating a vault source name associated with storage of a first set of encoded data slices of the second plurality of sets of encoded data slices, wherein a first data segment of one or more data segments of the second partition is dispersed error encoded into the first set of encoded data slices;

a segment size entry indicating a number of bytes of each data segment of the one or more data segments of the second partition; and

a total length entry indicating a number of bytes of all of the data segments of the one or more data segments of the second partition.

14. The computing device of claim 10, wherein the third SAT comprises:

a first data entry region including;

a start segment vault source name entry indicating a vault source name associated with storage of a first set of encoded data slices of the second plurality of sets of encoded data slices, wherein a first data segment of one or more data segments of the second partition is dispersed error encoded into the first set of encoded data slices;

a segment size entry indicating a number of bytes of each data segment of the one or more data segments of the second partition; and

a total length entry indicating a number of bytes of all of the data segments of the one or more data segments of the second partition; and

a second data entry region including;

a start segment vault source name entry indicating a vault source name associated with storage of a first set of encoded data slices of the second plurality of sets of encoded data slices, wherein a first data segment of one or more data segments of the second partition is dispersed error encoded into the first set of encoded data slices;

a segment size entry indicating a number of bytes of each data segment of the one or more data segments of the second partition; and

a total length entry indicating a number of bytes of all of the data segments of the one or more data segments of the second partition.

15. The computing device of claim 10, wherein the processing module is operable to:

dispersed storage error encode the third SAT to produce a third set of SAT slices; and

send the third set of SAT slices to the first set of storage units or the second set of storage units.

16. The computing device of claim 10, wherein the processing module is operable to:

update a directory with a first SAT vault source name, a second SAT vault source name, and a third SAT vault source name.

17. The computing device of claim 10, wherein the processing module is operable to:

further partition the data object into a third partition;

dispersed storage error encode the third partition into a third plurality of sets of encoded data slices;

generate a fourth SAT regarding storage of the third plurality of sets of encoded data slices in a third set of storage units of the DSN;

dispersed storage error encode the fourth SAT to produce a fourth set of SAT slices;

send the third plurality of sets of encoded data slices and the fourth set of SAT slices to the third set of storage units; and

generate a fifth SAT regarding storage of the first, second, and third sets of SAT slices in the first, second, and third set of storage units.

18. The computing device of claim 10, wherein the processing module is operable to:

dispersed storage error encode the first partition into the first plurality of sets of encoded data slices and the second partition into the second plurality of sets of encoded data slices substantially concurrently.