Accessing a large data object in a dispersed storage network

• US 8,607,122 B2
• Filed: 09/12/2012
• Issued: 12/10/2013
• Est. Priority Date: 11/01/2011
• Status: Active Grant

• Current Assignee: Pure Storage, Inc.
• Original Assignee: Cleversafe Incorporated (International Business Machines Corporation)
• Inventors: Resch, Jason K., Leggette, Wesley
• Primary Examiner(s): Abraham, Esaw
• Assistant Examiner(s): ALSHACK, OSMAN M
• Application Number:
  US13/611,866
• Publication Number:
  US 20130111300A1
• Time in Patent Office: 454 Days
• Field of Search: 714/763, 714/770
• US Class Current: 714/763
• CPC Class Codes: G06F 11/073 in a memory management context, e.g. virtual memory or cache management memory management G06F12/00; testing of static memory units G11C29/00
• CPC Class Codes: G06F 11/1076 Parity data used in redundant arrays of independent storages, e.g. in RAID systems
• CPC Class Codes: G06F 21/62 Protecting access to data via a platform, e.g. using keys or access control rules
• CPC Class Codes: G06F 2211/1028 Distributed, i.e. distributed RAID systems with parity
• CPC Class Codes: H04L 1/0057 Block codes H04L1/0061, H04L1/0064 take precedence
• CPC Class Codes: H04L 1/0061 Error detection codes
• CPC Class Codes: H04L 1/0072 Error control for data other than payload data, e.g. control data
• CPC Class Codes: H04L 1/0083 Formatting with frames or packets; Protocol or part of protocol for error control
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

A method begins by a dispersed storage (DS) processing module generating a data object identifier for data to be stored in a dispersed storage network (DSN) and partitioning the data into a plurality of data partitions based on a set of retrieval preferences and data boundary information. For a data partition, the method continues with the DS processing module dispersed storage error encoding the data partition to produce a plurality of sets of encoded data slices and generating a plurality of sets of DSN addresses for the plurality of sets of encoded data slices, wherein a DSN address of the plurality of sets of DSN addresses includes a representation of the data object identifier, a representation of one or more retrieval preferences of the set of retrieval preferences, a representation of a corresponding portion of the data boundary information, and dispersed storage addressing information.

Claims

1. A method comprises:

generating a data object identifier for data to be stored in a dispersed storage network (DSN);

generating data boundary information based on the data and one or more retrieval preferences, wherein the one or more retrieval preferences facilitates searching for the data, or a partition thereof, once the data is stored in the DSN and wherein the one or more retrieval preferences includes information regarding the data, or the partition thereof;

partitioning the data into a plurality of data partitions based on the data boundary information;

for a data partition of the plurality of data partitions;

segmenting the data partition into a plurality of data segments;

dispersed storage error encoding the plurality of data segments to produce a plurality of sets of encoded data slices;

generating a plurality of sets of DSN addresses for the plurality of sets of encoded data slices, wherein a DSN address of the plurality of sets of DSN addresses includes;

a representation of the data object identifier;

a representation of the one or more retrieval preferences;

a representation of a corresponding portion of the data boundary information; and

dispersed storage addressing information, wherein the data partition is retrievable from storage by;

obtaining the data object identifier via a file directory access; and

calculating the plurality of sets of DSN addresses based on the data object identifier and the one or more retrieval preferences.

2. The method of claim 1 further comprises:

analyzing the data in accordance with the data boundary information to identify data elements of the data that correspond to a retrieval preference of the one or more retrieval preferences to determine the data partition, wherein the retrieval preference includes one or more of a keyword, a data type, a previous retrieval preference, a data transition indicator, a pattern, a data value, and a data size.

3. The method of claim 1 further comprises:

inserting the data boundary information into the data.

4. The method of claim 1, wherein the representation of the data object identifier comprises one of:

a DSN base source name that is generated from the data object identifier;

a resultant of a deterministic function being performed on the data object identifier; and

the data object identifier.

5. The method of claim 1, wherein the representation of one or more retrieval preferences comprises one of:

a code value representing a retrieval preference of the one or more retrieval preferences;

a resultant of a deterministic function being performed on the one or more retrieval preferences; and

the one or more retrieval preferences.

6. The method of claim 1, wherein the representation of a corresponding portion of the data boundary information comprises one of:

a code value representing the corresponding portion of the data boundary information;

a resultant of a deterministic function being performed on the corresponding portion of the data boundary information; and

the corresponding portion of the data boundary information.

7. The method of claim 1, wherein the dispersed storage addressing information comprises:

a storage node identifier;

a data segment number corresponding to a data segment of the data portion; and

a vault identifier that identifies user device information associated with the data.

8. A method comprises:

receiving a data object identifier via a file directory access, wherein the data object identifier identifies data stored in a dispersed storage network (DSN);

receiving a retrieval preference to retrieve a data portion of the data, wherein the retrieval preference facilitates searching for the data portion and wherein the retrieval preferences include information regarding the data portion;

determining corresponding data boundary information based on the retrieval preference;

calculating a plurality of sets of DSN addresses based on the data object identifier, the retrieval preference, and the corresponding data boundary information;

retrieving a plurality of sets of encoded data slices from the DSN based on the plurality of sets of DSN addresses;

decoding the plurality of sets of encoded data slices to produce a plurality of sets of data segments, and

reconstructing the data portion from the plurality of data segments.

9. The method of claim 8, wherein a DSN address of the plurality of sets of encoded DSN addresses comprises:

a representation of the data object identifier;

a representation of the retrieval preference;

a representation of the corresponding data boundary information; and

dispersed storage addressing information.

10. The method of claim 9, wherein the representation of the data object identifier comprises one of:

a DSN base source name that is generated from the data object identifier;

a resultant of a deterministic function being performed on the data object identifier; and

the data object identifier.

11. The method of claim 9, wherein the representation of one or more retrieval preferences comprises one of:

a code value representing a retrieval preference of the retrieval preference;

a resultant of a deterministic function being performed on the retrieval preference; and

the retrieval preference.

12. The method of claim 9, wherein the representation of the corresponding data boundary information comprises one of:

a code value representing the corresponding data boundary information;

a resultant of a deterministic function being performed on the corresponding the data boundary information; and

the corresponding data boundary information.

13. The method of claim 9, wherein the dispersed storage addressing information comprises:

a storage node identifier;

a data segment number corresponding to a data segment of the data portion;

a vault identifier that identifier user device information associated with the data.

14. The method of claim 8, wherein the receiving the retrieval preference comprises:

receiving a requested data preference; and

equating the requested data preference to one or more preferences of a set of retrieval preferences to produce the retrieval preference.

15. A dispersed storage (DS) module comprises:

a first module, when operable within a computing device, causes the computing device to;

generate a data object identifier for data to be stored in a dispersed storage network (DSN);

a second module, when operable within the computing device, causes the computing device to;

generate data boundary information based on the data and one or more retrieval preferences, wherein the one or more retrieval preferences facilitates searching for the data, or a partition thereof, once the data is stored in the DSN and wherein the one or more retrieval preferences includes information regarding the data, or the partition thereof; and

partition the data into a plurality of data partitions based on the data boundary information;

a third module, when operable within the computing device, causes the computing device to;

for a data partition of the plurality of data partitions;

segment the data partition into a plurality of data segments;

dispersed storage error encode the plurality of data segments to produce a plurality of sets of encoded data slices; and

a fourth module, when operable within the computing device, causes the computing device to;

for a data partition of the plurality of data partitions, generate a plurality of sets of DSN addresses for the plurality of sets of encoded data slices, wherein a DSN address of the plurality of sets of DSN addresses includes;

a representation of the data object identifier;

a representation of the one or more retrieval preferences;

a representation of a corresponding portion of the data boundary information; and

dispersed storage addressing information, wherein the data partition is retrievable from storage by;

obtaining the data object identifier via a file directory access; and

calculating the plurality of sets of DSN addresses based on the data object identifier and the one or more retrieval preferences.

16. The DS module of claim 15 further comprises:

the second module further functions to analyze the data in accordance with the data boundary information to identify data elements of the data that correspond to a retrieval preference of the one or more retrieval preferences to determine the data partition, wherein the retrieval preference includes one or more of a keyword, a data type, a previous retrieval preference, a data transition indicator, a pattern, a data value, and a data size.

17. The DS module of claim 15 further comprises

the second module further functions to insert the data boundary information into the data.

18. The DS module of claim 15, wherein the representation of the data object identifier comprises one of:

a DSN base source name that is generated from the data object identifier;

a resultant of a deterministic function being performed on the data object identifier; and

the data object identifier.

19. The DS module of claim 15, wherein the representation of one or more retrieval preferences comprises one of:

a code value representing a retrieval preference of the one or more retrieval preferences;

a resultant of a deterministic function being performed on the one or more retrieval preferences; and

the one or more retrieval preferences.

20. The DS module of claim 15, wherein the representation of a corresponding portion of the data boundary information comprises one of:

a code value representing the corresponding portion of the data boundary information;

a resultant of a deterministic function being performed on the corresponding portion of the data boundary information; and

the corresponding portion of the data boundary information.

21. The DS module of claim 15, wherein the dispersed storage addressing information comprises:

a storage node identifier;

a data segment number corresponding to a data segment of the data portion; and

a vault identifier that identifies user device information associated with the data.

22. A dispersed storage (DS) module comprises:

a first module, when operable within a computing device, causes the computing device to;

receive a data object identifier via a file directory access, wherein the data object identifier identifies data stored in a dispersed storage network (DSN); and

receive a retrieval preference to retrieve a data portion of the data, wherein the retrieval preference facilitates searching for the data portion and wherein the retrieval preferences include information regarding the data portion;

a second module, when operable within the computing device, causes the computing device to;

determine corresponding data boundary information based on the retrieval preference;

a third module, when operable within the computing device, causes the computing device to;

calculate a plurality of sets of DSN addresses based on the data object identifier, the retrieval preference, and the corresponding data boundary information; and

a fourth module, when operable within the computing device, causes the computing device to;

retrieve a plurality of sets of encoded data slices from the DSN based on the plurality of sets of DSN addresses;

decode the plurality of sets of encoded data slices to produce a plurality of sets of data segments, and

reconstruct the data portion from the plurality of data segments.

23. The DS module of claim 22, wherein a DSN address of the plurality of sets of encoded DSN addresses comprises:

a representation of the data object identifier;

a representation of the retrieval preference;

a representation of the corresponding data boundary information; and

dispersed storage addressing information.

24. The DS module of claim 23, wherein the representation of the data object identifier comprises one of:

a DSN base source name that is generated from the data object identifier;

a resultant of a deterministic function being performed on the data object identifier; and

the data object identifier.

25. The DS module of claim 23, wherein the representation of one or more retrieval preferences comprises one of:

a code value representing a retrieval preference of the retrieval preference;

a resultant of a deterministic function being performed on the retrieval preference; and

the retrieval preference.

26. The DS module of claim 23, wherein the representation of the corresponding data boundary information comprises one of:

a code value representing the corresponding data boundary information;

a resultant of a deterministic function being performed on the corresponding the data boundary information; and

the corresponding data boundary information.

27. The DS module of claim 23, wherein the dispersed storage addressing information comprises:

a storage node identifier;

a data segment number corresponding to a data segment of the data portion;

a vault identifier that identifier user device information associated with the data.

28. The DS module of claim 22, wherein the first module functions to receive the retrieval preference by:

receiving a requested data preference; and

equating the requested data preference to one or more preferences of a set of retrieval preferences to produce the retrieval preference.