Accessing a large data object in a dispersed storage network

• US 9,208,009 B2
• Filed: 12/09/2013
• Issued: 12/08/2015
• Est. Priority Date: 11/01/2011
• Status: Active Grant

• Current Assignee: Pure Storage, Inc.
• Original Assignee: Cleversafe Incorporated (International Business Machines Corporation)
• Inventors: Resch, Jason K., Leggette, Wesley
• Primary Examiner(s): Abraham, Esaw
• Assistant Examiner(s): ALSHACK, OSMAN M
• Application Number:
  US14/100,368
• Publication Number:
  US 20140095650A1
• Time in Patent Office: 729 Days
• Field of Search: 714/776, 714/770, 714/763, 709/213
• US Class Current: 1/1
• CPC Class Codes: G06F 11/073 in a memory management context, e.g. virtual memory or cache management memory management G06F12/00; testing of static memory units G11C29/00
• CPC Class Codes: G06F 11/1076 Parity data used in redundant arrays of independent storages, e.g. in RAID systems
• CPC Class Codes: G06F 21/62 Protecting access to data via a platform, e.g. using keys or access control rules
• CPC Class Codes: G06F 2211/1028 Distributed, i.e. distributed RAID systems with parity
• CPC Class Codes: H04L 1/0057 Block codes H04L1/0061, H04L1/0064 take precedence
• CPC Class Codes: H04L 1/0061 Error detection codes
• CPC Class Codes: H04L 1/0072 Error control for data other than payload data, e.g. control data
• CPC Class Codes: H04L 1/0083 Formatting with frames or packets; Protocol or part of protocol for error control
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

A method begins by a dispersed storage (DS) processing module generating a data object identifier for data to be stored in a dispersed storage network (DSN) and partitioning the data into a plurality of data partitions based on a set of retrieval preferences and data boundary information. For a data partition, the method continues with the DS processing module dispersed storage error encoding the data partition to produce a plurality of sets of encoded data slices and generating a plurality of sets of DSN addresses for the plurality of sets of encoded data slices, wherein a DSN address of the plurality of sets of DSN addresses includes a representation of the data object identifier, a representation of one or more retrieval preferences of the set of retrieval preferences, a representation of a corresponding portion of the data boundary information, and dispersed storage addressing information.

Claims

1. A method for execution by a computing device, the method comprises:

obtaining a data object identifier regarding a data object to identify the data object;

obtaining a plurality of boundary indicators based on the data object and one or more retrieval preferences, wherein the one or more retrieval preferences are utilized to set partition boundaries associated with partitioning the data object into a plurality of data partitions, and wherein the plurality of boundary indicators indicate respective partition boundaries, in which respective data partitions are further segmented into a plurality of data segments and the data segments are dispersed storage error encoded into one or more sets of encoded data slices; and

generating sets of dispersed storage network (DSN) addresses corresponding to respective data partitions of the plurality of data partitions based on the plurality of boundary indicators and the data object identifier, such that a first DSN address set of the sets of DSN addresses has a first common identifier for a first data partition, a common indicator corresponding to retrieval preferences for the first data partition, and a first unique set of encoded data slice identifiers associated with encoded data slices of the first data partition and a second DSN address set of the sets of DSN addresses has a second common identifier for a second data partition, a common indicator corresponding to retrieval preferences for the second data partition, and a second unique set of encoded data slice identifiers associated with encoded data slices of the second data partition.

2. The method of claim 1 further comprises:

generating one or more write requests incorporating one or more sets of the DSN addresses.

3. The method of claim 1 further comprises:

generating one or more read requests incorporating one or more sets of the DSN addresses.

4. The method of claim 1, wherein the obtaining the data object identifier comprises:

receiving a data name and a storage vault identifier of the data object;

generating the data object identifier to include the vault identifier and an object number, wherein the object number is a random number or is a result of a deterministic function performed on the data name; and

generating associating information, which associates the data name and the data object identifier.

5. The method of claim 1 further comprises:

the first common identifier for the first data partition including a representation of the first data partition, one or more retrieval preferences, and a corresponding boundary indicator for the first data partition;

the common indicator corresponding to the retrieval preferences of the first data partition including a vault identifier; and

the first unique set of encoded data slice identifiers including one or more of a slice index and a segment identifier.

6. The method of claim 1, wherein the one or more retrieval preferences comprise one or more of:

a keyword, a data type, a previous retrieval preference, a data transition indicator, a pattern, a data value, and a data size.

7. A dispersed storage processing module comprises:

directory accessor of a computing device configured to obtain a data object identifier regarding a data object;

boundary generator of the computing device configured to obtain a plurality of boundary indicators based on the data object and one or more retrieval preferences, wherein the one or more retrieval preferences are utilized to set partition boundaries associated with partitioning the data object into a plurality of data partitions, and wherein the plurality of boundary indicators indicate respective partition boundaries, in which respective data partitions are further segmented into a plurality of data segments and the data segments are dispersed storage error encoded into one or more sets of encoded data slices; and

one or more address generators of the computing device configured to generate sets of dispersed storage network (DSN) addresses corresponding to respective data partitions of the plurality of data partitions based on the plurality of boundary indicators and the data object identifier, such that a first DSN address set of the sets of DSN addresses has a first common identifier for a first data partition, a common indicator corresponding to retrieval preferences for the first data partition, and a first unique set of encoded data slice identifiers associated with encoded data slices of the first data partition and a second DSN address set of the sets of DSN addresses has a second common identifier for a second data partition, a common indicator corresponding to retrieval preferences for the second data partition, and a second unique set of encoded data slice identifiers associated with encoded data slices of the second data partition.

8. The dispersed storage processing module of claim 7 further comprises:

write request generator of the computing device configured to generate one or more write requests incorporating one or more sets of the DSN addresses.

9. The dispersed storage processing module of claim 7 further comprises:

read request generator of the computing device configured to generate one or more read requests incorporating one or more sets of the DSN addresses.

10. The dispersed storage processing module of claim 7, wherein the directory accessor causes the computing device to:

receive a data name and a storage vault identifier of the data object;

generate the data object identifier to include the vault identifier and an object number, wherein the object number is a random number or is a result of a deterministic function performed on the data name; and

generate associating information, which associates the data name and the data object identifier.

11. The dispersed storage processing module of claim 7 further comprises:

the first common identifier for the first data partition including a representation of the first data partition, one or more retrieval preferences, and a corresponding boundary indicator for the first data partition;

the common indicator corresponding to the retrieval preferences of the first data partition including a vault identifier; and

the first unique set of encoded data slice identifiers including one or more of a slice index and a segment identifier.

12. The dispersed storage processing module of claim 7, wherein the one or more retrieval preferences comprise one or more of:

a keyword, a data type, a previous retrieval preference, a data transition indicator, a pattern, a data value, and a data size.