Storing data and metadata in a distributed storage network

• US 8,694,545 B2
• Filed: 06/20/2012
• Issued: 04/08/2014
• Est. Priority Date: 07/06/2011
• Status: Active Grant

• Current Assignee: Pure Storage, Inc.
• Original Assignee: Cleversafe Incorporated (International Business Machines Corporation)
• Inventors: Leggette, Wesley
• Primary Examiner(s): LE, DEBBIE M
• Application Number:
  US13/527,778
• Publication Number:
  US 20130013958A1
• Time in Patent Office: 657 Days
• Field of Search: 707/793, 707/803, 707/812, 711/171, 711/174
• US Class Current: 707/793
• CPC Class Codes: G06F 16/24 Querying
• CPC Class Codes: G06F 16/40 of multimedia data, e.g. slideshows comprising image and additional audio data retrieval of still image data G06F16/50; retrieval of audio data G06F16/60; retrieval of video data G06F16/70
• CPC Class Codes: H04L 1/0045 Arrangements at the receiver end
• CPC Class Codes: H04L 65/612 for unicast
• CPC Class Codes: H04L 65/80 Responding to QoS
• CPC Class Codes: H04N 21/23103 using load balancing strategies, e.g. by placing or distributing content on different disks, different memories or different servers storage management G06F3/0604; allocation of resources considering the load in multiprogramming arrangements G06F9/505; techniques for rebalancing the load in a distributed system G06F9/5083; access to distributed or replicated servers, e.g. load balancing, in data networks H04L67/1001
• CPC Class Codes: H04N 21/23109 by placing content in organized collections, e.g. EPG data repository details of retrieval of video data and associated meta data in video databases G06F16/739
• CPC Class Codes: H04N 21/23113 involving housekeeping operations for stored content, e.g. prioritizing content for deletion because of storage space restrictions storage management, e.g. defragmentation G06F3/0604; snloading stored programs G06F9/445; housekeeping operations in file systems, e.g. deletion policies G06F16/10; buffering arrangements in a network node or in an end terminal in packet networks H04L49/90
• CPC Class Codes: H04N 21/2407 Monitoring of transmitted content, e.g. distribution time, number of downloads arrangements for monitoring programmes for broadcast or for distribution combined with broadcast H04H20/14
• CPC Class Codes: H04N 21/262 Content or additional data distribution scheduling, e.g. sending additional data at off-peak times, updating software modules, calculating the carousel transmission frequency, delaying a video stream transmission, generating play-lists scheduling strategies for dispatcher in multiprogramming arrangements G06F9/4881; arrangements for scheduling broadcast services or broadcast-related services H04H60/06; flow control in packet networks H04L47/10; establishing a time schedule or organising the servicing of application requests H04L67/62
• CPC Class Codes: H04N 21/6543 for forcing some client operations, e.g. recording remote booting in general G06F9/4416
• CPC Class Codes: H04N 21/8456 by decomposing the content in the time domain, e.g. in time segments

Abstract

A method begins by a dispersed storage (DS) processing module generating a set of encoded data slices based on a data segment of data and dispersed storage network (DSN) addressing information regarding metadata of the data. The method continues with the DS processing module generating a set of encoded metadata slices based on the metadata and DSN addressing information regarding the data. The method continues with the DS processing module generating a set of metadata write commands regarding storing the set of encoded metadata in a first set of DS units of DSN memory. The method continues with the DS processing module generating a set of data segment write commands regarding storing the set of encoded data slices in a second set of DS units of the DSN memory.

Claims

1. A method for execution by one or more processing units of one or more computing devices, the method comprises:

generating, by the one or more processing units of the one or more computing devices, a set of encoded data slices based on a data segment of data and dispersed storage network (DSN) addressing information regarding metadata of the data;

generating a set of encoded metadata slices based on the metadata and DSN addressing information regarding the data;

generating a set of metadata write commands regarding storing the set of encoded metadata slices in a first set of dispersed storage (DS) units of DSN memory; and

generating a set of data segment write commands regarding storing the set of encoded data slices in a second set of DS units of the DSN memory.

2. The method of claim 1 further comprises:

the DSN addressing information regarding the metadata including a metadata source name; and

the DSN addressing information regarding the data including a data source name.

3. The method of claim 1, wherein the generating the set of encoded data slices comprises:

combining the DSN addressing information regarding metadata with the data segment to produce an amended data segment; and

dispersed storage error encoding the amended data segment to produce the set of encoded data slices.

4. The method of claim 1, wherein the generating the set of encoded data slices comprises:

dispersed storage error encoding the data segment to produce a preliminary set of encoded data slices; and

appending the DSN addressing information regarding the metadata with at least some of the encoded data slices of the preliminary set of encoded data slices to produce the set of encoded data slices.

5. The method of claim 1, wherein the generating the set of encoded metadata slices comprises:

combining the DSN addressing information regarding the data with the metadata to produce amended metadata; and

dispersed storage error encoding the amended metadata to produce the set of encoded metadata slices.

6. The method of claim 1, wherein the generating the set of encoded metadata slices comprises:

dispersed storage error encoding the metadata to produce a preliminary set of encoded metadata slices; and

appending the DSN addressing information regarding the data with at least some of the encoded metadata slices of the preliminary set of encoded metadata slices to produce the set of encoded metadata slices.

7. The method of claim 1 further comprises:

generating a database entry to include one or more of a data name of the data, the metadata, the DSN addressing information regarding the metadata, and DSN addressing information regarding the data; and

updating a database to include the database entry.

8. A method for execution by one or more processing units of one or more computing devices, the method comprises:

combining, by the one or more processing units of the one or more computing devices, dispersed storage network (DSN) addressing information regarding metadata with a data segment of data to produce an amended data segment, wherein the metadata is regarding the data;

combining DSN addressing information regarding the data with the metadata to produce amended metadata;

dispersed storage error encoding the amended data segment to produce a set of encoded data slices;

dispersed storage error encoding the amended metadata to produce a set of encoded metadata slices; and

outputting the set of encoded data slices and the set of encoded metadata slices for storage in a dispersed storage network.

9. The method of claim 8 further comprises:

the DSN addressing information regarding the metadata including a metadata source name; and

the DSN addressing information regarding the data including a data source name.

10. The method of claim 8 further comprises:

generating a database entry to include one or more of a data name of the data, the metadata, the DSN addressing information regarding the metadata, and DSN addressing information regarding the data; and

updating a database to include the database entry.

11. A computing device comprises:

a first module, when operable within the computing device, causes the computing device to;

generate a set of encoded data slices based on a data segment of data and dispersed storage network (DSN) addressing information regarding metadata of the data;

a second module, when operable within the computing device, causes the computing device to;

generate a set of encoded metadata slices based on the metadata and DSN addressing information regarding the data;

a third module, when operable within the computing device, causes the computing device to;

generate a set of metadata write commands regarding storing the set of encoded metadata slices in a first set of dispersed storage (DS) units of DSN memory; and

a fourth module, when operable within the computing device, causes the computing device to;

generate a set of data segment write commands regarding storing the set of encoded data slices in a second set of DS units of the DSN memory.

12. The computing device of claim 11 further comprises:

the DSN addressing information regarding the metadata includes a metadata source name; and

the DSN addressing information regarding the data includes a data source name.

13. The computing device of claim 11, wherein the first module is operable to generate the set of encoded data slices by:

combining the DSN addressing information regarding metadata with the data segment to produce an amended data segment; and

dispersed storage error encoding the amended data segment to produce the set of encoded data slices.

14. The computing device of claim 11, wherein the first module is operable to generate the set of encoded data slices by:

dispersed storage error encoding the data segment to produce a preliminary set of encoded data slices; and

appending the DSN addressing information regarding the metadata with at least some of the encoded data slices of the preliminary set of encoded data slices to produce the set of encoded data slices.

15. The computing device of claim 11, wherein the second module is operable to generate the set of encoded metadata slices by:

combining the DSN addressing information regarding the data with the metadata to produce amended metadata; and

dispersed storage error encoding the amended metadata to produce the set of encoded metadata slices.

16. The computing device of claim 11, wherein the second module is operable to generate the set of encoded metadata slices by:

dispersed storage error encoding the metadata to produce a preliminary set of encoded metadata slices; and

appending the DSN addressing information regarding the data with at least some of the encoded metadata slices of the preliminary set of encoded metadata slices to produce the set of encoded metadata slices.

17. The computing device of claim 11 further comprises:

a fifth module, when operable within the computing device, causes the computing device to;

generate a database entry to include one or more of a data name of the data, the metadata, the DSN addressing information regarding the metadata, and DSN addressing information regarding the data; and

update a database to include the database entry.

18. A computing device comprises:

a first module, when operable within the computing device, causes the computing device to;

combine dispersed storage network (DSN) addressing information regarding metadata with a data segment of data to produce an amended data segment, wherein the metadata is regarding the data; and

combine DSN addressing information regarding the data with the metadata to produce amended metadata;

a second module, when operable within the computing device, causes the computing device to;

dispersed storage error encode the amended data segment to produce a set of encoded data slices; and

dispersed storage error encode the amended metadata to produce a set of encoded metadata slices; and

a third module, when operable within the computing device, causes the computing device to;

output the set of encoded data slices and the set of encoded metadata slices for storage in a dispersed storage network.

19. The computing device of claim 18 further comprises:

the DSN addressing information regarding the metadata including a metadata source name; and

the DSN addressing information regarding the data including a data source name.

20. The computing device of claim 18 further comprises:

a fourth module, when operable within the computing device, causes the computing device to;

generate a database entry to include one or more of a data name of the data, the metadata, the DSN addressing information regarding the metadata, and DSN addressing information regarding the data; and

update a database to include the database entry.