Storing data and metadata in a distributed storage network

• US 9,104,719 B2
• Filed: 04/07/2014
• Issued: 08/11/2015
• Est. Priority Date: 07/06/2011
• Status: Active Grant

• Current Assignee: Pure Storage, Inc.
• Original Assignee: Cleversafe Incorporated (International Business Machines Corporation)
• Inventors: Leggette, Wesley
• Primary Examiner(s): LE, DEBBIE M
• Application Number:
  US14/246,253
• Publication Number:
  US 20140222827A1
• Time in Patent Office: 491 Days
• Field of Search: 707/769, 707/756, 707/706
• US Class Current: 1/1
• CPC Class Codes: G06F 16/24 Querying
• CPC Class Codes: G06F 16/40 of multimedia data, e.g. slideshows comprising image and additional audio data retrieval of still image data G06F16/50; retrieval of audio data G06F16/60; retrieval of video data G06F16/70
• CPC Class Codes: H04L 1/0045 Arrangements at the receiver end
• CPC Class Codes: H04L 65/612 for unicast
• CPC Class Codes: H04L 65/80 Responding to QoS
• CPC Class Codes: H04N 21/23103 using load balancing strategies, e.g. by placing or distributing content on different disks, different memories or different servers storage management G06F3/0604; allocation of resources considering the load in multiprogramming arrangements G06F9/505; techniques for rebalancing the load in a distributed system G06F9/5083; access to distributed or replicated servers, e.g. load balancing, in data networks H04L67/1001
• CPC Class Codes: H04N 21/23109 by placing content in organized collections, e.g. EPG data repository details of retrieval of video data and associated meta data in video databases G06F16/739
• CPC Class Codes: H04N 21/23113 involving housekeeping operations for stored content, e.g. prioritizing content for deletion because of storage space restrictions storage management, e.g. defragmentation G06F3/0604; snloading stored programs G06F9/445; housekeeping operations in file systems, e.g. deletion policies G06F16/10; buffering arrangements in a network node or in an end terminal in packet networks H04L49/90
• CPC Class Codes: H04N 21/2407 Monitoring of transmitted content, e.g. distribution time, number of downloads arrangements for monitoring programmes for broadcast or for distribution combined with broadcast H04H20/14
• CPC Class Codes: H04N 21/262 Content or additional data distribution scheduling, e.g. sending additional data at off-peak times, updating software modules, calculating the carousel transmission frequency, delaying a video stream transmission, generating play-lists scheduling strategies for dispatcher in multiprogramming arrangements G06F9/4881; arrangements for scheduling broadcast services or broadcast-related services H04H60/06; flow control in packet networks H04L47/10; establishing a time schedule or organising the servicing of application requests H04L67/62
• CPC Class Codes: H04N 21/6543 for forcing some client operations, e.g. recording remote booting in general G06F9/4416
• CPC Class Codes: H04N 21/8456 by decomposing the content in the time domain, e.g. in time segments

Abstract

A method begins with receiving a data search criteria. The method continues by accessing a master database that includes a plurality of entries, wherein an entry includes a data name field, a metadata field, a data DSN address, and a metadata DSN address. The method continues by indexing the database based on a comparison of the data search criteria with metadata contained in the metadata field of the entries. The method continues by, when one or more entries of the database have the metadata that substantially matches the data search criteria, utilizing the data DSN address of the one or more entries to retrieve one or more sets of encode data slices. The method continues by decoding the one or more sets of encoded data slices to retrieve one or more data segments corresponding to the data search criteria.

Claims

1. A method for execution by one or more processing units of one or more computing devices within a dispersed storage network (DSN), the method comprises:

receiving, by the one or more processing units of the one or more computing devices, a data search criteria;

accessing a master database that includes a plurality of entries, wherein an entry of the plurality of entries includes a data name field, a metadata field, a data DSN address, and a metadata DSN address;

indexing the database based on a comparison of the data search criteria with metadata contained in the metadata field of the plurality of entries;

when one or more entries of the database have the metadata that substantially matches the data search criteria, utilizing the data DSN address of the one or more entries to retrieve one or more sets of encode data slices; and

decoding the one or more sets of encoded data slices to retrieve one or more data segments corresponding to the data search criteria.

2. The method of claim 1, wherein the accessing the master database comprises:

retrieving one or more sets of encoded database slices from storage units of the DSN; and

decoding the one or more sets of encoded database slices to reconstruct the master database.

3. The method of claim 1, wherein the indexing the database comprises:

when the one or more entries of the database have the metadata that substantially matches the data search criteria;

utilizing the metadata DSN address of the one or more entries to retrieve one or more sets of encoded metadata slices;

decoding the one or more sets of encoded metadata slices to produce one or more reconstructed metadata;

comparing the metadata of the master database with the reconstructed metadata; and

when the metadata of the master database compares favorably with the reconstructed metadata, indicating that the one or more entries of the database have the metadata that substantially matches the data search criteria.

4. The method of claim 3 further comprises:

when the metadata of the master database does not compare favorably with the reconstructed metadata, determining whether the metadata of the master database is out of date or whether the reconstructed metadata is out of date;

when the metadata of the master database is out of date, updating the metadata of the master database based on the reconstructed metadata; and

when the reconstructed metadata is out of data, updating the reconstructed metadata based on the metadata of the master database.

5. The method of claim 1, wherein the indexing the database comprises:

when the one or more entries of the database have the metadata that substantially matches the data search criteria;

utilizing the metadata DSN address of the one or more entries to retrieve one or more sets of encoded metadata slices;

decoding the one or more sets of encoded metadata slices to produce one or more reconstructed metadata and one or more reconstructed data DSN addresses;

comparing the data DSN address of the one or more entries with the one or more reconstructed data DSN addresses; and

when the data DSN address of the one or more entries compares favorably with the one or more reconstructed data DSN addresses, indicating that the one or more entries of the database have the metadata that substantially matches the data search criteria.

6. The method of claim 5 further comprises:

when the data DSN address of the one or more entries does not compare favorably with the one or more reconstructed data DSN addresses, determining whether the data DSN address of the one or more entries is out of date or whether the one or more reconstructed data DSN addresses is out of date;

when the data DSN address of the one or more entries is out of date, updating the master database based on the reconstructed data DSN addresses; and

when the reconstructed data DSN addresses are out of date, updating the one or more sets of encoded metadata slices based on the data DSN address of the one or more entries.

7. The method of claim 1, wherein the decoding the one or more sets of encoded data slices further comprises:

producing one or more reconstructed metadata DSN addresses;

comparing the one or more reconstructed metadata DSN addresses with the metadata DSN addresses of the one or more entries of the master database;

when the one or more reconstructed metadata DSN addresses compares unfavorably with the metadata DSN addresses of the one or more entries of the master database, determining whether the one or more reconstructed metadata DSN addresses is out of date or whether the metadata DSN addresses of the one or more entries of the master database is out of date;

when the one or more reconstructed metadata DSN addresses is out of date, updating the one or more sets of encoded data slices based on the metadata DSN addresses of the one or more entries of the master database; and

when the metadata DSN addresses of the one or more entries of the master database is out of date, updating the one or more entries of the database based on the one or more reconstructed metadata DSN addresses.

8. A computing device comprises:

a first module, when operable within the computing device, causes the computing device to;

receive a data search criteria;

a second module, when operable within the computing device, causes the computing device to;

access a master database that includes a plurality of entries, wherein an entry of the plurality of entries includes a data name field, a metadata field, a data dispersed storage network (DSN) address, and a metadata DSN address;

a third module, when operable within the computing device, causes the computing device to;

index the database based on a comparison of the data search criteria with metadata contained in the metadata field of the plurality of entries;

a fourth module, when operable within the computing device, causes the computing device to;

when one or more entries of the database have the metadata that substantially matches the data search criteria, utilize the data DSN address of the one or more entries to retrieve one or more sets of encode data slices; and

a fifth module, when operable within the computing device, causes the computing device to;

decode the one or more sets of encoded data slices to retrieve one or more data segments corresponding to the data search criteria.

9. The computing device of claim 8, wherein the second module, when operable within the computing device, further causes the computing device to access the master database by:

retrieving one or more sets of encoded database slices from storage units of the DSN; and

decoding the one or more sets of encoded database slices to reconstruct the master database.

10. The computing device of claim 8, wherein the third module, when operable within the computing device, further causes the computing device to index the database by:

when the one or more entries of the database have the metadata that substantially matches the data search criteria;

utilizing the metadata DSN address of the one or more entries to retrieve one or more sets of encoded metadata slices;

decoding the one or more sets of encoded metadata slices to produce one or more reconstructed metadata;

comparing the metadata of the master database with the reconstructed metadata; and

when the metadata of the master database compares favorably with the reconstructed metadata, indicating that the one or more entries of the database have the metadata that substantially matches the data search criteria.

11. The computing device of claim 10, wherein the third module, when operable within the computing device, further causes the computing device to:

when the metadata of the master database does not compare favorably with the reconstructed metadata, determine whether the metadata of the master database is out of date or whether the reconstructed metadata is out of date;

when the metadata of the master database is out of date, update the metadata of the master database based on the reconstructed metadata; and

when the reconstructed metadata is out of data, update the reconstructed metadata based on the metadata of the master database.

12. The computing device of claim 8, wherein the third module, when operable within the computing device, further causes the computing device to index the database by:

when the one or more entries of the database have the metadata that substantially matches the data search criteria;

utilizing the metadata DSN address of the one or more entries to retrieve one or more sets of encoded metadata slices;

decoding the one or more sets of encoded metadata slices to produce one or more reconstructed metadata and one or more reconstructed data DSN addresses;

comparing the data DSN address of the one or more entries with the one or more reconstructed data DSN addresses; and

when the data DSN address of the one or more entries compares favorably with the one or more reconstructed data DSN addresses, indicating that the one or more entries of the database have the metadata that substantially matches the data search criteria.

13. The computing device of claim 12, wherein the third module, when operable within the computing device, further causes the computing device to:

when the data DSN address of the one or more entries does not compare favorably with the one or more reconstructed data DSN addresses, determine whether the data DSN address of the one or more entries is out of date or whether the one or more reconstructed data DSN addresses is out of date;

when the data DSN address of the one or more entries is out of date, update the master database based on the reconstructed data DSN addresses; and

when the reconstructed data DSN addresses are out of date, update the one or more sets of encoded metadata slices based on the data DSN address of the one or more entries.

14. The computing device of claim 12, wherein the fifth module, when operable within the computing device, further causes the computing device to decode the one or more sets of encoded data slices further by:

producing one or more reconstructed metadata DSN addresses;

comparing the one or more reconstructed metadata DSN addresses with the metadata DSN addresses of the one or more entries of the master database;

when the one or more reconstructed metadata DSN addresses compares unfavorably with the metadata DSN addresses of the one or more entries of the master database, determining whether the one or more reconstructed metadata DSN addresses is out of date or whether the metadata DSN addresses of the one or more entries of the master database is out of date;

when the one or more reconstructed metadata DSN addresses is out of date, updating the one or more sets of encoded data slices based on the metadata DSN addresses of the one or more entries of the master database; and

when the metadata DSN addresses of the one or more entries of the master database is out of date, updating the one or more entries of the database based on the one or more reconstructed metadata DSN addresses.