Data transmission utilizing data processing and dispersed storage error encoding

• US 8,918,693 B2
• Filed: 10/03/2011
• Issued: 12/23/2014
• Est. Priority Date: 10/06/2010
• Status: Active Grant

• Current Assignee: Pure Storage, Inc.
• Original Assignee: Cleversafe Incorporated (International Business Machines Corporation)
• Inventors: Dhuse, Greg, Baptist, Andrew, Volvovski, Ilya, Grube, Gary W., Markison, Timothy W., Gladwin, S. Christopher, Resch, Jason K.
• Primary Examiner(s): Decady, Albert
• Assistant Examiner(s): ALSHACK, OSMAN M
• Application Number:
  US13/251,618
• Publication Number:
  US 20120089887A1
• Time in Patent Office: 1,177 Days
• Field of Search: 714/752, 714/770, 714/774, 714/786, 714/746
• US Class Current: 714/752
• CPC Class Codes: G06F 11/1028 Adjacent errors, e.g. error in n-bit (n>1) wide storage units, i.e. package error
• CPC Class Codes: G06F 11/1076 Parity data used in redundant arrays of independent storages, e.g. in RAID systems
• CPC Class Codes: G06F 16/182 Distributed file systems
• CPC Class Codes: H03M 13/05 using block codes, i.e. a predetermined number of check bits joined to a predetermined number of information bits H03M13/2906 takes precedence
• CPC Class Codes: H03M 13/1515 Reed-Solomon codes
• CPC Class Codes: H03M 13/29 combining two or more codes or code structures, e.g. product codes, generalised product codes, concatenated codes, inner and outer codes
• CPC Class Codes: H03M 13/373 with erasure correction and erasure determination, e.g. for packet loss recovery or setting of erasures for the decoding of Reed-Solomon codes
• CPC Class Codes: H03M 13/3761 using code combining, i.e. using combining of codeword portions which may have been transmitted separately, e.g. Digital Fountain codes, Raptor codes or Luby Transform [LT] codes
• CPC Class Codes: H04L 45/24 Multipath

Abstract

A method begins by a processing module dispersed storage error encoding fundamental component data of data in accordance with dispersed storage error coding parameters to produce a plurality of sets of encoded data slices, wherein the data includes the fundamental component data and enhancement component data. The method continues with the processing module transmitting a set of the plurality of sets of encoded data slices and transmitting a corresponding portion of the enhancement component data substantially concurrently with the transmitting of the set of the plurality of sets of encoded data slices.

Claims

1. A method comprises:

partitioning data into a first portion of fundamental component data and a second portion of enhancement component data, wherein the fundamental component data of the first portion includes content of the data of a higher priority than the enhancement component data of the second portion;

dispersed storage error encoding the fundamental component data in accordance with dispersed storage error coding parameters to produce encoded data, segmenting the encoded data into a plurality of encoded data segments and slicing the plurality of encoded data segments into a plurality of sets of encoded data slices, wherein a set of the plurality of sets of encoded data slices corresponds to a data segment of the fundamental component data and wherein a threshold number of encoded data slices of the set of encoded data slices are required to reproduce the data segment;

transmitting a set of the plurality of sets of encoded data slices to a receiving entity; and

transmitting a corresponding portion of the enhancement component data substantially concurrently with the transmitting of the set of the plurality of sets of encoded data slices to the receiving entity.

2. The method of claim 1, wherein the receiving entity comprises one of:

a receiving dispersed storage (DS) processing unit; and

storage units of dispersed storage network (DSN) memory.

3. The method of claim 1 further comprises:

partitioning the data in accordance with a partitioning scheme that includes one or more of;

a data component priority segregation function;

a spectral analysis function; and

a data filter function.

4. The method of claim 1 further comprises:

the fundamental component data including first level of fundamental component data and second level of fundamental component data;

the dispersed storage error coding parameters including first dispersed storage error coding parameters and second dispersed storage error coding parameters;

the plurality of sets of encoded data slices including a first plurality of sets of encoded data slices and a second plurality of sets of encoded data slices;

dispersed storage error encoding the first fundamental component data in accordance with the first dispersed storage error coding parameters to produce the first plurality of sets of encoded data slices; and

dispersed storage error encoding the second fundamental component data in accordance with the second dispersed storage error coding parameters to produce the second plurality of sets of encoded data slices.

5. The method of claim 1 further comprises:

the enhancement component data including a first level of enhancement component data and a second level of enhancement component data.

6. The method of claim 1 further comprises:

determining the dispersed storage error coding parameters based on at least one of metadata associated with the data and metadata associated with the fundamental component data.

7. The method of claim 1 further comprises:

dispersed storage error encoding the enhancement component data in accordance with second dispersed storage error coding parameters to produce a second plurality of sets of encoded data slices, wherein the second dispersed storage error coding parameters have a lower reception reliability than the first dispersed storage error coding parameters; and

transmitting, as the corresponding portion of the enhancement component data, a set of the second set of encoded data slices.

8. The method of claim 1 further comprises:

the data including video data; and

processing the video data in accordance with a video processing function to produce the fundamental component data and the enhancement component data.

9. A method comprises:

receiving sets of encoded data slices, wherein data is partitioned into first portion of fundamental component data and a second portion of enhancement component data, wherein the fundamental component data of the first portion includes content of the data of a higher priority than the enhancement component data of the second portion, wherein a set of the plurality of sets of encoded data slices corresponds to a data segment of the fundamental component data, wherein a threshold number of encoded data slices of the set of encoded data slices are required to reproduce the data segment, wherein the fundamental component data was encoded in accordance with dispersed storage error coding parameters to produce encoded data, the encoded data was segmented into a plurality of data segments and the plurality of data segments were sliced into the sets of encoded data slices, wherein the data includes the fundamental component data and enhancement component data;

receiving corresponding portions of the enhancement component data substantially concurrently with the receiving of the sets of encoded data slices;

dispersed storage error decoding the sets of encoded data slices to reproduce the fundamental component data; and

processing the fundamental component data and the enhancement component data in accordance with a processing function to reproduce the data.

10. The method of claim 9, wherein the processing function comprises one or more of:

a data component priority segregation function;

a spectral analysis function; and

a data filter function.

11. The method of claim 9 further comprises:

the fundamental component data including first level of fundamental component data and second level of fundamental component data;

the sets of encoded data slices including a first plurality of sets of encoded data slices and a second plurality of sets of encoded data slices;

dispersed storage error decoding the first plurality of sets of encoded data slices in accordance with first dispersed storage error coding parameters to reproduce the first level of fundamental component data; and

dispersed storage error decoding the second plurality of sets of encoded data slices in accordance with second dispersed storage error coding parameters to reproduce the second level of fundamental component data.

12. The method of claim 9 further comprises:

receiving, as the enhancement component data, a second plurality of sets of encoded data slices; and

dispersed storage error decoding the enhancement component data in accordance with dispersed storage error coding parameters to reproduce the enhancement component data, wherein the second dispersed storage error coding parameters have a lower reception reliability than the first dispersed storage error coding parameters.

13. The method of claim 9 further comprises:

the data including video data; and

processing the fundamental component data and the enhancement component data in accordance with a video processing function to reproduce the video data.

14. A computer comprises:

an interface;

a memory; and

a processing module operable to;

partition data into a first portion of fundamental component data and a second portion of enhancement component data, wherein the fundamental component data of the first portion includes content of the data of a higher priority than the enhancement component data of the second portion;

dispersed storage error encode the fundamental component data in accordance with dispersed storage error coding parameters to produce encoded data, segment the encoded data into a plurality of encoded data segments and slice the plurality of encoded data segments into a plurality of sets of encoded data slices, wherein a set of the plurality of sets of encoded data slices corresponds to a data segment of the fundamental component data and wherein a threshold number of encoded data slices of the set of encoded data slices are required to reproduce the data segment;

transmit, via the interface, a set of the plurality of sets of encoded data slices to a receiving entity; and

transmit, via the interface, a corresponding portion of the enhancement component data substantially concurrently with the transmitting of the set of the plurality of sets of encoded data slices to the receiving entity.

15. The computer of claim 14, wherein the receiving entity comprises one of:

a receiving dispersed storage (DS) processing unit; and

storage units of dispersed storage network (DSN) memory.

16. The computer of claim 14, wherein the processing module further functions to:

partition the data in accordance with a partitioning scheme that includes one or more of;

a data component priority segregation function;

a spectral analysis function; and

a data filter function.

17. The computer of claim 14 further comprises:

the fundamental component data including first level of fundamental component data and second level of fundamental component data;

the dispersed storage error coding parameters including first dispersed storage error coding parameters and second dispersed storage error coding parameters;

the plurality of sets of encoded data slices including a first plurality of sets of encoded data slices and a second plurality of sets of encoded data slices; and

the processing module further functions to;

dispersed storage error encode the first fundamental component data in accordance with the first dispersed storage error coding parameters to produce the first plurality of sets of encoded data slices; and

dispersed storage error encode the second fundamental component data in accordance with the second dispersed storage error coding parameters to produce the second plurality of sets of encoded data slices.

18. The computer of claim 14 further comprises:

the enhancement component data including a first level of enhancement component data and a second level of enhancement component data.

19. The computer of claim 14, wherein the processing module further functions to:

determine the dispersed storage error coding parameters based on at least one of metadata associated with the data and metadata associated with the fundamental component data.

20. The computer of claim 14, wherein the processing module further functions to:

dispersed storage error encode the enhancement component data in accordance with second dispersed storage error coding parameters to produce a second plurality of sets of encoded data slices, wherein the second dispersed storage error coding parameters have a lower reception reliability than the first dispersed storage error coding parameters; and

transmit, via the interface, as the corresponding portion of the enhancement component data, a set of the second set of encoded data slices.

21. The computer of claim 14 further comprises:

the data including video data; and

the processing module further functions to;

process the video data in accordance with a video processing function to produce the fundamental component data and the enhancement component data.

22. A computer comprises:

an interface;

a memory; and

a processing module operable to;

receive, via the interface, sets of encoded data slices, wherein data is partitioned into first portion of fundamental component data and a second portion of enhancement component data, wherein the fundamental component data of the first portion includes content of the data of a higher priority than the enhancement component data of the second portion, wherein a set of the plurality of sets of encoded data slices corresponds to a data segment of the fundamental component data, wherein a threshold number of encoded data slices of the set of encoded data slices are required to reproduce the data segment, wherein the fundamental component data was encoded in accordance with dispersed storage error coding parameters to produce encoded data, the encoded data was segmented into a plurality of data segments and the plurality of data segments were sliced into the sets of encoded data slices, wherein the data includes the fundamental component data and enhancement component data;

receive, via the interface, corresponding portions of the enhancement component data substantially concurrently with the receiving of the sets of encoded data slices;

dispersed storage error decode the sets of encoded data slices to reproduce the fundamental component data; and

process the fundamental component data and the enhancement component data in accordance with a processing function to reproduce the data.

23. The computer of claim 22, wherein the processing function comprises one or more of:

a data component priority segregation function;

a spectral analysis function; and

a data filter function.

24. The computer of claim 22 further comprises:

the fundamental component data including first level of fundamental component data and second level of fundamental component data;

the sets of encoded data slices including a first plurality of sets of encoded data slices and a second plurality of sets of encoded data slices; and

the processing module further functions to;

dispersed storage error decode the first plurality of sets of encoded data slices in accordance with first dispersed storage error coding parameters to reproduce the first level of fundamental component data; and

dispersed storage error decode the second plurality of sets of encoded data slices in accordance with second dispersed storage error coding parameters to reproduce the second level of fundamental component data.

25. The computer of claim 22, wherein the processing module further functions to:

receive, via the interface, as the enhancement component data, a second plurality of sets of encoded data slices; and

dispersed storage error decode the enhancement component data in accordance with dispersed storage error coding parameters to reproduce the enhancement component data, wherein the second dispersed storage error coding parameters have a lower reception reliability than the first dispersed storage error coding parameters.

26. The computer of claim 22 further comprises:

the data including video data; and

the processing module further functions to;

process the fundamental component data and the enhancement component data in accordance with a video processing function to reproduce the video data.