Determining whether to compress a data segment in a dispersed storage network

• US 10,162,524 B2
• Filed: 02/08/2017
• Issued: 12/25/2018
• Est. Priority Date: 08/02/2010
• Status: Active Grant

• Current Assignee: Pure Storage, Inc.
• Original Assignee: International Business Machines Corporation
• Inventors: Cilfone, Bart R., Leggette, Wesley B., Resch, Jason K.
• Primary Examiner(s): Alphonse, Fritz
• Application Number:
  US15/427,860
• Publication Number:
  US 20170147220A1
• Time in Patent Office: 685 Days
• Field of Search: 714764, 714766, 714768, 714770
• CPC Class Codes: G06F 11/1076 Parity data used in redundant arrays of independent storages, e.g. in RAID systems
• CPC Class Codes: G06F 21/6218 to a system of files or objects, e.g. local or distributed file system or database
• CPC Class Codes: G06F 3/0608 Saving storage space on storage systems
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/064 Management of blocks
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: G06F 3/0689 Disk arrays, e.g. RAID, JBOD
• CPC Class Codes: H03M 13/1515 Reed-Solomon codes
• CPC Class Codes: H03M 13/6312 Error control coding in combination with data compression
• CPC Class Codes: H04L 1/00 Arrangements for detecting or preventing errors in the information received correcting synchronisation H04L7/00
• CPC Class Codes: H04L 1/0057 Block codes H04L1/0061, H04L1/0064 take precedence
• CPC Class Codes: H04L 1/0061 Error detection codes
• CPC Class Codes: H04L 63/0428 wherein the data content is protected, e.g. by encrypting or encapsulating the payload
• CPC Class Codes: H04L 63/08 for authentication of entities cryptographic mechanisms or cryptographic arrangements for entity authentication H04L9/32
• CPC Class Codes: H04L 67/04 specially adapted for terminals or networks with limited capabilities; specially adapted for terminal portability
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 67/306 User profiles
• CPC Class Codes: H04L 69/22 Parsing or analysis of headers

Abstract

A method for execution by a computing device of a dispersed storage network (DSN). The method begins by receiving a data segment of a data object for dispersed storage error encoding. Prior to the dispersed storage error encoding, the method continues by determining whether to compress the data segment by predicting a first estimated processing cost based on estimated processing costs to compress the data segment to produce a compressed data segment and estimated processing costs to dispersed storage error encode the compressed data segment and predicting a second estimated processing cost based on estimated processing costs to dispersed storage error encode the data segment. When the first estimated processing cost compares favorably to the second estimated processing cost, the method continues by compressing the data segment to produce the compressed data segment and dispersed storage error encoding the compressed data segment to produce a set of encoded data slices.

Claims

1. A method for execution by a computing device of a dispersed storage network (DSN), the method comprises:

receiving a data segment of a data object for dispersed storage error encoding;

prior to the dispersed storage error encoding, determining whether to compress the data segment by;

predicting a first estimated processing cost based on estimated processing costs to compress the data segment to produce a compressed data segment and estimated processing costs to dispersed storage error encode the compressed data segment;

predicting a second estimated processing cost based on estimated processing costs to dispersed storage error encode the data segment; and

when the first estimated processing cost compares favorably to the second estimated processing cost;

compressing the data segment to produce the compressed data segment; and

dispersed storage error encoding the compressed data segment to produce a set of encoded data slices.

2. The method of claim 1, wherein the determining whether to compress the data segment further comprises:

determining whether the data segment has previously undergone a compression function; and

when the data segment has previously undergone a compression function, determining that the first estimated processing cost compares unfavorably to the second estimated processing cost; and

when the first estimated processing cost compares unfavorably to the second estimated processing cost, foregoing the compressing of the data segment.

3. The method of claim 1, wherein the compressing the data segment comprises:

determining a compression codec; and

compressing the data segment in accordance with the compression codec to produce the compressed data segment.

4. The method of claim 1 further comprises:

prior to compressing the data segment;

determining whether to encrypt the data segment; and

when determined to encrypt the data segment, determining an encryption codec; and

performing the encryption codec and a compression codec on the data segment to produce the compressed data segment.

5. The method of claim 1 further comprises:

prior to compressing the data segment;

determining whether to produce an integrity check for the data segment; and

when determined to produce the integrity check for the data segment, determining an integrity check codec; and

performing the integrity check codec and a compression codec on the data segment to produce the compressed data segment.

6. The method of claim 1 further comprises:

prior to compressing the data segment;

determining whether to produce an integrity check for the data segment;

when determined to produce the integrity check for the data segment, determining an integrity check codec;

determining whether to encrypt the data segment; and

when determined to encrypt the data segment, determining an encryption codec; and

performing the integrity check codec, the encryption codec, and a compression codec on the data segment to produce the compressed data segment.

7. A computing device comprises:

memory;

an interface; and

a processing module operably coupled to the memory and the interface, wherein the processing module is operable to;

receive a data segment of a data object for dispersed storage error encoding;

prior to the dispersed storage error encoding, determine whether to compress the data segment by;

predicting a first estimated processing cost based on estimated processing costs to compress the data segment to produce a compressed data segment and estimated processing costs to dispersed storage error encode the compressed data segment;

predicting a second estimated processing cost based on estimated processing costs to dispersed storage error encode the data segment; and

when the first estimated processing cost compares favorably to the second estimated processing cost;

compressing the data segment to produce the compressed data segment; and

dispersed storage error encoding the compressed data segment to produce a set of encoded data slices.

8. The computing device of claim 7, wherein the processing module is further operable to determine whether to compress the data segment by:

determining whether the data segment has previously undergone a compression function; and

when the data segment has previously undergone a compression function, determining that the first estimated processing cost compares unfavorably to the second estimated processing cost; and

when the first estimated processing cost compares unfavorably to the second estimated processing cost, foregoing the compressing of the data segment.

9. The computing device of claim 7, wherein the processing module is further operable to compress the data segment by:

determining a compression codec; and

compressing the data segment in accordance with the compression codec to produce the compressed data segment.

10. The computing device of claim 7, wherein the processing module is further operable to:

prior to compressing the data segment;

determine whether to encrypt the data segment; and

when determined to encrypt the data segment, determine an encryption codec; and

perform the encryption codec and a compression codec on the data segment to produce the compressed data segment.

11. The computing device of claim 7, wherein the processing module is further operable to:

prior to compressing the data segment;

determine whether to produce an integrity check for the data segment; and

when determined to produce the integrity check for the data segment, determine an integrity check codec; and

perform the integrity check codec and a compression codec on the data segment to produce the compressed data segment.

12. The computing device of claim 7, wherein the processing module is further operable to:

prior to compressing the data segment;

determine whether to produce an integrity check for the data segment;

when determined to produce the integrity check for the data segment, determine an integrity check codec;

determine whether to encrypt the data segment; and

when determined to encrypt the data segment, determine an encryption codec; and

perform the integrity check codec, the encryption codec, and a compression codec on the data segment to produce the compressed data segment.