EFFICIENT DATA COMPRESSION AND ANALYSIS AS A SERVICE

• US 20140351229A1
• Filed: 05/22/2013
• Published: 11/27/2014
• Est. Priority Date: 05/22/2013
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: GUPTA, ANURAG WINDLASS
• Granted Patent: US 9,384,204 B2
• Time in Patent Office: Days
• US Class Current: 707/693
• CPC Class Codes: G06F 16/1744 using compression, e.g. sparse files
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: H03M 7/30 Compression speech analysis-synthesis for redundancy reduction G10L19/00; for image communication H04N; Expansion; Suppression of unnecessary data, e.g. redundancy reduction
• CPC Class Codes: H03M 7/3073 Time
• CPC Class Codes: H03M 7/3079 Context modeling
• CPC Class Codes: H03M 7/607 Selection between different types of compressors
• CPC Class Codes: H03M 7/6082 Selection strategies
• CPC Class Codes: H03M 7/6088 according to the data type

Abstract

Data may be efficiently analyzed and compressed as part of a data compression service. A data compression request may be received from a client indicating data to be compressed. An analysis of the data or metadata associated with the data may be performed. In at least some embodiments, this analysis may be a rules-based analysis. Some embodiments may employ one or more machine learning techniques to historical compression data to update the rules-based analysis. One or more compression techniques may be selected out of a plurality of compression techniques to be applied to the data. Data compression candidates may then be generated according to the selected compression techniques. In some embodiments, a compression service restriction may be enforced. One of the data compression candidates may be selected and sent in a response.

Claims

1. A system, comprising:

one or more computing devices configured to implement a compression service, comprising;

a plurality of compression engines, wherein each compression engine is configured to perform at least one compression technique out of a plurality of compression techniques;

a rules-based compression engine selector, configured to;

receive data from a client to be compressed;

in response to receiving the data;

perform a rules-based analysis on data or metadata associated with the data to be compressed in order to select one or more compression techniques out of the plurality of compression techniques to be applied to the data;

direct one or more of the plurality of compression engines to generate one or more compression data candidates according to the selected one or more compression techniques and in compliance with a given compression service restriction;

a response generation module, configured to;

select one of the one or more data compression candidates to send as requested compressed data according to a compression selection criteria.

2. The system of claim 1,

wherein, to perform a rules-based analysis on the data or the metadata associated with the data to be compressed in order to select the one or more compression techniques out of the plurality of compression techniques to be applied to the data, the rules-based compression engine selector is configured to apply a current set of compression selection rules to one or more data characteristics for the data;

wherein the compression service further comprises a machine-learning compression analysis module, configured to;

perform one or more machine-learning techniques on historical compression data to update the current set of compression selection rules.

3. The system of claim 1, wherein the response generation module is further configured to:

determine an entropy measure for the selected one of the one or more data compression candidates; and

in response to determining that the entropy measure for the selected one of the one or more data compression candidates is less than an entropy threshold, compress the selected one according to a system compression technique to generate multi-level compressed data to be sent as the requested compressed data.

4. A method, comprising:

performing, by one or more computing devices;

receiving a compression request from a client indicating data to be compressed;

in response to receiving the compression request;

performing an analysis on the data or metadata associated with the data to be compressed in order to select one or more compression techniques out of a plurality of compression techniques to be applied to the data;

generating one or more data compression candidates according to the one or more compression techniques;

selecting one of the one or more data compression candidates to send as requested compressed data according to a compression selection criteria; and

sending a response including the requested compressed data.

5. The method of claim 4, wherein said generating the one or more data compression candidates according to the one or more compression techniques comprises:

until a given compression service restriction is exceeded, generating a data compression candidate for each of the one or more compression techniques.

6. The method of claim 5, wherein the given compression service restriction is a time limit.

7. The method of claim 4, wherein the analysis on the data or metadata associated with the data to be compressed is a rules-based analysis.

8. The method of claim 7,

wherein the method further comprises:

performing one or more machine-learning techniques on historical compression data to update a current set of compression selection rules to be applied to select the one or more compression techniques;

wherein said performing the analysis on the data or the metadata associated with the data to be compressed in order to select the one or more compression techniques out of the plurality of compression techniques to be applied to the data comprises;

applying the updated set of compression selection rules to one or more data characteristics for the data to select the one or more compression techniques out of the plurality of compression techniques to be applied to the data.

9. The method of claim 8, wherein the metadata indicates a type or other one or more characteristics of the data to be compressed, and wherein the one or more machine-learning techniques are performed based, at least in part, on the type or other one or more characteristics for the data to be compressed.

10. The method of claim 8, wherein said performing the analysis on the data or the metadata associated with the data to be compressed in order to select the one or more compression techniques out of the plurality of compression techniques to be applied to the data further comprises including in the selected one or more compression techniques a randomly selected compression technique.

11. The method of claim 8, further comprising:

receiving a plurality of other data from a plurality of clients to be compressed;

for each of the plurality of other data;

performing said rules-based analysis, said generating, said selecting, and said sending; and

storing compression results data and data characteristics for the other data as part of the historical compression data.

12. The method of claim 4, further comprising:

prior to sending the response including the requested compressed data, compressing the selected data compression candidate according to a system compression technique to generate multi-level compressed data to be sent as the requested compressed data.

13. The method of claim 4, wherein the data to be compressed is a data stream comprising a plurality of data chunks, wherein said performing, said generating, and said selecting are performed for the first data chunk to be compressed of the plurality of data chunks to be compressed, and wherein the method further comprises:

for each of the subsequent data chunks of the plurality of data chunks;

generating a compressed data chunk according to the one or more data compression techniques applied to compress the first data chunk; and

sending a response including the data chunk.

14. The method of claim 4, further comprising:

receiving a decompression request from another client indicating compressed data, wherein said decompression request indicates one or more compression techniques applied to generate the compressed data;

decompressing the compressed data to generate a decompressed data object according to the indicated one or more compression techniques applied to generate decompressed data; and

sending a response to the other client including the decompressed data.

15. The method of claim 4, further comprising, prior to sending a response including the requested compressed data, encrypting the selected compressed data candidate according to one or more compression techniques to send to the

16. The method of claim 4,

wherein the one or more computing devices are work together to implement a network-based data compression service, wherein the compression request is received according to an interface for the network-based compression service;

wherein the method further comprises;

in response to receiving the compression request;

determining, based, at least in part, on the compression request, a fee structure for the compression request; and

generating a fee associated with the requested compressed data according to the determined fee structure;

wherein said performing the analysis on the data or metadata associated with the data to be compressed, said generating the one or more data compression candidates according to the one or more compression techniques, and said selecting the one of the one or more data compression candidates are performed in accordance with the determined fee structure.

22. The non-transitory, computer-readable storage medium of claim 14, wherein, in said generating the one or more data compression candidates according to the one or more compression techniques, the program instructions when executed by the one or more computing devices implement:

until a given compression service restriction is exceeded, generating a data compression candidate for each of the one or more compression techniques, wherein said given compression service restriction is a compression service fee cap.

23. The non-transitory, computer-readable storage medium of claim 14, wherein the selected one or more compression techniques to be applied to the data are ordered in a particular sequence, and wherein, in said generating the one or more data compression candidates according to the one or more compression techniques, the program instructions when executed by the one or more computing devices implement generating the one or more data compression candidates according to the particular sequence of the one or more compression techniques.

24. The non-transitory, computer-readable storage medium of claim 14,

wherein the program instructions, when executed by the one or more computing devices further implement:

performing one or more machine-learning techniques on historical compression data to update a current set of compression selection rules to be applied to select the one or more compression techniques;

wherein, in said performing the rules-based analysis on the data or the metadata associated with the data to be compressed in order to select the one or more compression techniques out of the plurality of compression techniques to be applied to the data, the program instructions when executed by the one or more computing devices implement;

applying the updated set of compression selection rules to one or more data characteristics for the data to select the one or more compression techniques out of the plurality of compression techniques to be applied to the data.

25. The non-transitory, computer-readable storage medium of claim 14, wherein the response including the requested compressed data is sent to a remote system different than the client.

17. A system, comprising:

a plurality of computing devices configured to implement a network-based service, comprising;

a data compression service module, configured to compress data according to one or more compression techniques;

a network-based service interface, configured to;

receive a compression request from a client indicating data to be compressed;

in response to receiving the compression request;

determine a fee structure associated with the compression request;

direct the data compression service module to generate requested compressed data according to the fee structure associated with the request;

generate a fee for the requested compressed data according to the fee structure; and

send the requested compressed data according to the compression request.

18. The system of claim 17, wherein the fee structure indicates a compression service restriction for the compression request, and wherein, to wherein, to generate the requested compressed data, the data service compression module is configured to generate the requested compressed data within the compression service restriction.

19. The system of claim 17, wherein the compression request further indicates one or more client selected compression techniques to be applied to the indicated data, and wherein, to generate the requested compressed data, the data service compression module is configured to perform the one or more client selected compression techniques to generate the requested compressed data.

20. The system of claim 17, wherein the compression request further requests compression analysis of the data to be compressed, and wherein, to compress the data according to one or more compression techniques, the data compression service module is configured to:

perform an analysis on the data or metadata associated with the data to be compressed in order to select one or more compression techniques out of a plurality of compression techniques to be applied to the data;

generate one or more data compression candidates according to the one or more compression techniques; and

select one of the one or more data compression candidates as the requested compressed data according to a compression selection criteria.

21. A non-transitory, computer-readable storage medium, storing program instructions that when executed by one or more computing devices implement:

receiving a compression request from a client indicating data to be compressed;

in response to receiving the compression request;

performing an on the data or metadata associated with the data to be compressed in order to select one or more compression techniques out of a plurality of compression techniques to be applied to the data;

generating one or more data compression candidates according to the one or more compression techniques;

selecting one of the one or more data compression candidates to send as requested compressed data according to a compression selection criteria; and

sending a response including the requested compressed data.