Data Deduplication Apparatus and Method for Storing Data Received in a Data Stream From a Data Store

• US 20110022718A1
• Filed: 07/22/2010
• Published: 01/27/2011
• Est. Priority Date: 07/24/2009
• Status: Abandoned Application

• Current Assignee: Hewlett Packard Enterprise Development LP (Hewlett-Packard Enterprise Company)
• Original Assignee: Hewlett Packard Enterprise Development LP (Hewlett-Packard Enterprise Company)
• Inventors: Brady, Garry, Monk, Russell Ian, Evans, Nigel Ronald
• Application Number:
  US12/841,898
• Publication Number:
  US 20110022718A1
• Time in Patent Office: Days
• US Class Current: 709/231
• CPC Class Codes: G06F 11/1453 using de-duplication of the data
• CPC Class Codes: G06F 11/1464 for networked environments
• CPC Class Codes: G06F 16/24556 Aggregation; Duplicate elimination
• CPC Class Codes: G06F 3/0608 Saving storage space on storage systems
• CPC Class Codes: G06F 3/0641 De-duplication techniques
• CPC Class Codes: G06F 3/0686 Libraries, e.g. tape libraries, jukebox
• CPC Class Codes: H03M 7/30 Compression speech analysis-synthesis for redundancy reduction G10L19/00; for image communication H04N; Expansion; Suppression of unnecessary data, e.g. redundancy reduction

Abstract

A method of storing data received in a data stream from a data source is disclosed in which prior to performing deduplication on the data stream a processor decompresses selected compressed data entities in the data stream to provide a decompressed form of the data entities in the data stream in place of the compressed form, the data stream including the decompressed data entities is deduplicated and the deduplicated data is stored to a deduplicated data store.

Claims

1. Data deduplication apparatus for storing data received in a data stream from a data source, the apparatus comprising;

an encoded entity handler operable to;

identify, in the data stream, meta data associated with an encoded data entity, the meta data relating to an encoding process that has been used to encode the encoded data entity;

use the meta data to decode the encoded data entity to provide a decoded form thereof; and

substitute said decoded form of the encoded data entity for the encoded form thereof in the data stream; and

a deduplication engine to;

perform deduplication on the data stream including at least one said decoded data entity to provided deduplicated data; and

store the deduplicated data to a deduplicated data store.

2. The data deduplication apparatus of claim 1, wherein said deduplicated data store comprises secondary storage.

3. The data deduplication apparatus of claim 1, wherein the meta data comprises header meta data according to a data compression scheme that has been used to encode the encoded data entity, the header meta data facilitating a decompression process by which the encoded entity handler decodes the encoded data entity.

4. The data deduplication apparatus of claim 1, wherein the encoded entity handler is further to remove the identified meta data from the data stream, and store the meta data in an encoded entity meta data store for access when required during a read operation.

5. The data deduplication apparatus of claim 1, further comprising a command handler to identify command meta data in the received data stream, remove the command meta data from the data stream, and store the command meta data in a command meta data store for access when required during a read operation.

6. The data deduplication apparatus of claim 5, wherein the command handler is to remove the command meta data from the data stream prior to processing of the data stream by the encoded entity handler.

7. The data deduplication apparatus of claim 5, wherein the received data stream is a tape data backup stream formatted according to a tape data format, and the command meta data comprises command descriptor blocks relating to records and file marks.

8. A method of storing data received in a data stream from a data source, the method comprising:

prior to performing deduplication on a data stream, using a processor to decompress selected compressed data entities in the data stream to provide a decompressed form thereof to replace of the compressed form thereof;

deduplicating the data stream including the decompressed data entities; and

storing the deduplicated data to a deduplicated data store.

9. The method of claim 8, wherein storing the deduplicated data to a data store comprises storing the deduplicated data to secondary storage.

10. The method of claim 8, further comprising removing meta data from the data stream, and storing the meta data to a meta data store for access when required during a read operation.

11. The method of claim 10, wherein the meta data comprises header meta data according to a data compression scheme that has been used to encode the data entity, the header meta data enabling the data deduplication apparatus to perform decompression to decode the data entity.

12. The method of claim 10, wherein the meta data comprises command meta data in the received data stream.

13. Data deduplication storage apparatus for in-line processing of data received in a data stream from a data source, the apparatus comprising:

an encoded entity handler to;

receive the data stream and identify meta data in the data stream that is indicative of recognised encoded data formats, the identified meta data being associated with encoded data in the data stream;

use the identified meta data to decode the associated encoded data and provide a decoded form of the data in the data stream in place of the encoded form thereof; and

remove the identified meta data from the data stream; and

a deduplication engine to;

receive the data stream downstream of the encoded data entity handler and perform deduplication on the data stream to provide deduplicated data; and

secondary storage in which said deduplicated data is stored.

14. The data deduplication apparatus of claim 13, wherein said encoded entity handler is to remove said meta data from the data stream to a meta data store.

15. The data deduplication apparatus of claim 13, further comprising a command handler to identify command data in the data stream upstream of said encoded entity handler and remove the identified command meta data from the data stream to a meta data store.

16. The data deduplication apparatus of claim 15, wherein the received data stream is a tape data backup stream formatted according to a tape data format, and the command meta data comprises command descriptor blocks relating to records and file marks.

17. The data deduplication apparatus of claim 13, further comprising a buffer that receives the data stream downstream of the encoded entity data handler, said deduplication engine comprising a module that divides the data in the buffer into segments that are analysed for duplication by the deduplication engine.