SCALABLE DE-DUPLICATION MECHANISM

• US 20090182789A1
• Filed: 01/16/2009
• Published: 07/16/2009
• Est. Priority Date: 08/05/2003
• Status: Active Grant

• Current Assignee: Hitachi Vantara, LLC (Hitachi, Ltd.)
• Original Assignee: Sepaton Incorporated (Hitachi, Ltd.)
• Inventors: Reiter, Timmie G., Sandorfi, Miklos
• Granted Patent: US 8,280,926 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 11/1004 to protect a block of data words, e.g. CRC or checksum G06F11/1076 takes precedence; security arrangements for protecting computers or computer systems against unauthorized activity G06F21/00
• CPC Class Codes: G06F 11/1453 using de-duplication of the data

Abstract

A method for removing redundant data from a backup storage system is presented. In one example, the method may include receiving the application layer data object, selecting a de-duplication domain from a plurality of de-duplication domains based at least in part on a data object characteristic associated with the de-duplication domain, determining that the application layer data object has the characteristic and directing the application layer data object to the de-duplication domain.

Claims

1. A method for directing de-duplication of an application layer data object, the method comprising:

receiving the application layer data object;

selecting a de-duplication domain from a plurality of de-duplication domains based at least in part on a data object characteristic associated with the de-duplication domain;

determining that the application layer data object has the characteristic; and

directing the application layer data object to the selected de-duplication domain.

2. The method according to claim 1, wherein receiving the application layer data object comprises:

receiving a data stream; and

identifying the application layer data object using metadata included in the data stream.

3. The method according to claim 2, wherein receiving the data stream includes receiving a multiplexed data stream.

4. The method according to claim 2, further comprising extracting metadata included in the data stream with the application layer data object.

5. The method according to claim 4, wherein selecting the de-duplication domain from the plurality of de-duplication domains includes comparing the extracted metadata associated with the application layer data object to the at least one characteristic associated with the de-duplication domain.

6. The method according to claim 4, wherein extracting the metadata included in the data stream includes extracting at least one of a back-up policy name, a data source type, a data source name, a back-up application name, an operating system type, a data type, a back-up type, a filename, a directory structure and chronological information.

7. The method according to claim 1, further comprising configuring each of the plurality of de-duplication domains to use one of a plurality of de-duplication methods.

8. The method according to claim 7, wherein configuring each of the plurality of de-duplication domains includes configuring each of the plurality of de-duplication domains to use one de-duplication method selected from the group comprising hash-fingerprinting, pattern recognition and content aware de-duplication.

9. The method according to claim 1, further comprising associating each of the plurality of de-duplication domains with at least one data object characteristic.

10. The method according to claim 1, further comprising:

de-duplicating, within the selected de-duplication domain, the application layer data object; and

adjusting the data object characteristic associated with at least one of the plurality of de-duplication domains based on a result of the act of de-duplicating.

11. The method according to claim 10, wherein adjusting the data object characteristic includes storing data in a de-duplication domain database.

12. A computer-readable medium having computer-readable signals stored thereon that define instructions that, as a result of being executed by a computer, instruct the computer to perform the method of claim 1.

13. The method according to claim 1, wherein the method is executed in a grid computing environment.

14. The method according to claim 1, wherein the method is executed on a back-up storage system while data is not being backed-up to the back-up storage system.

15. The method according to claim 1, wherein the method is executed on a back-up storage system while data is being backed-up to the back-up storage system.

16. A system for directing de-duplication of an application layer data object, the system comprising:

a plurality of de-duplication domains, each de-duplication domain of the plurality of de-duplication domains associated with at least one characteristic common to a plurality of application layer data objects; and

a controller coupled to the plurality of de-duplication domains and configured to;

receive the application layer data object;

determine that the application layer data object has the at least one characteristic associated with a de-duplication domain; and

direct the application layer data object to the de-duplication domain.

17. The system according to claim 16, wherein the controller is further configured to:

receive a data stream; and

identify the application layer data object using metadata included in the data stream.

18. The system according to claim 17, wherein the data stream is multiplexed.

19. The system according to claim 17, wherein the controller is further configured to extract metadata included in the data stream with the application layer data object.

20. The system according to claim 19, wherein the controller is further configured to determine that the application layer data object has the at least one characteristic associated with the de-duplication domain by comparing the extracted metadata associated with the application layer data object to the at least one characteristic associated the with de-duplication domain.

21. The system according to claim 19, wherein the controller is further configured to extract at least one of a back-up policy name, a data source type, a data source name, a back-up application name, an operating system type, a data type, a back-up type, a filename, a directory structure and chronological information.

22. The system according to claim 16, wherein the controller is further arranged to configure each of the plurality of de-duplication domains to use one of a plurality of de-duplication methods.

23. The system according to claim 22, wherein the controller is further arranged to configure each of the plurality of de-duplication domains to use one de-duplication method selected from the group comprising hash-fingerprinting, pattern recognition and content aware de-duplication.

24. The system according to claim 16, wherein the controller is further configured to associate each of the plurality of de-duplication domains with at least one data object characteristic.

25. The system according to claim 16, wherein the controller is further configured to:

cause, within the selected de-duplication domain, de-duplication of the application layer data object; and

adjust the data object characteristic associated with at least one of the plurality of de-duplication domains based on a result of the act of de-duplicating.

26. The system according to claim 25, wherein the controller is further configured to store data in a de-duplication domain database.

27. The system according to claim 16, wherein the system is included in a grid computing environment.

28. The system according to claim 16, wherein the controller is further configured to receive the application layer data object, determine that the application layer data object has the at least one characteristic associated with a de-duplication domain and direct the application layer data object to the de-duplication domain while data is not being backed-up to the system.

29. The system according to claim 16, wherein the controller is further configured to receive the application layer data object, determine that the application layer data object has the at least one characteristic associated with a de-duplication domain and direct the application layer data object to the de-duplication domain while data is being backed-up to the system.