Distinguishing data streams to enhance data storage efficiency

• US 8,352,540 B2
• Filed: 03/06/2008
• Issued: 01/08/2013
• Est. Priority Date: 03/06/2008
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Anglin, Matthew Joseph, Hochberg, Avishai Haim, Smith, James Patrick, Rasmussen, Neil Gregory
• Primary Examiner(s): Moore, Ian N
• Assistant Examiner(s): TANG, KIET G
• Application Number:
  US12/043,797
• Publication Number:
  US 20090228599A1
• Time in Patent Office: 1,769 Days
• Field of Search: 709/201, 709/217, 709/220, 709/226, 711/162, 707/100
• US Class Current: 709/201
• CPC Class Codes: G06F 11/1453 using de-duplication of the data
• CPC Class Codes: G06F 3/0608 Saving storage space on storage systems
• CPC Class Codes: G06F 3/0641 De-duplication techniques
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]

Abstract

Systems, methods, and computer products for communicating between a client and server by identifying and separating file data streams within a file are provided. The method comprises indicating the type of file data stream to be transmitted and transmitting the indicated file data stream. The transmitted file data stream is compared with a stored data stream. A non-redundant file data stream is stored based upon the outcome of the comparison. The transmitted file data stream and stored data stream may be compared according to a deduplication table based on data stream profiles.

Claims

1. A method of backing up a data file from a first system to a second system, wherein the data file comprises first data and second data, the method comprising:

identifying data included in the data file, prior to transmission of the data file in a data stream to the second system, such that the second system is able to distinguish the first data from the second data in the data file,

wherein the first data comprises information stored in the data file as content that is to be backed up on storage media by the second system, and the second data comprises information defining one or more attributes associated with said content in the first data,

wherein the second system processes the second data to determine whether content included in the first data is a candidate for deduplication, prior to comparing the content of the first data with third data, previously backed up on the storage media, to determine any redundancies between the content in the first data and the third data,

wherein if processing the second data indicates that an attribute associated with the content in the first data is different from an attribute associated with the content in the third data, then content of the first data is not deduplicated with respect to the third data,

wherein if processing the second data indicates that an attribute associated with the content in the first data is the same as an attribute associated with the content in the third data, then content of the first data is deduplicated with respect to the third data,

such that portions of content in the first data that are non-redundant with respect to the third data are stored in the storage media by the second system, and portions of the content in the first data that are redundant with respect to the third data are not stored in the storage media and instead a reference pointer is provided to portions of the third data that include a copy of the redundant portions of the content in the first data, instead of duplicating the redundant portions of the content on the storage media,

wherein the second data comprises metadata including attributes for identifying the content in the first data as belonging to a class, category or type, wherein the metadata is fixed in size, and

wherein the second data further comprises associated stream data related to the content in the first data,

wherein the associated stream data provides additional information about the content in the first data,

wherein said additional information is not included in the metadata, wherein the associated stream data is not fixed in size,

wherein the metadata is used to determine whether the associated stream data is to be deduplicated.

2. The method of claim 1 wherein the second data is used to classify the first data according to a plurality of data types, wherein the data types provide information about operating environment associated with the first system, wherein the second system processes the first data according to the data types to select the third data with which to compare the first data for the purpose of deduplication.

3. The method of claim 1 wherein the second data is used to classify the first data according to a plurality of data types, wherein the data types provide information about data categories to which the first data may belong, wherein the second system processes the first data according to the data types to select the third data with which to compare the first data for the purpose of deduplication.

4. The method of claim 2, wherein the information about the operating environment comprises information about the first system'"'"'s operating system.

5. The method of claim 2, wherein the information about the operating environment comprise information about the first system'"'"'s file system.

6. The method of claim 3, wherein the data categories identify nature of the content in the first data.

7. The method of claim 6, wherein the nature of the content may be identified as multimedia or text.

8. The method of claim 6, wherein the nature of the content may be identified according to software application utilized to create the content.

9. A storage management system comprising:

a first system communicatively coupled to a network, such that the first system may transmit a data file to a second system communicatively coupled to the network, wherein the data file comprises first data and second data, said first system configured to identify data included in the data file, prior to transmission of the data file in a data stream to the second system, such that the second system is able to distinguish the first data from the second data in the data file,

wherein the first data comprises information stored in the data file as content that is to be backed up on storage media by the second system, and the second data comprises information defining one or more attributes associated with said content in the first data,

wherein the second system processes the second data to determine whether content included in the first data is a candidate for deduplication, prior to comparing the content of the first data with third data, previously backed up on the storage media, to determine any redundancies between the content in the first data and the third data,

wherein if processing the second data indicates that an attribute associated with the content in the first data is different from an attribute associated with the content in the third data, then content of the first data is not deduplicated with respect to the third data,

wherein if processing the second data indicates that an attribute associated with the content in the first data is the same as an attribute associated with the content in the third data, then content of the first data is deduplicated with respect to the third data,

such that portions of content in the first data that are non-redundant with respect to the third data are stored in the storage media by the second system, and portions of the content in the first data that are redundant with respect to the third data are not stored in the storage media and instead a reference pointer is provided to portions of the third data that include a copy of the redundant portions of the content in the first data, instead of duplicating the redundant portions of the content on the storage media,

wherein the second data comprises metadata including attributes for identifying the content in the first data as belonging to a class, category or type, wherein the metadata is fixed in size, and

wherein the second data further comprises associated stream data related to the content in the first data,

wherein the associated stream data provides additional information about the content in the first data,

wherein said additional information is not included in the metadata, wherein the associated stream data is not fixed in size,

wherein the metadata is used to determine whether the associated stream data is to be deduplicated.

10. The storage management system of claim 9 wherein the second data is used to classify the first data according to a plurality of data types, wherein the data types provide information about operating environment associated with the first system, wherein the second system processes the first data according to the data types to select the third data with which to compare the first data for the purpose of deduplication.

11. The storage management system of claim 9 wherein the second data is used to classify the first data according to a plurality of data types, wherein the data types provide information about data categories to which the first data may belong, wherein the second system processes the first data according to the data types to select the third data with which to compare the first data for the purpose of deduplication.

12. The storage management system of claim 10, wherein the information about the operating environment comprises information about the first system'"'"'s operating system.

13. The storage management system of claim 10, wherein the information about the operating environment comprises information about the first system'"'"'s file system.

14. The storage management system of claim 11, wherein the data categories identify nature of the content in the first data.

15. A computer program product comprising a non-transitory computer readable storage medium including computer usable program code for transmitting a data file from a first system to a second system, said computer program product including:

computer usable program code for identifying data included in the data file, prior to transmission of the data file in a data stream to the second system, such that the second system is able to distinguish the first data from the second data in the data file,

wherein the first data comprises information stored in the data file as content that is to be backed up on storage media by the second system, and the second data comprises information defining one or more attributes associated with said content in the first data,

wherein the second system processes the second data to determine whether content included in the first data is a candidate for deduplication, prior to comparing the content of the first data with third data, previously backed up on the storage media, to determine any redundancies between the content in the first data and the third data,

wherein if processing the second data indicates that an attribute associated with the content in the first data is different from an attribute associated with the content in the third data, then content of the first data is not deduplicated with respect to the third data,

wherein if processing the second data indicates that an attribute associated with the content in the first data is the same as an attribute associated with the content in the third data, then content of the first data is deduplicated with respect to the third data,

such that portions of content in the first data that are non-redundant with respect to the third data are stored in the storage media by the second system, and portions of the content in the first data that are redundant with respect to the third data are not stored in the storage media and instead a reference pointer is provided to portions of the third data that include a copy of the redundant portions of the content in the first data, instead of duplicating the redundant portions of the content on the storage media,

wherein the second data comprises metadata including attributes for identifying the content in the first data as belonging to a class, category to type, wherein the metadata is fixed in size, and

wherein the second data further comprises associated stream data related to the content in the first data,

wherein the associated stream data provides additional information about the content in the first data,

wherein said additional information is not included in the metadata, wherein the associated stream data is not fixed in size,

wherein the metadata is used to determine whether the associated stream data is to be deduplicated.

16. The computer program product of claim 15 wherein the second data is used to classify the first data according to a plurality of data types, wherein the data types provide information about operating environment associated with the first system, wherein the second system processes the first data according to the data types to select the third data with which to compare the first data for the purpose of deduplication.

17. The computer program product of claim 16 wherein the second data is used to classify the first data according to a plurality of data types, wherein the data types provide information about data categories to which the first data may belong, wherein the second system processes the first data according to the data types to select the third data with which to compare the first data for the purpose of deduplication.