Method and system to scan data from a system that supports deduplication

• US 8,832,042 B2
• Filed: 03/15/2010
• Issued: 09/09/2014
• Est. Priority Date: 03/15/2010
• Status: Active Grant

• Current Assignee: Veritas Technologies, LLC (Whitehouse Group Ltd.)
• Original Assignee: Symantec Corporation (NortonLifeLock Inc.)
• Inventors: Agrawal, Mukund
• Primary Examiner(s): Spieler, William
• Application Number:
  US12/724,235
• Publication Number:
  US 20110225129A1
• Time in Patent Office: 1,639 Days
• Field of Search: 707/693, 707/692
• US Class Current: 707/692
• CPC Class Codes: G06F 16/1748 De-duplication implemented within the file system, e.g. based on file segments de-duplication techniques in storage systems for the management of data blocks G06F3/0641

Abstract

An interface is disclosed that makes information obtained from a file deduplication process available to an application for the efficient operation thereof. A data deduplication repository is scanned to determine a plurality of file segments and respective checksum values associated with the segments. A data structure is generated that allows shared segments to be identified by indexing using a common checksum value. The segments also indicate the file to which they belong and may also include a timestamp value. This data structure is updated as files are modified, etc. The data structure is accessible to an application program so that the application program can readily determine which segments are shared between multiple files. With this information, the application can efficiently process the segment once rather than multiple times. Timestamps can be used by the application to efficiently identify only those segments that were accessed after a given time.

Claims

1. A method of providing file information relating to data deduplication in a computer system, comprising:

accessing a plurality of files, wherein each file comprises a plurality of segments;

accessing a data repository storing data resultant from a file deduplication process including accessing a plurality of checksum values associated with said plurality of segments, wherein a first application program performs said file deduplication process;

identifying segments of said plurality of segments having a same checksum value;

generating a data association structure by associating said segments of said plurality of segments having said same checksum value, wherein a first checksum value is operable as an index into said data association structure for obtaining segments having said first checksum value;

storing, using a deduplication database, said data association structure;

storing a plurality of respective timestamps associated with said plurality of segments, wherein each respective timestamp indicates a last time an associated segment was altered;

accessing said stored data association structure in said computer memory by a second application program using a received application timestamp and accessing one or more segments of said plurality of segments having an associated timestamp of said plurality of respective timestamps that is newer than the received application timestamp, wherein the second application program identifies, using only said associated timestamp, one or more segments of said plurality of segments that are not processed by the second application program;

receiving, by the second application program, a listing of files to which a segment of said one or more segments of said plurality of segments belongs; and

comparing said listing of files against a defined subset of files to exclude from processing to further determine whether said segment needs to be processed.

2. The method as described in claim 1 wherein within said data association structure, each segment indicates a respective file to which each segment is associated.

3. The method as described by claim 1 further comprising:

accessing said data repository to identify a new checksum value associated with an altered segment of said plurality of segments;

updating said data association structure based on said new checksum value to form an updated data association structure; and

storing said updated data association structure.

4. The method as described by claim 1, wherein said data association structure further comprises said plurality of time stamps associated with said plurality of checksum values.

5. The method as described by claim 4 further comprising:

from said data repository, accessing a new time stamp associated with an altered segment of said plurality of segments;

updating said data association structure to form an updated data association structure; and

storing said updated data association structure.

6. The method as described by claim 1 further comprising:

providing a portion of said data association structure to an application requesting access to a segment of said plurality of segments corresponding to said data association structure.

7. The method as described by claim 1 further comprising:

accessing a new file comprising a plurality of segments;

from said data repository, accessing a plurality of respective checksum values associated with said plurality of segments of said new file;

identifying segments of said new file and segments of said plurality of files having a same checksum value;

associating segments having a same checksum value to one another based on said identifying segments of said new file and segments of said plurality of files having a same checksum value to form an updated data association structure; and

storing said updated data association structure.

8. The method as described by claim 1 wherein storing said plurality of respective timestamps associated with said plurality of segments comprises storing said plurality of respective timestamps in said data association structure.

9. A non-transitory computer readable storage medium having stored thereon computer executable instructions that, if executed by a computer system, cause the computer system to perform a method of providing file information relating to data deduplication, said method comprising:

accessing a plurality of files, wherein each file comprises a plurality of segments;

accessing a data repository storing data resultant from a file deduplication process including accessing a plurality of checksum values associated with said plurality of segments, wherein a first application program performs said file deduplication process;

identifying segments of said plurality of segments having a same checksum value;

generating a data association structure by associating said segments of said plurality of segments having said same checksum value, wherein a first checksum value is operable as an index into said data association structure for obtaining segments having said first checksum value, and wherein, within said data association structure, each segment indicates a respective file to which each segment is associated;

storing, using a deduplication database, said data association structure;

storing a plurality of respective time stamps associated with said plurality of segments, wherein each respective time stamp indicates a last time an associated segment was altered;

accessing said stored data association structure in said computer memory by a second application program using a received application timestamp and accessing one or more segments of said plurality of segments having an associated timestamp of said plurality of respective time stamps that is newer than the received application timestamp, wherein the second application program identifies, using only said respective timestamps, one or more segments of said plurality of segments that are not processed by the second application program;

receiving, by the second application program, a listing of files to which a segment of said one or more segments of said plurality of segments belongs; and

comparing said listing of files against a defined subset of files to exclude from processing to further determine whether said segment needs to be processed.

10. The non-transitory computer readable storage medium as described by claim 9 wherein said method further comprises:

accessing said data repository to identify a new checksum value associated with an altered segment of said plurality of segments;

updating said data association structure based on said new checksum value to form an

updated data association structure; and

storing said updated data association structure.

11. The non-transitory computer readable storage medium as described by claim 9, wherein said data association structure further comprises said plurality of time stamps associated with said plurality of checksum values.

12. The non-transitory computer readable storage medium as described by claim 11 wherein said method further comprises:

from said data repository, accessing a new time stamp associated with an altered segment of said plurality of segments;

updating said data association structure to form an updated data association structure; and

storing said updated data association structure.

13. The non-transitory computer readable storage medium as described by claim 9 wherein said method further comprises:

providing a portion of said data association structure to an application requesting access to a segment of said plurality of segments corresponding to said data association structure.

14. The non-transitory computer readable storage medium as described by claim 9 wherein said method further comprises:

accessing a new file comprising a plurality of segments;

from said data repository, accessing a plurality of respective checksum values associated with said plurality of segments of said new file;

identifying segments of said new file and segments of said plurality of files having a same checksum value;

associating segments having a same checksum value to one another based on said identifying segments of said new file and segments of said plurality of files having a same checksum value to form an updated data association structure; and

storing said updated data association structure;

storing a plurality of respective time stamps associated with said segments, wherein each respective time stamp indicates a last time an associated segment was altered; and

accessing said stored data association using a received application timestamp, wherein accessing said stored data association comprises accessing one or more segments of said plurality of segments having a newer associated timestamp of said plurality of respective time stamps than the received application timestamp.

15. The non-transitory computer readable storage medium as described by claim 9 wherein storing said plurality of respective time stamps associated with said plurality of segments comprises storing a plurality of respective time stamps in said data association structure.

16. A method of providing information relating to a deduplication process in a computer system, said method comprising:

generating a segment index data structure responsive to performing a data deduplication process on a plurality of files of a file system, wherein each file of said plurality of files comprises a plurality of segments and wherein said segment index data structure comprises a listing of unique segments within said plurality of segments, and wherein a first application performs said data deduplication process;

storing, using a deduplication database, a plurality of respective time stamps associated with said plurality of segments, wherein each respective time stamp indicates a last time an associated segment was altered;

receiving a request from a requesting application for segment information;

responsive to said request, scanning said segment index data structure;

responsive to said scanning, supplying segment information to said requesting application, wherein said request includes an application timestamp indicating a last time said requesting application processed data within said file system, and wherein said scanning comprises scanning only respective segments of said segment index data structure having respective timestamps of said plurality of respective time stamps having a time after said application timestamp, wherein the requesting application identifies, using only said respective timestamps, one or more segments of said respective segments that are not processed by the requesting application;

receiving, by the second application program, a listing of files to which a segment of said one or more segments of said plurality of segments belongs; and

comparing said listing of files against a defined subset of files to exclude from processing to further determine whether said segment needs to be processed.

17. The method as described in claim 16 wherein said segment information comprises, for each respective segment, a segment identifier and a list of files to which said respective segment belongs and further comprising said requesting application using said segment information to determine whether or not to process respective segments.

18. The method as described in claim 16 wherein said scanning and said supplying are performed iteratively.

19. The method as described in claim 16 wherein said unique segments are identified by respective unique checksum values and wherein further said segment index data structure resides within a repository unit of a data deduplication unit.

20. The method as described in claim 16 wherein storing said plurality of respective time stamps associated with said plurality of segments comprises storing said plurality of respective time stamps in said segment index data structure.