Systems and methods for facilitating data discovery

• US 9,135,261 B2
• Filed: 12/15/2009
• Issued: 09/15/2015
• Est. Priority Date: 12/15/2009
• Status: Active Grant

• Current Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Original Assignee: EMC Corporation (Dell Technologies Inc.)
• Inventors: Maunder, Anurag S., Tryfonas, Christos, Sudhakar, Muddu
• Primary Examiner(s): Truong, Dennis
• Application Number:
  US12/638,067
• Publication Number:
  US 20110145217A1
• Time in Patent Office: 2,100 Days
• Field of Search: 707/999.003, 707/999.005, 707/999.1, 707/705, 707/706, 707/709, 707/710, 707/736, 707/741, 707/755, 707/756, 707/790, 707/791, 709/200, 709/204, 709/206, 709/207, 705/1, 726/22, 713/188
• US Class Current: 1/1
• CPC Class Codes: G06F 16/134 Distributed indices
• CPC Class Codes: G06F 16/148 File search processing

Abstract

A system for facilitating data discovery on a network, wherein the network has one or more data storage devices. The system may include a crawler program configured to select at least a first set of files and a second set of files, each of the first set of files and the second set of files being stored in at least one of the one or more data storage devices. The system may also include a data fetcher program configured to obtain a copy of the first set of files, the data fetcher program being further configured to resist against obtaining a copy of the second set of files. The system may also include circuit hardware implementing one or more functions of one or more of the crawler program and the data fetcher program.

Claims

1. A system for facilitating data discovery on a network, the network having one or more data storage devices, the system comprising:

a crawler program configured to scan files stored on the one or more data storage devices, and identify a first set of files and a second set of files as being relevant, the scanning and the identifying being performed at a crawler operating speed, the crawler program being further configured to delay scanning based on at least one of the following conditions;

(1) a file path length associated with one or more files in the first set of files exceeds a file length threshold; and

     2) one or more filer lengths associated with one or more files of the of the first set of files exceed a filer length threshold;

a data fetcher program configured to receive a location of the first set of files identified by the crawler program, the location being on the one or more data storage devices, and copy the first set of files from the received location at a data fetcher operating speed, the data fetcher program being further configured to delay copying the second set of files, thereby causing the crawler program to adjust the crawler operating speed of the scanning and the identifying according to the data fetcher operating speed based on at least one of the following conditions;

(1) a file size associated with a file in the second set of files is smaller than a file size threshold, (2) a quantity of files in one of the first set and the second set of files exceeds a file quantity threshold, (3) a file format associated with a file of one of the first set and the second set of files does not belong to a predetermined set of file formats, and (4) an amount of text to index in the first set of files exceeds a text amount threshold; and

circuit hardware implementing one or more functions of one or more of the crawler program and the data fetcher program.

2. The system of claim 1 wherein the crawler program is further configured to create at least a checkpoint when the crawler program scans the first set of files, the checkpoint providing at least status notification of scanning performed by the crawler program, the crawler program resuming the scanning from the checkpoint after an interruption of the scanning.

3. The system of claim 1 wherein a quantity of files in the first set of files is different from a quantity of files in the second set of files.

4. The system of claim 1 wherein a quantity of files in the first set of files changes over time.

5. A system for facilitating data discovery on a network, the network having one or more data storage devices, the system comprising:

a crawler program configured to scan files stored on the one or more data storage devices, and identify a first set of files, a second set of files, a third set of files, and a fourth set of files as being relevant, the scanning and the identifying being performed at a crawler operating speed, the crawler program being further configured to delay scanning the fourth set of files based on at least one of the following conditions;

(1) a file path length associated with one or more files in the third set of files exceeds a file length threshold; and

     2) one or more filer lengths associated with one or more files of the of the third set of files exceed a filer length threshold;

a data fetcher program configured to receive a location of the first set of files identified by the crawler program, the location being on the one or more data storage devices, and copy the first set of files, a copy of the second set of files, and a copy of the third set of files from the received location at a data fetcher operating speed, the data fetcher program being further configured to delay copying the fourth set of files, thereby causing the crawler program to adjust crawler operating speed of the scanning and the identifying according to the data fetcher operating speed based on at least one of the following conditions;

(1) a file size associated with one or more files of the first set, the second set and the third set of files is smaller than a file size threshold, (2) one or more quantities of files in one of the first set, second set and third of files exceeds a file quantity threshold, (3) a file format associated with a file of one of the first set, second set and third set of files does not belong to a predetermined set of file formats, and (4) an amount of text to index in one of the first set, second set and third set of files exceeds a text amount threshold;

a processing program configured to perform one or more services on the copy of the first set of files and the copy of the second set of files, the processing program being further configured to delay performing any services on the copy of the third set of files;

a search indexing program configured to generate at least a search index using the copy of the first set of files, the search indexing program being further configured to delay generating any search index from the copy of the second set of files; and

circuit hardware implementing one or more functions of one or more of the crawler program, the data fetcher program, the processing program, and the search indexing program.

6. The system of claim 5 wherein the one or more services include extracting metadata from at least one of the copy of the first set of files and the copy of the second set of files.

7. The system of claim 5 wherein the one or more services include generate at least a hash code using data contained in at least one of the copy of the first set of files and the copy of the second set of files.

8. The system of claim 5 wherein the crawler program delays scanning the fourth set of files when one or more file path lengths associated with one or more files of the third set of files exceed a file path length threshold.

9. The system of claim 5 wherein the data fetcher program is further configured to notify the crawler program that the data fetcher is ready to obtain the copy of the fourth set of files after the data fetcher has delayed obtaining the copy of the fourth set of files.

10. The system of claim 5 wherein the processing program is further configured to notify the data fetcher program that the processing program is ready to perform at least a service on the copy of the third set of files after the processing program has delayed performing any services on the copy of the third set of files.

11. The system of claim 5 wherein the processing program delays performing any services on the copy of the third set of files when one or more file formats associated with one of more files of at least one of the first set of files and the second set of files do not belong to a predetermined set of file formats.

12. The system of claim 5 wherein the processing program delays performing any services on the copy of the third set of files when one or more file formats associated with one of more files of the third set of files do not belong to a predetermined set of file formats.

13. The system of claim 5 wherein the search indexing program is further configured to notify the processing program that the search indexing program is ready to generate a search index from the copy of the second set of files after the search index program has delayed generating any search index from the copy of the second set of files.

14. The system of claim 5 wherein the search indexing delays generating any search index from the copy of the second set of files in at least one of a first condition and a second condition, the first condition being that an amount of text to index in the first set of files exceeds a first text amount threshold, the second condition being that an amount of text to index in the second set of files exceeds a second text amount threshold.

15. A method for facilitating data discovery on a network, the network having one or more data storage devices, the method comprising:

scanning files stored on the one or more data storage devices, and identifying a first set of files and a second set of files as being relevant, the scanning and the identifying being performed by a crawler program at a crawler program operating speed, the crawler program being configured to delay scanning based on at least one of the following conditions;

(1) a file path length associated with one or more files in the first set of files exceeds a file length threshold; and

     2) one or more filer lengths associated with one or more files of the of the first set of files exceed a filer length threshold;

receiving, by a data fetcher program, a location of the first set of files identified by the crawler program, the location being on the one or more data storage devices; and

copying the first set of files from the received location at a data fetcher operating speed, the data fetcher program being configured to delay copying the second set of files, thereby causing the crawler program to adjust the crawler operating speed of the scanning and the identifying according to the data fetcher operating speed based on at least one of the following conditions;

(1) a file size associated with a file in the second set of files is smaller than a file size threshold, (2) a quantity of files in one of the first set and the second set of files exceeds a file quantity threshold, (3) a file format associated with a file of one of the first set and the second set of files does not belong to a predetermined set of file formats, and (4) an amount of text to index in the first set of files exceeds a text amount threshold.

16. The method of claim 15, further comprising creating at least a checkpoint when the crawler program scans the first set of files, the checkpoint providing at least status notification of scanning performed by the crawler program, the crawler program resuming the scanning from the checkpoint after an interruption of the scanning.

17. The method of claim 15 wherein a quantity of files in the first set of files is different from a quantity of files in the second set of files.

18. The method of claim 15 wherein a quantity of files in the first set of files changes over time.

22. The computer program product of claim 15 wherein a quantity of files in the first set of files changes over time.

19. A computer program product comprising computer-readable program code to be executed by one or more processors when retrieved from a non-transitory computer-readable medium, the program code including instructions to:

scan files stored on the one or more data storage devices, and identify, by the crawler program, a first set of files and a second set of files as being relevant, the scanning and the identifying being performed by a crawler program at a crawler program operating speed, the crawler program being configured to delay scanning based on at least one of the following conditions;

(1) a file path length associated with one or more files in the first set of files exceeds a file length threshold; and

     2) one or more filer lengths associated with one or more files of the of the first set of files exceed a filer length threshold; and

receive, by a data fetcher program, a location of the first set of files identified by the crawler program, the location being on the one or more data storage devices;

copying, by the data fetcher program, the first set of files from the received location at a data fetcher operating speed, the data fetcher program being configured to delay copying the second set of files, thereby causing the crawler program to adjust the crawler operating speed of the scanning and the identifying according to the data fetcher operating speed based on at least one of the following conditions;

(1) a file size associated with a file in the second set of files is smaller than a file size threshold, (2) a quantity of files in one of the first set and the second set of files exceeds a file quantity threshold, (3) a file format associated with a file of one of the first set and the second set of files does not belong to a predetermined set of file formats, and (4) an amount of text to index in the first set of files exceeds a text amount threshold.

20. The computer program product of claim 19, the program code including further instructions to create at least a checkpoint when the crawler program scans the first set of files, the checkpoint providing at least status notification of scanning performed by the crawler program, the crawler program resuming the scanning from the checkpoint after an interruption of the scanning.

21. The computer program product of claim 19 wherein a quantity of files in the first set of files is different from a quantity of files in the second set of files.