Monitoring document changes in a file system of documents with the document change information stored in a persistent log

• US 6,067,541 A
• Filed: 09/17/1997
• Issued: 05/23/2000
• Est. Priority Date: 09/17/1997
• Status: Expired due to Term

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Shoroff, Srikanth, Peltonen, Kyle G., Raju, Sitaram C. V.
• Primary Examiner(s): Lintz, Paul R.
• Assistant Examiner(s): Colbert, Ella
• Application Number:
  US08/932,714
• Time in Patent Office: 979 Days
• Field of Search: 707/3, 707/100, 707/104, 707/200, 707/201, 707/203, 707/202, 707/204, 364/282.1, 364/283.1, 364/283.3
• US Class Current: 1/1
• CPC Class Codes: G06F 16/10 File systems; File servers
• CPC Class Codes: Y10S 707/99933 Query processing, i.e. searching
• CPC Class Codes: Y10S 707/99952 Coherency, e.g. same view to multiple users
• CPC Class Codes: Y10S 707/99953 Recoverability

Abstract

A method and system for improved monitoring of document changes in a search engine by an indexing program. Once an indexing or other such monitoring program is halted, upon restart the monitoring program needs to update its own files and its indexes to reflect document changes that occurred while halted. A file system such as the Windows NT file system persistently logs document change information on disk in a monotonically increasing, uniquely-numbered persistent record, which further identifies the file that has changed. The method and system utilize the logged change information to efficiently maintain the indexes, and to rapidly update the indexes after a shutdown and subsequent restart.

Claims

1. In a computer system having a file system of documents, wherein the file system stores document change information in a persistent log of records, a method of updating at least one other file of document information, comprising the steps of, receiving notification of an addition to the persistent log, reading a record from the persistent log, processing the record to obtain change information about a document and a sequence number representative of the position of the document change information relative to other document change information, updating the other file based on the change information of the document, and recording the sequence number of the record in a persistent storage.

2. The method of claim 1 wherein the step of processing the record includes the steps of writing the record to a memory, periodically serializing the record into a persistent data structure and reading the record from the persistent data structure.

3. The method of claim 1 wherein the step of processing the record includes the steps of writing the record to a memory, periodically serializing the record into a cache, structure and reading the record from the cache.

4. The method of claim 1 wherein the records are serially added to the log file and the sequence number corresponds to a location of each record in the log file.

5. The method of claim 1 wherein the step of updating the other file includes the steps of obtaining a document identifier and the change information from the record that was read from the persistent data structure, providing the document identifier and the change information to a search engine for updating the other file, and removing the record from the persistent data storage.

6. The method of claim 1 further comprising the steps of retrieving the sequence number of the record from the persistent storage, and reading a record from the persistent log based on the sequence number retrieved.

7. The method of claim 6 further comprising the steps of reading at least one other record from the persistent log that is later in sequence than the record corresponding to the retrieved sequence number, and processing that record to obtain change information about a document corresponding thereto.

8. The method of claim 1 further comprising the steps of receiving directory information from a user, reading a plurality of sequential records from the log file, filtering the plurality of records based on the directory information to obtain a subset of records, and processing the subset of records to obtain change information about a plurality of documents and sequence numbers representative of the location of the records in the log file.

9. The method of claim 1 further comprising the step of waiting for the notification.

10. The method of claim 9 wherein the step of waiting for the notification of the new record comprises the step of polling for the notification.

11. The method of claim 9 wherein the step of waiting for the notification of the new record comprises the step of awaiting the receipt of a signal indicative of the new record.

12. The method of claim 1 wherein the other file is an index containing document properties, and wherein the step of updating the other file includes the step of modifying the index based on a changed property of the document.

13. The method of claim 1 wherein the documents are stored on a plurality of volumes, wherein the file system serially stores document change information in a plurality of persistent logs of records, with each log corresponding to a volume, and further comprising the step of attaching a volume identifier to each record read from one of the logs.

14. In a computer system having a file system of documents, wherein the file system stores information about changed documents in a persistent log of records, and wherein each record contains a document identifier corresponding to the document, a system for updating at least one index which includes document properties, comprising, means for receiving a notification of a new record in the log, an interface for reading the new record from the persistent log, means for processing the record to obtain the document identifier and change information about the document corresponding thereto, and a search engine for updating the index based on the document identifier and the change information.

15. The system of claim 14 wherein each record further includes a sequence number indicative of the relative position of the record in the log, and wherein the means for processing the record includes means for obtaining the sequence number therefrom, and further comprising a persistent data structure for recording the sequence number of the new record.

16. The system of claim 15 further comprising means for retrieving the sequence number from the persistent data structure, and means for providing a value based on that sequence number to the interface for reading a record corresponding to the value.

17. The system of claim 15 wherein the documents are stored on a plurality of separate volumes and wherein the file system serially stores the document change information in a plurality of persistent logs of records, with each log corresponding to a volume, further comprising means for attaching a volume identifier to each record read from one of the logs, and wherein the persistent data structure for recording the sequence number of the new record includes a plurality of locations for recording a sequence number for each separate volume.

18. The system of claim 15 further comprising at least one other file system that does not store information about changed documents in a persistent log of records, and further comprising means for receiving other change notifications from the other file system and means for updating another index based on the other change notifications.

19. The system of claim 14 further comprising a memory for storing the record read from the log, means for serializing the record into a persistent data structure, and wherein the means for processing the record includes means for reading the persistent data structure.

20. The system of claim 14 wherein the means for receiving a notification includes means for receiving a signal from the file system.

21. The system of claim 14 wherein the means for receiving a notification includes a polling mechanism.

22. In a computer system having a file system of documents, wherein the file system stores document change information in a persistent log of records, a method of updating at least one index of document properties with document information, comprising the steps of, reading a record from the persistent log, processing the record to obtain change information about a document and a sequence number representative of the relative position of the record in the log file, updating the index based on the change information of the document by modifying the index based on a changed property of the document, and recording the sequence number of the record in a persistent data structure.

23. The method of claim 22 further comprising the steps of retrieving the sequence number of the record from the persistent data structure, and reading a record from the persistent log based on the sequence number retrieved.

24. The method of claim 23 further comprising the steps of reading at least one other record from the persistent log that is later in sequence than the record based on to the retrieved sequence number, and processing the other record to obtain change information about a document corresponding thereto.

25. The method of claim 22 further comprising the steps of receiving directory information from a user, reading a plurality of sequential records from the log file, filtering the plurality of records based on the directory information to obtain a subset of records, processing the subset of records to obtain change information about a plurality of documents and sequence numbers representative of the location of the records in the log file, updating the index based on the change information of the documents, and recording the sequence number of the record having the highest sequence number in a persistent data structure.

26. The method of claim 22 further comprising the steps of waiting for a notification of a new record added to the persistent log and receiving the notification.

27. The method of claim 26 wherein the step of waiting for the notification of the new record comprises the step of polling the file system for the notification.

28. The method of claim 26 wherein the step of waiting for the notification of the new record comprises the step of awaiting the receipt of a signal from the file system.

29. The method of claim 22 wherein the step of processing the record includes the steps of writing the record to a memory, periodically serializing the record into a persistent data structure and reading the record from the persistent data structure.

30. The method of claim 29 wherein the step of updating the index includes the steps of obtaining a document identifier and the change information from the record that was read from the persistent data structure, providing the document identifier and the change information to a search engine for updating the indexes, and removing the record from the persistent data structure.

31. The method of claim 22 wherein the documents are stored on a plurality of volumes and wherein the file system serially stores document change information in a plurality of persistent logs of records, with each log corresponding to a volume, further comprising the step of attaching a volume identifier to each record read from one of the logs, the volume identifier corresponding to the volume of that log, and wherein the step of recording the sequence number of the record in a persistent data structure includes the step of recording the sequence number in association with the volume identifier.

32. The method of claim 22 further comprising the steps of detecting an error when reading a record, and in response, determining a set of documents indexed in the index, selecting a tree of documents in the file system and selecting each document in that tree, determining if each selected document is indexed in the index, and if not, indexing that document in the index, determining if a document is indexed in the index but is not in the tree and if so, removing the indexing of that document from the index, and if a selected document is in the tree and is indexed in the index, determining if the document in the tree is synchronized with the document in the index, and if not, re-indexing that document.

33. The method of claim 22 further comprising the steps of detecting a corrupt index, and reconstructing the index by scanning a tree of documents in the file system.

34. In a computer system having a source of changeable data wherein change information about changes to the changeable data are provided in a persistent log of records, a method of updating a data structure, comprising the steps of, receiving notification of a change to the persistent log, reading a record from the persistent log, processing the record to obtain data change information and a sequence number representative of the position of that change information relative to other change information, updating the data structure based on the change information of the document, and recording the sequence number of the record in a persistent storage.

35. The method of claim 34 wherein the step of processing the record includes the steps of writing the record to a memory, periodically serializing the record into a persistent data structure and reading the record from the persistent data structure.

36. The method of claim 34 wherein the source of data is a file system.

37. The method of claim 34 wherein the source of data is a database.

38. The method of claim 34 wherein the data structure is a file.

39. The method of claim 34 wherein the data structure is an index.

40. The method of claim 39 wherein the step of updating the other file includes the steps of obtaining a document identifier and the change information from the record that was read from the persistent data structure, providing the document identifier and the change information to a search engine for updating the index, and removing the record from the persistent data storage.

41. In a computer system having a file system of documents, wherein the file system stores document change information in a persistent log of records, a method of updating at least one index comprising document properties, comprising, reading a record from the persistent log, processing the record to obtain change information about a document and a sequence number representative of the position of the document change information relative to other document change information, updating the index based on the change information of the document by modifying the index based on a changed property of the document, and recording the sequence number of the record in a persistent storage.

42. In a computer system having a file system of documents stored on a plurality of volumes, wherein the file system serially stores document change information in a plurality of persistent logs of records with each log corresponding to a volume, a method of updating at least one other file of document information, comprising, reading a record from one of the persistent logs, attaching a volume identifier to the record read, processing the record to obtain change information about a document and a sequence number representative of the position of the document change information relative to other document change information, updating the other file based on the change information of the document, and recording the sequence number of the record in a persistent storage.

43. A computer-readable medium having computer-executable instructions for performing steps to update an index, wherein change information about changes to a source of changeable data are maintained in a persistent log, the steps comprising:

reading a set of change information from the persistent log;

processing the set of change information to obtain data change information and a sequence number representative of the position of the set of change information relative to other change information;

updating the index based on the change information of the document, including obtaining a document identifier and the change information from the set that was read from the persistent data structure, providing the document identifier and the change information to a search engine for updating the index, and removing the set of change information from the persistent data storage; and

recording the sequence number of the set in a persistent storage.

44. In a computer system having a file system of documents, wherein the file system stores document change information in a persistent log of records, a method of updating at least one other file of document information, comprising, reading a record corresponding to a document from the persistent log, appending the record to a queue independent of whether the queue has another corresponding record queued for the document, processing the record to obtain change information about the document and a sequence number representative of the position of the document change information relative to other document change information, and updating the other file based on the change information of the document.