Fast adaptive document filtering

• US 7,516,146 B2
• Filed: 05/15/2003
• Issued: 04/07/2009
• Est. Priority Date: 05/15/2003
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Walker, Stephen, Zobel, Justin, Robertson, Stephen
• Primary Examiner(s): Breene; John E
• Assistant Examiner(s): COLAN, GIOVANNA B
• Application Number:
  US10/438,606
• Publication Number:
  US 20040230598A1
• Time in Patent Office: 2,154 Days
• Field of Search: 707/5, 707/526, 707/6, 707/102, 707/200, 707/1
• US Class Current: 1/1
• CPC Class Codes: G06F 16/335 Filtering based on additional data, e.g. user or group profiles filtering in web context G06F16/9535, G06F16/9536
• CPC Class Codes: G06F 16/337 Profile generation, learning or modification
• CPC Class Codes: Y10S 707/99935 Query augmenting and refining, e.g. inexact access
• CPC Class Codes: Y10S 707/99936 Pattern matching access
• CPC Class Codes: Y10S 707/99943 Generating database or data structure, e.g. via user interface

Abstract

Data structures, stored on various types of computer-readable media, include information related to user profiles and/or to various documents. The information included in these data structures is arranged and stored in manner that allows for rapid user profile updating to be performed as new or changed documents are processed in a document filtering system.

Claims

1. A system comprising:

a first computer-readable storage media having stored thereon a reference dictionary file data structure, the reference dictionary file data structure including terms parsed from a plurality of documents stored in a document repository, and terms parsed from a new document received by the system but not stored in the document repository, the plurality of documents and the new document parsed without regard to any user profile, a user profile updated based at least in part on terms from the new document included in the reference dictionary file data structure and feedback from a user regarding relevance of a document received by the user before the system determines whether the new document is relevant to any user, the user profile specifying one or more areas of interest of the user;

the first computer-readable storage media having stored thereon a parsed term data structure, the parsed term data structure including a one or more parsed terms and a term selection value associated with each of the one or more parsed terms, each of the parsed terms either;

present in the reference dictionary file data structure indicating at least one document indicated as relevant to the user contains the term, or

present in an original user profile,

 

at least one of the parsed terms present in an original user profile and not present in the reference dictionary file data, the term selection value used for determining whether the associated term is to be included in the undated user profile; and

a second computer-readable storage media having stored thereon a document dictionary index data structure and the document repository, the document dictionary index data structure including only terms located in the plurality of documents stored in the document repository.

2. A system as defined in claim 1, the feedback from the user inferred from the user'"'"'s action associated with the new document.

3. A system as defined in claim 1, the first computer-readable storage media including main memory and the second computer-readable storage media including mass storage.

4. A system as defined in claim 1, the parsed term data structure further including an original profile indicator associated with each of the parsed terms in the parsed term data structure, each original profile indicator indicating whether a parsed term associated therewith is present in a specified user profile.

5. A system as defined in claim 1, the parsed term data structure further including a term weight associated with each of the parsed terms in the parsed term data structure.

6. A system as defined in claim 1, the parsed term data structure further including a term weight associated with each term in the parsed term data structure, and a term selection value associated with each term in the parsed term data structure, the term selection value a product of the term weight and a total number of relevant documents in which the term appears.

7. A system as defined in claim 1, further comprising a profile index data structure stored on the first computer-readable storage media, the profile index data structure including a profile term field for a document term, an associated profile count field for a value indicating a number of profile records in a profile records list that include the document term, and an associated profile records list of profile records that include the document term.

8. A system as defined in claim 1, further comprising a profile repository stored on the first computer-readable storage media.

9. A system as defined in claim 1, further comprising the parsed term data structure stored on the first computer-readable storage media.

10. A system as defined in claim 1, further comprising a profile repository stored on the first computer-readable storage media.

11. A system as defined in claim 1, further comprising a profile repository stored on the first computer-readable storage media, and a profile index data structure stored on the first computer-readable storage media.

12. A system as defined in claim 1, further comprising an adaptive document filtering module to update the user profile based on information contained in the parsed term data structure.

13. A system as defined in claim 1, further comprising an adaptive document filtering module operable to receive the new document and to update the reference dictionary file data structure to include terms from the received new document.

14. A system as defined in claim 1, further comprising an adaptive document filtering module operable to receive the new document and to update the document dictionary index data structure to include terms from the received new document.

15. A system as defined in claim 1, further comprising an adaptive document filtering module operable to receive the new document and to update the reference dictionary file data structure and the document dictionary index data structure to include terms from the new received document.

16. A system as defined in claim 1, further comprising an adaptive document filtering module operable to receive a stream of new documents and to update the reference dictionary file data structure after the receipt of each new document in the stream of new documents.

17. A system as defined in claim 1, further comprising an adaptive document filtering module operable to receive a stream of new documents, to update the reference dictionary file data structure after the receipt of each new document in the stream of new documents, and to periodically update the document dictionary index data structure.

18. A system as defined in claim 1, further comprising an adaptive document filtering module operable to receive a stream of new documents, to update the reference dictionary file data structure after the receipt of each new document in the stream of new documents, and to create a parsed term data structure using the reference dictionary file data structure.

19. A system as defined in claim 1, wherein the feedback from the user is received directly from the user.

20. A system as defined in claim 1 wherein the user profile is updated based at least in part on terms from the new document included in the reference dictionary file data structure and the feedback from the user regarding relevance of the document received by the user before the system notifies any user about the new document.

21. A system comprising:

a main memory having stored a reference dictionary file data structure, the reference dictionary file data structure including terms parsed from a plurality of documents stored in a document repository and terms parsed from a new document received by the system but not stored in the document repository, the terms parsed from the plurality of documents and the new document without regard to any user profile,

the reference dictionary file data structure including a reference count value indicating a total number of documents that include a term, the total number of documents including documents stored in the document repository and the new document, the reference count value accessible by a user profile update operation for updating a user profile before the system determines whether the new document is relevant to any user,

the terms parsed from the new document added to the reference dictionary file data structure as follows;

when a term is included in the document statistics file data structure but not in the reference dictionary file data structure, the term is added to the reference dictionary file data structure and the reference count value in the reference dictionary file data structure associated with the term is initialized to one;

when a term is included in both the document statistics file data structure and the reference dictionary file data structure, the reference count value in the reference dictionary file data structure associated with the term is incremented; and

a mass storage having stored the document repository and a document index that indexes terms from documents stored in the document repository.

22. The system of claim 21 wherein the reference count value is accessible by the user profile update operation for updating the user profile before the system notifies any user about the new document.

23. The system of claim 21 wherein the new document is stored in the document repository after the terms parsed from the new document are stored in the reference dictionary file data structure.

24. The system of claim 21 wherein terms parsed from new documents received at the system are added to the reference dictionary file data structure each time a new document is received by the system, wherein the new documents are stored in the document repository in a batch process after a plurality of new documents have been received by the system.

25. The system of claim 21 wherein the main memory further includes a document statistics file data structure, wherein the document statistics file data structure includes:

each term parsed from the new document; and

a term frequency value for each term parsed from the new document that indicates a total number of times the term is included in the new document.