Fast adaptive document filtering

• US 8,095,559 B2
• Filed: 02/27/2009
• Issued: 01/10/2012
• Est. Priority Date: 05/15/2003
• Status: Expired

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Robertson, Stephen, Walker, Stephen, Zobel, Justin
• Primary Examiner(s): COLAN, GIOVANNA B
• Application Number:
  US12/394,047
• Publication Number:
  US 20090198683A1
• Time in Patent Office: 1,047 Days
• Field of Search: 707/790, 707/705, 707/609, 707/736, 707/758, 707/791, 707/802, 715/526
• US Class Current: 707/791
• CPC Class Codes: G06F 16/335 Filtering based on additional data, e.g. user or group profiles filtering in web context G06F16/9535, G06F16/9536
• CPC Class Codes: G06F 16/337 Profile generation, learning or modification
• CPC Class Codes: Y10S 707/99935 Query augmenting and refining, e.g. inexact access
• CPC Class Codes: Y10S 707/99936 Pattern matching access
• CPC Class Codes: Y10S 707/99943 Generating database or data structure, e.g. via user interface

Abstract

Data structures, stored on various types of computer-readable media, include information related to user profiles and/or to various documents. The information included in these data structures is arranged and stored in manner that allows for rapid user profile updating to be performed as new or changed documents are processed in a document filtering system.

Claims

1. A computer-readable storage medium having instructions that, when executed by a processor, cause the processor to:

receive a new document;

create a document statistics file that indicates a frequency of a first set of terms in the new document;

determine a profile score using the document statistics file and information associated with a user profile;

notify a user associated with the user profile if the profile score exceeds a profile threshold value;

update a reference dictionary file to include the first set of terms from the new document, the reference dictionary file including a second set of terms located in a plurality of documents stored in a document repository and a third set of terms not located in the plurality of documents stored in the document repository;

update a document dictionary index to include the first set of terms from the new document;

create a parsed term data structure based on the reference dictionary file; and

update the user profile based on the parsed term data structure.

2. A computer-readable storage medium as defined in claim 1, wherein the instructions are executable by the processor to:

store the reference dictionary file on a first computer-readable storage media; and

store the document dictionary index on a second computer-readable storage media.

3. A computer-readable storage medium as defined in claim 1, wherein the reference dictionary file is stored in Random Access Memory (RAM) and the document dictionary index is stored on a mass storage device.

4. A computer-readable storage medium as defined in claim 1, wherein the reference dictionary file includes the second set of terms located in the plurality of documents stored in the document repository and the first set of terms that are located in the new document but not located in the plurality of documents.

5. A computer-readable storage medium as defined in claim 1, wherein the document dictionary index includes the second set of terms located in the plurality of documents.

6. A computer-readable storage medium as defined in claim 5, wherein the parsed term data structure includes a plurality of term weights and a plurality of term selection values, wherein each term weight and term selection value is associated with a particular term in the parsed term data structure.

7. A computer-readable storage medium as defined in claim 1, wherein the instructions are executable by the processor to:

retrieve one or more documents;

retrieve a user profile associated with the user; and

form forming a parsed term data structure including information derived from the one or more documents, the retrieved user profile and the reference dictionary file.

8. A computer-readable storage medium as defined in claim 1, wherein the instructions are executable by the processor to:

retrieve one or more documents specified as relevant to the user;

retrieve the user profile associated with the user;

modify the parsed term data structure including information derived from the one or more documents, the retrieved user profile and the reference dictionary file; and

update the user profile based on the information in the parsed term data structure.

9. A method comprising:

receiving a new document at a document filtering system;

determining a frequency of first terms in the new document;

storing the frequency of the first terms in a document statistics file;

determining a profile score based on the document statistics file and based on a user profile;

in response to determining that the profile score does not satisfy a profile threshold, notifying a user associated with the user profile;

updating a reference dictionary file to include the first terms from the new document, the reference dictionary file including second terms located in a plurality of documents stored in a document repository and including third terms not located in the plurality of documents;

updating a document dictionary index to include the first terms from the new document, the document dictionary index including the second terms located in the plurality of documents;

creating a parsed term data structure including the first terms and the second terms in the reference dictionary file; and

updating the user profile based on the parsed term data structure.

10. A method of claim 9, further comprising:

storing the reference dictionary file in random access memory; and

storing the document dictionary index in a disk drive.

11. A method of claim 9, wherein the reference dictionary file includes the second terms located in the plurality of documents stored in the document repository and the first terms from the new document.

12. A computer-implemented method of claim 9, wherein the parsed term data structure includes a plurality of term weights and a plurality of term selection values, wherein each term weight and term selection value is associated with a separate one of the terms in the parsed term data structure.

13. A computer-implemented method of claim 9, further comprising:

retrieving one or more documents;

retrieving the user profile associated with the user; and

forming a second parsed term data structure based on the one or more documents, the user profile, and the reference dictionary file.

14. A computer-implemented method of claim 9, further comprising:

retrieving one or more documents that the user has indicated as relevant;

retrieving the user profile associated with the user;

creating a second parsed term data structure based on the one or more documents, the retrieved user profile, and the reference dictionary file; and

updating the user profile based on the parsed term data structure.

15. A system comprising:

a processor;

a memory; and

an adaptive document filter module stored in the memory and executable by the processor to;

receive a new document;

create a document statistics file comprising a frequency of a first set of terms included in the new document;

retrieve a user profile;

calculate a profile score using the document statistics file and the user profile;

notify a user associated with the user profile when the profile score satisfies a profile threshold;

update a reference dictionary file to include the first set of terms from the new document, the reference dictionary file including a second set of terms located in a plurality of documents stored in a document repository and a third set of terms not located in the plurality of documents stored in the document repository;

update a document dictionary index to include the first set of terms from the document;

create a parsed term data structure including the first set of terms and the second set of terms; and

update the user profile based on the parsed term data structure.

16. A system of claim 15, wherein the reference dictionary file includes the second set of terms located in the plurality of documents stored in the document repository and includes the first set of terms that are located in the new document but not located in the plurality of documents, and wherein the document dictionary index includes the second set of terms located in the plurality of documents.

17. A system of claim 15, wherein the adaptive document filter module is executable by the processor to modify the parsed term data structure to include a plurality of term weights and a plurality of term selection values, wherein each term weight and term selection value is associated with a particular term in the parsed term data structure.

18. A system of claim 15, wherein the adaptive document filter module is executable by the processor to:

retrieve one or more documents;

retrieve the user profile associated with the user; and

modify the parsed term data structure to include information from the one or more documents, the retrieved user profile, and the reference dictionary file.

19. A system of claim 15, wherein the adaptive document filter module is executable by the processor to:

retrieve one or more documents that the user has specified as relevant;

retrieve the user profile associated with the user;

create a parsed term data structure based on the one or more documents, the user profile, and the reference dictionary file; and

update the user profile based on the information in the parsed term data structure.