Method and apparatus for document filtering using ensemble filters

• US 20040158569A1
• Filed: 11/14/2003
• Published: 08/12/2004
• Est. Priority Date: 11/15/2002
• Status: Active Grant

• Current Assignee: Justsystems Evans Research Incorporated
• Original Assignee: Justsystems Evans Research Incorporated
• Inventors: Evans, David A., Bennett, Jeffrey, Shanahan, James G., Sheftel, Victor M., Montgomery, Jesse
• Granted Patent: US 7,426,509 B2
• Time in Patent Office: Days
• US Class Current: 707/100
• CPC Class Codes: G06F 16/337 Profile generation, learning or modification
• CPC Class Codes: Y10S 707/99933 Query processing, i.e. searching
• CPC Class Codes: Y10S 707/99935 Query augmenting and refining, e.g. inexact access
• CPC Class Codes: Y10S 707/99936 Pattern matching access

Abstract

A technique for representing an information need and employing one or more filters to select documents that satisfy the represented information need, including a technique of creating filters that involves (a) dividing a set of documents into one or more subsets such that each subset can be used as the source of features for creating a filtering profile or used to set or validate the score threshold for the profile and (b) determining whether multiple profiles are required and how to combine them to create an effective filter. Multiple profiles can be incorporated into an individual filter and the individual filters combined to create an ensemble filter. Ensemble filters can then be further combined to create meta filters.

Claims

1. A process for creating an ensemble filter for selecting documents, comprising:

identifying a set of documents for training;

identifying a first coherent set of documents from said training set of documents;

identifying a first profile corresponding to said first coherent set of documents;

identifying a second coherent set of documents and a remainder set of documents from said training set of documents using said first profile;

identifying at least one coherent set of documents from said remainder set of documents;

identifying at least one remainder profile corresponding to each of said identified coherent sets of documents from said remainder set of documents;

creating a first sub-filter using said first profile;

creating at least one remainder sub-filter using at least one of said remainder profiles; and

combining said first sub-filter with at least one remainder sub-filter to create an ensemble filter.

2. A process, as in claim 1, further comprising:

clustering said training set of documents to identify said first coherent set of documents.

3. A process, as in claim 1, further comprising:

clustering said training set of documents and selecting said largest cluster to identify said first coherent set of documents.

4. A process, as in claim 1, further comprising:

cascading said first sub-filter and at least one remainder sub-filter to create at least part of said ensemble filter.

5. A process, as in claim 1, further comprising:

mutiplexing said first sub-filter with at least one remainder sub-filter to create at least part of said ensemble filter.

6. A process, as in claim 2, further comprising:

cascading said first sub-filter and at least one remainder sub-filter to create at least part of said ensemble filter.

7. A process, as in claim 3, further comprising:

cascading said first sub-filter and at least one remainder sub-filter to create at least part of said ensemble filter.

8. A process, as in claim 2, further comprising:

mutiplexing said first sub-filter with at least one remainder sub-filter to create at least part of said ensemble filter.

9. A process, as in claim 3, further comprising:

mutiplexing said first sub-filter with at least one remainder sub-filter to create at least part of said ensemble filter.

10. A process for selecting documents from a stream of documents, comprising:

identifying a set of documents for training;

identifying a first coherent set of documents from said training set of documents;

identifying a first profile corresponding to said first coherent set of documents;

identifying a second coherent set of documents and a remainder set of documents from said training set of documents using said first profile;

identifying at least one coherent set of documents from said remainder set of documents;

identifying at least one remainder profile corresponding to each of said identified coherent sets of documents from said remainder set of documents;

creating a first sub-filter using said first profile;

creating at least one remainder sub-filter using at least one of said remainder profiles;

combining said first sub-filter with at least one remainder sub-filter to create an ensemble filter; and

passing said stream of documents through said ensemble filter.

11. A process, as in claim 10, further comprising:

clustering said training set of documents to identify said first coherent set of documents.

12. A process, as in claim 10, further comprising:

clustering said training set of documents and selecting said largest cluster to identify said first coherent set of documents.

13. A process, as in claim 10, further comprising:

cascading said first sub-filter and at least one remainder sub-filter to create at least part of said ensemble filter.

14. A process, as in claim 10, further comprising:

mutiplexing said first sub-filter with at least one remainder sub-filter to create at least part of said ensemble filter.

15. A process, as in claim 11, further comprising:

cascading said first sub-filter and at least one remainder sub-filter to create at least part of said ensemble filter.

16. A process, as in claim 12, further comprising:

cascading said first sub-filter and at least one remainder sub-filter to create at least part of said ensemble filter.

17. A process, as in claim 11, further comprising:

mutiplexing said first sub-filter with at least one remainder sub-filter to create at least part of said ensemble filter.

18. A process, as in claim 12, further comprising:

mutiplexing said first sub-filter with at least one remainder sub-filter to create at least part of said ensemble filter.

19. A process for selecting documents from a database of documents, comprising:

identifying a set of documents for training;

identifying a first coherent set of documents from said training set of documents;

identifying a first profile corresponding to said first coherent set of documents;

identifying a second coherent set of documents and a remainder set of documents from said training set of documents using said first profile;

identifying at least one coherent set of documents from said remainder set of documents;

identifying at least one remainder profile corresponding to each of said identified coherent sets of documents from said remainder set of documents;

creating a first sub-filter using said first profile;

creating at least one remainder sub-filter using at least one of said remainder profiles;

combining said first sub-filter with at least one remainder sub-filter to create an ensemble filter; and

applying said ensemble filter to said database to select documents.

20. A process, as in claim 19, further comprising:

clustering said training set of documents to identify said first coherent set of documents.

21. A process, as in claim 19, further comprising:

clustering said training set of documents and selecting said largest cluster to identify said first coherent set of documents.

22. A process, as in claim 19, further comprising:

cascading said first sub-filter and at least one remainder sub-filter to create at least part of said ensemble filter.

23. A process, as in claim 19, further comprising:

mutiplexing said first sub-filter with at least one remainder sub-filter to create at least part of said ensemble filter.

24. A process, as in claim 20, further comprising:

cascading said first sub-filter and at least one remainder sub-filter to create at least part of said ensemble filter.

25. A process, as in claim 21, further comprising:

cascading said first sub-filter and at least one remainder sub-filter to create at least part of said ensemble filter.

26. A process, as in claim 20, further comprising:

mutiplexing said first sub-filter with at least one remainder sub-filter to create at least part of said ensemble filter.

27. A process, as in claim 21, further comprising:

mutiplexing said first sub-filter with at least one remainder sub-filter to create at least part of said ensemble filter.