Document tagging and retrieval using per-subject dictionaries including subject-determining-power scores for entries

• US 9,971,828 B2
• Filed: 10/20/2015
• Issued: 05/15/2018
• Est. Priority Date: 05/10/2013
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Gattiker, Anne Elizabeth, Gebara, Fadi H., Hylick, Anthony N., Kanj, Rouwaida N., Li, Jian
• Primary Examiner(s): Fan, Shiow-Jy
• Application Number:
  US14/887,495
• Publication Number:
  US 20160041975A1
• Time in Patent Office: 938 Days
• CPC Class Codes: G06F 16/245 Query processing
• CPC Class Codes: G06F 16/313 Selection or weighting of terms for indexing
• CPC Class Codes: G06F 16/93 Document management systems
• CPC Class Codes: G06F 40/242 Dictionaries

Abstract

Techniques for managing big data include tagging of documents and subsequent retrieval using per-subject dictionaries having entries with subject-determining-power scores. The subject-determining-power scores provide an indication of the descriptive power of the term with respect to the subject of the dictionary containing the term. The same term may have entries in multiple dictionaries with different subject-determining-power scores in each of the dictionaries. A retrieval request for one or more documents containing search terms descriptive of the one or more documents can be processed identifying a set of candidate documents tagged with subjects and optional terms, and then applying subject-determining-power scores from the multiple dictionaries for the search term to determine a subject for the search term. The method then selects the one or more documents from the candidate documents according to the subject.

Claims

1. A computer-performed method of organizing a collection of electronic documents, the method comprising:

in a computer system, storing entries in multiple dictionaries separate from and not associated with any particular one of the electronic documents, wherein the multiple dictionaries are data structures within the computer system, wherein individual ones of the multiple dictionaries correspond to one of a plurality of different subjects, wherein the entries contain a descriptive term and a subject-determining-power score corresponding to the descriptive term, wherein an individual subject-determining-power score indicates the relative strength or weakness of the corresponding descriptive term with respect to the subject of the one of the multiple dictionaries containing an entry in which the descriptive term is stored, and wherein at least some of the descriptive terms are present in two or more of the multiple dictionaries;

responsive to requests within the computer system to identify one or more of the electronic documents, wherein the requests include at least one search term descriptive of the one or more electronic documents, accessing the collection of electronic documents by matching the at least one search term with descriptive terms in the multiple dictionaries to determine one or more subjects of the request from subjects of one or more of the multiple dictionaries that contain the descriptive terms matching the at least one search term, and applying the subject-determining power scores to determine which of the subjects of the one or more of the multiple dictionaries that contain the descriptive terms that match the at least one search term are most applicable to the request;

ranking the one or more subjects determined by the matching according to a match score computed for individual ones of the one or more subjects with respect to the at least one search term;

until a predetermined number of documents are identified, collecting document tags having a best match to the highest-ranking subject for which the document tags have not yet been collected, in a collected set of document tags for the request, wherein the collecting document tags collects the document tags from a tag database separate from the documents and the dictionaries, whereby a speed of matching the documents to the one or more subjects is increased, wherein the document tags include for each of one or more subject entries in the document tags, multiple tag terms with subject-power determining scores corresponding to the tag terms and a confidence score, wherein the collecting document tags further determines the best-match to the highest-ranking subject for which the document tags have not been collected by multiplying all of the tag terms in the entry corresponding to the highest-ranking subject for which the document tags have not been collected by the confidence score of the entry to generate a document subject match score, and compare the computed document subject match scores to determine the best match; and

storing a representation of the collected set of document tags that identify the electronic documents in a memory of the computer system to provide the response to the request to identify the one or more electronic documents.

2. The method of claim 1, wherein the collecting further matches the at least one search term with terms contained within the tags, and wherein the collecting further selects electronic documents having associated tags matching the search term.

3. The method of claim 1, wherein the one or more subjects comprises multiple subjects, and wherein the collecting reduces the number of the one or more electronic documents according to values of the subject-determining-power scores for the multiple subjects to refine the response to the request.

4. The method of claim 1, further comprising determining a context for the request, wherein the context is information external to the at least one search term, wherein the tags contain context-dependent information, and wherein the collecting further matches the context with the context-dependent information stored in the tags to obtain the collected set of electronic documents.

5. The method of claim 4, wherein the matching further selects from among multiple context-dependent sets of entries in the one or more of the multiple dictionaries according to a context provided with the request, whereby the applying applies different subject-determining-power scores from the one or more of the multiple dictionaries for the at least one search term according to the context.

6. The method of claim 1, further comprising tagging electronic documents within the collection of electronic documents, the tagging comprising:

finding terms within the electronic documents to be tagged;

matching the terms with the descriptive terms within entries in the multiple dictionaries to determine a ranking of the multiple dictionaries by relative relation of their associated subjects to the content of the electronic documents to be tagged; and

storing a result of the matching for subsequent retrieval when performing electronic document searches.

7. A computer system comprising:

a processor for executing program instructions; and

a memory for storing the program instructions, wherein the program instructions are program instructions for organizing a collection of electronic documents, and wherein the program instructions comprise program instructions that store entries in multiple dictionaries separate from and not associated with any particular one of the electronic documents, wherein the multiple dictionaries are data structures within the computer system, wherein individual ones of the multiple dictionaries correspond to one of a plurality of different subjects, wherein the entries contain a descriptive term and a subject-determining-power score corresponding to the descriptive term, wherein an individual subject-determining-power score indicates the relative strength or weakness of the corresponding descriptive term with respect to the subject of the one of the multiple dictionaries containing an entry in which the descriptive term is stored, and wherein at least some of the descriptive terms are present in two or more of the multiple dictionaries, program instructions that, responsive to requests to identify one or more of the electronic documents, wherein the requests include at least one search term descriptive of the one or more electronic documents, access the collection of electronic documents by matching the at least one search term with descriptive terms in the multiple dictionaries to determine one or more subjects of the request from subjects of one or more of the multiple dictionaries that contain the descriptive terms matching the at least one search term, and applying the subject-determining power scores to determine which of the subjects of the one or more of the multiple dictionaries that contain the descriptive terms that match the at least one search term are most applicable to the request, program instructions that rank the one or more subjects determined by the matching according to a match score computed for individual ones of the one or more subjects with respect to the at least one search term, program instructions that collect document tags having a best match to the highest-ranking subject for which the document tags have not yet been collected, in a collected set of document tags for the request, wherein the program instructions for collecting document tags collect the document tags from a tag database separate from the documents and the dictionaries, whereby a speed of matching the documents to the one or more subjects is increased, until a predetermined number of documents are identified, wherein the document tags include for each of one or more subject entries in the document tags, multiple tag terms with subject-power determining scores corresponding to the tag terms and a confidence score, wherein the program instructions that collect document tags further determine the best-match to the highest-ranking subject for which the document tags have not been collected by multiplying all of the tag terms in the entry corresponding to the highest-ranking subject for which the document tags have not been collected by the confidence score of the entry to generate a document subject match score, and compare the computed document subject match scores to determine the best match, and program instructions that store a representation of the collected set of document tags that identify the electronic documents in a memory of the computer system to provide the response to the request to identify the one or more electronic documents.

8. The computer system of claim 7, wherein the program instructions that collect further match the at least one search term with terms contained within the tags, and wherein the program instructions that collect further select electronic documents having associated tags matching the search term.

9. The computer system of claim 7, wherein the one or more subjects comprises multiple subjects, and wherein the program instructions that collect reduce the number of the one or more electronic documents according to values of the subject-determining-power scores for the multiple subjects to refine the response to the request.

10. The computer system of claim 7, wherein the program instructions further comprise program instructions that determine a context for the request, wherein the context is information external to the at least one search term, wherein the tags contain context-dependent information, and wherein the program instructions that collect further match the context with the context-dependent information stored in the tags to obtain the collected set of electronic documents.

11. The computer system of claim 10, wherein the program instructions that access further select from among multiple context-dependent sets of entries in the one or more of the multiple dictionaries according to a context provided with the request, whereby the applying applies different subject-determining-power scores from the one or more of the multiple dictionaries for the at least one search term according to the context.

12. The computer system of claim 7, further comprising program instructions that tag electronic documents within the collection of electronic documents, the program instructions for tagging comprising:

program instructions that find terms within the electronic documents to be tagged;

program instructions that match the terms with the descriptive terms within entries in the multiple dictionaries to determine a ranking of the multiple dictionaries by relative relation of their associated subjects to the content of the electronic documents to be tagged; and

program instructions that store a result of the matching for subsequent retrieval when performing electronic document searches.

13. A computer program product comprising a computer-readable storage device storing program instructions for organizing a collection of electronic documents, wherein the program instructions comprise program instructions for:

in a computer system, storing entries in multiple dictionaries separate from and not associated with any particular one of the electronic documents, wherein the multiple dictionaries are data structures within the computer system, wherein individual ones of the multiple dictionaries correspond to one of a plurality of different subjects, wherein the entries contain a descriptive term and a subject-determining-power score corresponding to the descriptive term, wherein an individual subject-determining-power score indicates the relative strength or weakness of the corresponding descriptive term with respect to the subject of the one of the multiple dictionaries containing an entry in which the descriptive term is stored, and wherein at least some of the descriptive terms are present in two or more of the multiple dictionaries;

responsive to requests within the computer system to identify one or more of the electronic documents, wherein the requests include at least one search term descriptive of the one or more electronic documents, accessing the collection of electronic documents by matching the at least one search term with descriptive terms in the multiple dictionaries to determine one or more subjects of the request from subjects of one or more of the multiple dictionaries that contain the descriptive terms matching the at least one search term, and applying the subject-determining power scores to determine which of the subjects of the one or more of the multiple dictionaries that contain the descriptive terms that match the at least one search term are most applicable to the request;

ranking the one or more subjects determined by the matching according to a match score computed for individual ones of the one or more subjects with respect to the at least one search term;

until a predetermined number of documents are identified, collecting document tags having a best match to the highest-ranking subject for which the document tags have not yet been collected, in a collected set of document tags for the request, wherein the collecting document tags collects the document tags from a tag database separate from the documents and the dictionaries, whereby a speed of matching the documents to the one or more subjects is increased, wherein the document tags include for each of one or more subject entries in the document tags, multiple tag terms with subject-power determining scores corresponding to the tag terms and a confidence score, wherein the program instructions for collecting document tags further determine the best-match to the highest-ranking subject for which the document tags have not been collected by multiplying all of the tag terms in the entry corresponding to the highest-ranking subject for which the document tags have not been collected by the confidence score of the entry to generate a document subject match score, and compare the computed document subject match scores to determine the best match; and

storing a representation of the collected set of document tags that identify the electronic documents in a memory of the computer system to provide the response to the request to identify the one or more electronic documents.

14. The computer program product of claim 13, wherein the one or more subjects comprise multiple subjects, and wherein the program instructions for collecting reduce the number of the one or more electronic documents according to values of the subject-determining-power scores for the multiple subjects to refine the response to the request.

15. The computer program product of claim 13, further comprising program instructions for determining a context for the request, wherein the context is information external to the at least one search term, wherein the tags contain context-dependent information, and wherein the program instructions for collecting further match the context with the context-dependent information stored in the tags to obtain the collected set of electronic documents.

16. The computer program product of claim 13, wherein the program instructions for the first matching further select from among multiple context-dependent sets of entries in the one or more of the multiple dictionaries according to a context provided with the request, whereby the applying applies different subject-determining-power scores from the one or more of the multiple dictionaries for the at least one search term according to the context.

17. The computer program product of claim 13, further comprising program instructions for tagging electronic documents within the collection of electronic documents, the tagging comprising program instructions for:

finding terms within the electronic documents to be tagged;

matching the terms with the descriptive terms within entries in the multiple dictionaries to determine a ranking of the multiple dictionaries by relative relation of their associated subjects to the content of the electronic documents to be tagged; and

storing a result of the matching for subsequent retrieval when performing electronic document searches.