Answer category data classifying using dynamic thresholds

• US 9,720,963 B2
• Filed: 12/15/2014
• Issued: 08/01/2017
• Est. Priority Date: 11/05/2014
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Barker, Kevin S., DeLima, Roberto, Eggebraaten, Thomas J., Megerian, Mark G., Setnes, Marie L.
• Primary Examiner(s): VO, TRUONG V
• Application Number:
  US14/570,436
• Publication Number:
  US 20160124963A1
• Time in Patent Office: 960 Days
• CPC Class Codes: G06F 16/24522 Translation of natural language queries to structured queries
• CPC Class Codes: G06F 16/24575 using context
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06F 16/284 Relational databases
• CPC Class Codes: G06F 16/285 Clustering or classification
• CPC Class Codes: G06F 16/3329 Natural language query formulation or dialogue systems
• CPC Class Codes: G06F 16/3332 Query translation
• CPC Class Codes: G06F 16/3334 Selection or weighting of terms from queries, including natural language queries
• CPC Class Codes: G06F 16/3344 using natural language analysis
• CPC Class Codes: G06F 16/367 Ontology
• CPC Class Codes: G06F 16/90324 using system suggestions
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: G06F 16/9538 Presentation of query results
• CPC Class Codes: G06F 16/955 using information identifiers, e.g. uniform resource locators [URL]
• CPC Class Codes: G06F 40/205 Parsing
• CPC Class Codes: G06F 40/211 Syntactic parsing, e.g. based on context-free grammar [CFG] or unification grammars
• CPC Class Codes: G06F 40/284 Lexical analysis, e.g. tokenisation or collocates
• CPC Class Codes: G06F 40/30 Semantic analysis
• CPC Class Codes: G06F 40/40 Processing or translation of natural language natural language analysis G06F40/20; semantic analysis G06F40/30
• CPC Class Codes: G06N 20/00 : Machine learning
• CPC Class Codes: G06N 5/022 : Knowledge engineering; Knowledge acquisition
• CPC Class Codes: G06N 5/046 : Forward inferencing; Production systems
• CPC Class Codes: G06N 7/01 : Probabilistic graphical models, e.g. probabilistic networks
• CPC Class Codes: G06Q 30/0203 : Market surveys; Market polls
• CPC Class Codes: G09B 7/00 : Electrically-operated teaching apparatus or devices working with questions and answers mechanically operated G09B3/00; computing arrangements G06F
• CPC Class Codes: G09B 7/02 : of the type wherein the student is expected to construct an answer to the question which is presented or wherein the machine gives an answer to the question presented by a student
• CPC Class Codes: G16H 10/20 : for electronic clinical trials or questionnaires
• CPC Class Codes: G16H 40/20 : for the management or administration of healthcare resources or facilities, e.g. managing hospital staff or surgery rooms
• CPC Class Codes: G16H 50/70 : for mining of medical data, e.g. analysing previous cases of other patients
• CPC Class Codes: G16Z 99/00 : Subject matter not provided for in other main groups of this subclass

Abstract

Managing confidence data in a question-answering environment is disclosed. Managing confidence data can include sorting, based on a set of answer categories for a subject matter, a first set of a plurality of answers into a first answer category. The first set can correspond to at least one of a third set of a plurality of confidence scores and the second set can correspond to at least one of a fourth set of the plurality of confidence scores. Managing confidence data can include classifying confidence scores of the third set into one of a plurality of confidence buckets using a first threshold and determining a fifth set of a plurality of thresholds using the plurality of confidence scores. Managing confidence data can include classifying unclassified confidence scores of the third set into one of the plurality of confidence buckets using the fifth set of the plurality of thresholds.

Claims

1. A system for managing confidence data in a question-answering environment, the system comprising:

a processor; and

a computer readable storage medium having program instructions embodied therewith, the program instructions executable by the processor to cause the system to;

sort, based on a set of answer categories for a subject matter, a first set of a plurality of answers into a first answer category and a second set of the plurality of answers into a second answer category, each of the first set corresponding to at least one of a third set of a plurality of confidence scores and each of the second set corresponding to at least one of a fourth set of the plurality of confidence scores, the plurality of confidence scores representing confidence of answers to a query submitted to a question-answering system;

classify confidence scores of the third set into one of a plurality of confidence buckets using a first threshold;

determine a fifth set of a plurality of thresholds using the plurality of confidence scores; and

classify unclassified confidence scores of the third set into one of the plurality of confidence buckets using the fifth set of the plurality of thresholds.

2. The system of claim 1, wherein the program instructions executable by the processor further cause the system to:

classify confidence scores of the fourth set into one of the plurality of confidence buckets using the first threshold;

determine a sixth set of a plurality of thresholds using the plurality of confidence scores; and

classify unclassified confidence scores of the fourth set into one of the plurality of confidence buckets using the sixth set of the plurality of thresholds.

3. The system of claim 1, wherein the program instructions executable by the processor further cause the system to:

present, via the question-answering system, the first set of the plurality of answers corresponding to the third set of the plurality of confidence scores, in accordance with classifying of the third set of the plurality of confidence scores into the plurality of confidence buckets.

4. The system of claim 1, wherein:

the first threshold is a static threshold, and wherein the program instructions executable by the processor to cause the system to classify the third set of the plurality of confidence scores into one of the plurality of confidence buckets in accordance with the first threshold includes causing the system to;

determine that a portion of the third set of the plurality of confidence achieves the static threshold; and

classify the portion of the third set of the plurality of confidence scores into one of the plurality of confidence buckets in response to determining that the portion of the third set achieves the static threshold.

5. The system of claim 1, wherein:

the program instructions executable by the processor to cause the system to determine the fifth set of the plurality of thresholds using the plurality of confidence scores includes causing the system to;

determine the fifth set of the plurality of thresholds using the third set of the plurality of confidence scores.

6. The system of claim 5, wherein:

the program instructions executable by the processor to cause the system to determine the fifth set of a plurality of thresholds using the third set of the plurality of confidence scores includes causing the system to;

determine a plurality of gaps, each gap of the plurality of gaps located between consecutive confidence scores of the third set of the plurality of confidence scores;

determine a standard deviation associated with the plurality of gaps;

determine that a portion of the plurality of gaps is greater than or equal to the standard deviation; and

using the portion of the plurality of gaps as the fifth set of the plurality of thresholds in response to determining that the portion of the plurality of gaps is greater than or equal to the standard deviation.

7. The system of claim 5, wherein:

the program instructions executable by the processor to cause the system to determine the fifth set of the plurality of thresholds using the third set of the plurality of confidence scores includes causing the system to;

determine a plurality of rate changes, wherein each rate change of the plurality of rate changes is a rate change between consecutive confidence scores of the third set of the plurality of confidence scores;

determine a portion of the plurality of rate changes to be a largest of the plurality of rate changes; and

use the portion of the plurality of rate changes as the fifth set of the plurality of thresholds.

8. The system of claim 1, wherein the program instructions executable by the processor further cause the system to:

determine that one of the plurality of confidence buckets includes a number of confidence scores that achieves a second threshold;

determine a sixth set of a plurality of thresholds using the number of confidence scores; and

classify a portion of the number of confidence scores into one of the plurality of confidence buckets using the sixth set of the plurality of thresholds.

9. A computer program product for managing confidence data in a question-answering environment, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a computer to cause the computer to perform a method comprising:

sorting, based on a set of answer categories for a subject matter, a first set of a plurality of answers into a first answer category and a second set of the plurality of answers into a second answer category, each of the first set corresponding to at least one of a third set of a plurality of confidence scores and each of the second set corresponding to at least one of a fourth set of the plurality of confidence scores, the plurality of confidence scores representing confidence of answers to a query submitted to a question-answering system;

classifying confidence scores of the third set into one of a plurality of confidence buckets using a first threshold;

determining a fifth set of a plurality of thresholds using the plurality of confidence scores; and

classifying unclassified confidence scores of the third set into one of the plurality of confidence buckets using the fifth set of the plurality of thresholds.

10. The computer program product of claim 9, wherein the method further includes:

classifying confidence scores of the fourth set into one of the plurality of confidence buckets using the first threshold;

determining a sixth set of a plurality of thresholds using the plurality of confidence scores; and

classifying unclassified confidence scores of the fourth set into one of the plurality of confidence buckets using the sixth set of the plurality of thresholds.

11. The computer program product of claim 9, wherein the method further includes:

presenting, via the question-answering system, the first set of the plurality of answers corresponding to the third set of the plurality of confidence scores, in accordance with the classifying of the third set of the plurality of confidence scores into the plurality of confidence buckets.

12. The computer program product of claim 9, wherein:

the first threshold is a static threshold, and wherein classifying the third set of the plurality of confidence scores into one of the plurality of confidence buckets in accordance with the first threshold includes;

determining that a portion of the third set of the plurality of confidence achieves the static threshold; and

classifying the portion of the third set of the plurality of confidence scores into one of the plurality of confidence buckets in response to determining that the portion of the third set achieves the static threshold.