Answer category data classifying using dynamic thresholds

• US 9,946,747 B2
• Filed: 05/11/2015
• Issued: 04/17/2018
• Est. Priority Date: 11/05/2014
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Barker, Kevin S., DeLima, Roberto, Eggebraaten, Thomas J., Megerian, Mark G., Setnes, Marie L.
• Primary Examiner(s): Vo, Truong
• Application Number:
  US14/708,536
• Publication Number:
  US 20160125064A1
• Time in Patent Office: 1,072 Days
• CPC Class Codes: G06F 16/24522 Translation of natural language queries to structured queries
• CPC Class Codes: G06F 16/24575 using context
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06F 16/284 Relational databases
• CPC Class Codes: G06F 16/285 Clustering or classification
• CPC Class Codes: G06F 16/3329 Natural language query formulation or dialogue systems
• CPC Class Codes: G06F 16/3332 Query translation
• CPC Class Codes: G06F 16/3334 Selection or weighting of terms from queries, including natural language queries
• CPC Class Codes: G06F 16/3344 using natural language analysis
• CPC Class Codes: G06F 16/367 Ontology
• CPC Class Codes: G06F 16/90324 using system suggestions
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: G06F 16/9538 Presentation of query results
• CPC Class Codes: G06F 16/955 using information identifiers, e.g. uniform resource locators [URL]
• CPC Class Codes: G06F 40/205 Parsing
• CPC Class Codes: G06F 40/211 Syntactic parsing, e.g. based on context-free grammar [CFG] or unification grammars
• CPC Class Codes: G06F 40/284 Lexical analysis, e.g. tokenisation or collocates
• CPC Class Codes: G06F 40/30 Semantic analysis
• CPC Class Codes: G06F 40/40 Processing or translation of natural language natural language analysis G06F40/20; semantic analysis G06F40/30
• CPC Class Codes: G06N 20/00 : Machine learning
• CPC Class Codes: G06N 5/022 : Knowledge engineering; Knowledge acquisition
• CPC Class Codes: G06N 5/046 : Forward inferencing; Production systems
• CPC Class Codes: G06N 7/01 : Probabilistic graphical models, e.g. probabilistic networks
• CPC Class Codes: G06Q 30/0203 : Market surveys; Market polls
• CPC Class Codes: G09B 7/00 : Electrically-operated teaching apparatus or devices working with questions and answers mechanically operated G09B3/00; computing arrangements G06F
• CPC Class Codes: G09B 7/02 : of the type wherein the student is expected to construct an answer to the question which is presented or wherein the machine gives an answer to the question presented by a student
• CPC Class Codes: G16H 10/20 : for electronic clinical trials or questionnaires
• CPC Class Codes: G16H 40/20 : for the management or administration of healthcare resources or facilities, e.g. managing hospital staff or surgery rooms
• CPC Class Codes: G16H 50/70 : for mining of medical data, e.g. analysing previous cases of other patients
• CPC Class Codes: G16Z 99/00 : Subject matter not provided for in other main groups of this subclass

Abstract

Managing confidence data in a question-answering environment is disclosed. Managing confidence data can include sorting, based on a set of answer categories for a subject matter, a first set of a plurality of answers into a first answer category. The first set can correspond to at least one of a third set of a plurality of confidence scores and the second set can correspond to at least one of a fourth set of the plurality of confidence scores. Managing confidence data can include classifying confidence scores of the third set into one of a plurality of confidence buckets using a first threshold and determining a fifth set of a plurality of thresholds using the plurality of confidence scores. Managing confidence data can include classifying unclassified confidence scores of the third set into one of the plurality of confidence buckets using the fifth set of the plurality of thresholds.

Claims

1. A computer-implemented method for managing answer confidence data by a question-answering system, the method comprising:

receiving, via the question-answering system, a query about a subject matter;

parsing, using a natural language processing technique configured to analyze syntactic and semantic content, the query;

searching, based on the parsed query, a corpus having information about the subject matter;

generating, based on the search, a plurality of answers to the query, wherein each answer is associated with confidence score representing a likelihood that the answer is a correct answer to the query;

sorting, based on similarities and differences in types among the plurality of answers and without regard to their associated confidence scores, the plurality of answers into a plurality of answer categories, including sorting a first group of the plurality of answers into a first answer category and a second group of the plurality of answers into a second answer category, wherein each answer category includes answers that are similar in type to each other;

classifying, based on their associated confidence scores, each answer of a sub-group of the first group of answers into one of a plurality of confidence buckets using a first plurality of static, predetermined confidence thresholds associated with the first answer category;

generating, based on the confidence scores associated with the answers of the first group of answers, a first plurality dynamic thresholds associated with the first answer category;

classifying, based on their associated confidence scores, each unclassified answer of the first group answers into one of the plurality of confidence buckets using the first plurality of dynamic thresholds;

classifying, based on their associated confidence scores, each answer of a sub-group of the second group of answers into one of the plurality of confidence buckets using a second plurality of static, predetermined confidence thresholds associated with the second answer category;

generating, based on the confidence scores associated with the answers of the second group of answers, a second plurality dynamic thresholds associated with the second answer category;

classifying, based on their associated confidence scores, each unclassified answer of the second group answers into one of the plurality of confidence buckets using the second plurality of dynamic thresholds; and

presenting, via the question-answering system and as a response to the query, the plurality of answers sorted based on the plurality of confidence buckets.

2. The method of claim 1, wherein the generating the first plurality of dynamic thresholds includes:

detecting a plurality of gaps between consecutive confidence scores associated with the first group of answers;

calculating a standard deviation associated with the plurality of gaps;

identifying, based on the standard deviation, a portion of the plurality of gaps that meets or exceeds the standard deviation; and

using, responsive to its identification, the portion of the plurality of gaps as a basis for the first plurality of dynamic thresholds.

3. The method of claim 1, wherein the generating the first plurality of dynamic thresholds includes:

detecting a plurality of rate changes between consecutive confidence scores associated with the first group of answers;

identifying, based on the detected plurality of rate changes, a portion of the plurality of rate changes as the largest of the plurality of rate changes; and

using, responsive to its identification, the portion of the plurality of rate changes as a basis for the first plurality of dynamic thresholds.

4. The method of claim 1, further comprising:

detecting that a first confidence bucket of the plurality of confidence buckets includes a number of classified answers that exceeds a quantity threshold;

generating, based on the confidence scores associated with the answers classified into the first confidence bucket, a third plurality of dynamic thresholds associated with the first confidence bucket; and

reclassifying, based on the third plurality of dynamic thresholds, a portion of the plurality of classified answers into other confidence buckets.