Answer sequence evaluation

• US 9,501,525 B2
• Filed: 12/15/2014
• Issued: 11/22/2016
• Est. Priority Date: 11/05/2014
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Barker, Kevin S., Eggebraaten, Thomas J., Megerian, Mark G.
• Primary Examiner(s): Hill, Stanley K
• Assistant Examiner(s): BROWN JR, NATHAN H
• Application Number:
  US14/570,797
• Publication Number:
  US 20160125298A1
• Time in Patent Office: 708 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 16/24522 Translation of natural language queries to structured queries
• CPC Class Codes: G06F 16/24575 using context
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06F 16/284 Relational databases
• CPC Class Codes: G06F 16/285 Clustering or classification
• CPC Class Codes: G06F 16/3329 Natural language query formulation or dialogue systems
• CPC Class Codes: G06F 16/3332 Query translation
• CPC Class Codes: G06F 16/3334 Selection or weighting of terms from queries, including natural language queries
• CPC Class Codes: G06F 16/3344 using natural language analysis
• CPC Class Codes: G06F 16/367 Ontology
• CPC Class Codes: G06F 16/90324 using system suggestions
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: G06F 16/9538 Presentation of query results
• CPC Class Codes: G06F 16/955 using information identifiers, e.g. uniform resource locators [URL]
• CPC Class Codes: G06F 40/205 Parsing
• CPC Class Codes: G06F 40/211 Syntactic parsing, e.g. based on context-free grammar [CFG] or unification grammars
• CPC Class Codes: G06F 40/284 Lexical analysis, e.g. tokenisation or collocates
• CPC Class Codes: G06F 40/30 Semantic analysis
• CPC Class Codes: G06F 40/40 Processing or translation of natural language natural language analysis G06F40/20; semantic analysis G06F40/30
• CPC Class Codes: G06N 20/00 : Machine learning
• CPC Class Codes: G06N 5/022 : Knowledge engineering; Knowledge acquisition
• CPC Class Codes: G06N 5/046 : Forward inferencing; Production systems
• CPC Class Codes: G06N 7/01 : Probabilistic graphical models, e.g. probabilistic networks
• CPC Class Codes: G06Q 30/0203 : Market surveys; Market polls
• CPC Class Codes: G09B 7/00 : Electrically-operated teaching apparatus or devices working with questions and answers mechanically operated G09B3/00; computing arrangements G06F
• CPC Class Codes: G09B 7/02 : of the type wherein the student is expected to construct an answer to the question which is presented or wherein the machine gives an answer to the question presented by a student
• CPC Class Codes: G16H 10/20 : for electronic clinical trials or questionnaires
• CPC Class Codes: G16H 40/20 : for the management or administration of healthcare resources or facilities, e.g. managing hospital staff or surgery rooms
• CPC Class Codes: G16H 50/70 : for mining of medical data, e.g. analysing previous cases of other patients
• CPC Class Codes: G16Z 99/00 : Subject matter not provided for in other main groups of this subclass

Abstract

Aspects of the present disclosure are directed toward evaluating an answer sequence. Aspects are directed toward receiving a set of answer sequences including a first answer sequence. The first answer sequence may have a first set of answers. Aspects are also directed toward identifying a set of scores coupled with the first set of answers. Aspects are also directed toward determining, based on a subject matter corresponding to the first answer sequence, a set of evaluation rules. Aspects are also directed toward generating, based on the set of scores and the set of evaluation rules, a sequence evaluation score for the first answer sequence.

Claims

1. A computer system, comprising:

at least one processor;

a memory coupled to the at least one processor;

an answer sequence evaluation mechanism executed by one or more of the at least one processor, the one or more of the at least one processor configured to;

receive a set of answer sequences including a first answer sequence having a first set of answers;

identify a set of scores coupled with the first set of answers;

identify a set of answer categories corresponding to the first set of answers of the first answer sequence;

collect, for the set of answer categories corresponding to the first set of answers of the first answer sequence, context data indicating a relative importance of a first answer category of the set of answer categories to the first answer sequence;

determine, based on a subject matter corresponding to the first answer sequence, a set of evaluation rules, wherein a first evaluation rule of the set of evaluation rules;

assigns, based on the context data, a first weighting value to the first answer category of the set of answer categories;

assigns, based on the context data, a second weighting value to a second answer category of the set of answer categories; and

calculates, by a statistical algorithm using the first weighting value and the second weighting value, an aggregate score;

receive, from a user, a first set of answer preference data indicating an inclination for the first answer category of the set of answer categories;

increase, based on the first set of answer preference data, the first weighting value assigned to the first answer category; and

generate, based on the set of scores and the set of evaluation rules, a sequence evaluation score for the first answer sequence, wherein generating the sequence evaluation score for the first answer sequence includes assigning, to the first answer sequence, the aggregate score as the sequence evaluation score.

2. The system of claim 1, wherein the one or more of the at least one processor is further configured to:

compute, based on the subject matter, a caution value for the first answer sequence;

determine, by comparing the caution value to a first caution threshold, that the caution value achieves the first caution threshold; and

select, in response to determining that the caution value achieves the first caution threshold, a second evaluation rule.

3. The system of claim 2, wherein:

the second evaluation rule identifies, from the set of scores coupled with the first set of answers, a first score of the set of scores, wherein the first score does not achieve a first score threshold.

4. The system of claim 2, wherein the one or more of the at least one processor is further configured to:

determine, by comparing the caution value to a second caution threshold, that the caution value does not achieve the second caution threshold;

select, in response to determining that the caution value does not achieve the second caution threshold, a third evaluation rule.

5. A computer program product comprising a computer readable storage medium having a computer readable program stored therein, wherein the computer readable program, when executed on a first computing device, causes the computing device to:

receive a set of answer sequences including a first answer sequence having a first set of answers;

identify a set of scores coupled with the first set of answers;

identify a set of answer categories corresponding to the first set of answers of the first answer sequence;

collect, for the set of answer categories corresponding to the first set of answers of the first answer sequence, context data indicating a relative importance of a first answer category of the set of answer categories to the first answer sequence;

determine, based on a subject matter corresponding to the first answer sequence, a set of evaluation rules, wherein a first evaluation rule of the set of evaluation rules;

assigns, based on the context data, a first weighting value to the first answer category of the set of answer categories;

assigns, based on the context data, a second weighting value to a second answer category of the set of answer categories; and

calculates, by a statistical algorithm using the first weighting value and the second weighting value, an aggregate score;

receive, from a user, a first set of answer preference data indicating an inclination for the first answer category of the set of answer categories;

increase, based on the first set of answer preference data, the first weighting value assigned to the first answer category; and

generate, based on the set of scores and the set of evaluation rules, a sequence evaluation score for the first answer sequence, wherein generating the sequence evaluation score for the first answer sequence includes assigning, to the first answer sequence, the aggregate score as the sequence evaluation score.

6. The computer program product of claim 5, further comprising computer readable program code configured to:

evaluate, using a natural language processing technique configured to parse semantic and syntactic content, the context data;

determine, in response to evaluating the context data, that the context data achieves a satisfaction criterion; and

select, in response to determining that the context data achieves the satisfaction criterion, a second evaluation rule.

7. The computer program product of claim 5, further comprising computer readable program code configured to:

receive, from the user, a second set of answer preference data indicating a disinclination for the second answer category of the set of answer categories; and

decrease, based on the second set of answer preference data, the second weighting value assigned to the second answer category.

8. A computer program product comprising a computer readable storage medium having a computer readable program stored therein, wherein the computer readable program, when executed on a first computing device, causes the computing device to:

receive a set of answer sequences including a first answer sequence having a first set of answers;

identify a set of scores coupled with the first set of answers;

identify a set of answer categories corresponding to the first set of answers of the first answer sequence;

collect, for the set of answer categories corresponding to the first set of answers of the first answer sequence, context data indicating a relative importance of a first answer category of the set of answer categories to the first answer sequence;

determine, based on a subject matter corresponding to the first answer sequence, a set of evaluation rules, wherein a first evaluation rule of the set of evaluation rules;

assigns, based on the context data, a first weighting value to the first answer category of the set of answer categories;

assigns, based on the context data, a second weighting value to a second answer category of the set of answer categories; and

calculates, by a statistical algorithm using the first weighting value and the second weighting value, an aggregate score;

receive, from a user, a first set of answer preference data indicating a disinclination for the second answer category of the set of answer categories;

decrease, based on the first set of answer preference data, the second weighting value assigned to the second answer category; and

generate, based on the set of scores and the set of evaluation rules, a sequence evaluation score for the first answer sequence, wherein generating the sequence evaluation score for the first answer sequence includes assigning, to the first answer sequence, the aggregate score as the sequence evaluation score.