Validating topical data of unstructured text in electronic forms to control a graphical user interface based on the unstructured text relating to a question included in the electronic form

• US 10,380,246 B2
• Filed: 12/18/2014
• Issued: 08/13/2019
• Est. Priority Date: 12/18/2014
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Clark, Adam T., Huebert, Jeffrey K., Payton, Aspen L., Petri, John E.
• Primary Examiner(s): Yehl, Walter
• Application Number:
  US14/575,283
• Publication Number:
  US 20160179788A1
• Time in Patent Office: 1,699 Days
• CPC Class Codes: G06F 40/10 Text processing natural language analysis G06F40/20; semantic analysis G06F40/30; processing or translation of natural language G06F40/40
• CPC Class Codes: G06F 40/242 Dictionaries
• CPC Class Codes: G06F 40/284 Lexical analysis, e.g. tokenisation or collocates
• CPC Class Codes: G06F 40/30 Semantic analysis
• CPC Class Codes: G06F 40/35 Discourse or dialogue representation
• CPC Class Codes: G06N 20/00 Machine learning

Abstract

Techniques disclose validating user-provided text feedback for topical relevance relative to a question asked. A form with at least a first field is received. The first field includes unstructured text content provided as feedback in response to a question. The unstructured text content of the first field is evaluated to identify an answer type. A measure of relevance of the unstructured text content relative to the question is determined based on the evaluation.

Claims

1. A computer program product, comprising:

a computer-readable storage medium having computer-readable program code embodied therewith, the computer-readable program code configured to perform an operation for validating user-provided feedback from a client on an electronic form, the operation comprising;

receiving, via a processor, an electronic form comprising unstructured text, the unstructured text including at least (i) a first section containing a question and (ii) a first field containing content provided as feedback in response to the question;

extracting the unstructured text from the electronic form,

identifying, from the unstructured text, a context for the question included in the first section, based on a natural language analysis of the question included in the first section;

determining expected concepts and answer types for the question included in the first section, based on lexical and topical data extracted from the unstructured text;

performing a lexical and content analysis of the content of the first field to identify interrogative words and key terms within the content;

training a machine-learning data model using a plurality of form submissions for the form, comprising;

extracting, from each of the plurality of form submissions, a respective portion of unstructured text;

analyzing the portions of unstructured text using the natural language analysis to determine a subject matter of a question in the form and expected answers for the question; and

tuning the machine-learning data model based on a set of sample answers to the form;

calculating a score indicating how closely the content of the first field resembles a relevant answer to the question included in the first section, by processing data associated with the unstructured text using the trained machine-learning data model, wherein the trained machine-learning data model evaluates an extent to which the content of the first field includes expected concepts, answer types, lexical answer types, and topics relating to the question included in the first section;

classifying the electronic form into a first category of a plurality of predefined categories, based on the calculated score not exceeding a predefined threshold; and

returning the classification of the electronic form to an application service executing on an application server, wherein the application service is configured to, upon receiving the classification, dynamically control a graphical user interface through which the user-provided feedback was received to block the client from proceeding and to prompt for re-entry of the user-provided feedback.

2. The computer program product of claim 1, wherein the operation further comprises:

upon determining that the measure of relevance exceeds a specified threshold, accepting the feedback as relevant to the question.

3. The computer program product of claim 1, wherein the operation further comprises:

upon determining that the measure of relevance does not exceed a specified threshold, rejecting the feedback; and

generating a prompt requesting new feedback responsive to the question.

4. The computer program product of claim 1, wherein the measure of relevance is determined based on an expected answer type associated with the question.

5. The computer program product of claim 4, wherein the expected answer type is identified based on lexical and contextual data extracted from the form.

6. The computer program product of claim 5, wherein the expected answer type is further determined using natural language processing techniques on the lexical and contextual data extracted from the form.

7. The computer program product of claim 1, wherein the operation further comprises:

determining, based on the measure of relevance, whether to forward the feedback to a reviewer.

8. A system, comprising:

a processor; and

a memory storing a program, which, when executed on the processor, performs an operation for validating user-provided feedback from a client on an electronic form, the operation comprising;

receiving an electronic form comprising unstructured text, the unstructured text including at least (i) a first section containing a question and (ii) a first field containing content provided as feedback in response to the question;

extracting the unstructured text from the electronic form;

identifying, from the unstructured text, a context for the question included in the first section, based on a natural language analysis of the question included in the first section;

determining expected concepts and answer types for the question included in the first section, based on lexical and topical data extracted from the unstructured text;

performing a lexical and content analysis of the content of the first field to identify interrogative words and key terms within the content;

training a machine-learning data model using a plurality of form submissions for the form, comprising;

extracting, from each of the plurality of form submissions, a respective portion of unstructured text;

analyzing the portions of unstructured text using the natural language analysis to determine a subject matter of a question in the form and expected answers for the question; and

tuning the machine-learning data model based on a set of sample answers to the form;

calculating a score indicating how closely the content of the first field resembles a relevant answer to the question included in the first section, by processing data associated with the unstructured text using the trained machine-learning data model, wherein the trained machine-learning data model evaluates an extent to which the content of the first field includes expected concepts, answer types, lexical answer types, and topics relating to the question included in the first section;

classifying the electronic form into a first category of a plurality of predefined categories, based on the calculated score not exceeding a predefined threshold; and

returning the classification of the electronic form to an application service executing on an application server, wherein the application service is configured to, upon receiving the classification, dynamically control a graphical user interface through which the user-provided feedback was received to block the client from proceeding and to prompt for re-entry of the user-provided feedback.

9. The system of claim 8, wherein the operation further comprises:

upon determining that the measure of relevance exceeds a specified threshold, accepting the feedback as relevant to the question.

10. The system of claim 8, wherein the operation further comprises:

upon determining that the measure of relevance does not exceed a specified threshold, rejecting the feedback; and

generating a prompt requesting new feedback responsive to the question.

11. The system of claim 8, wherein the measure of relevance is determined based on an expected answer type associated with the question, the expected answer type identified based on lexical and contextual data extracted from the form.