Disambiguating concepts in natural language

• US 10,372,824 B2
• Filed: 05/15/2017
• Issued: 08/06/2019
• Est. Priority Date: 05/15/2017
• Status: Active Grant

• Current Assignee: Merative US LP
• Original Assignee: International Business Machines Corporation
• Inventors: Bull, Brendan, Felt, Paul Lewis
• Primary Examiner(s): Shah, Bharatkumar S
• Application Number:
  US15/594,829
• Publication Number:
  US 20180329885A1
• Time in Patent Office: 813 Days
• Field of Search: 704 9
• CPC Class Codes: G06F 16/00 Information retrieval; Database structures therefor; File system structures therefor
• CPC Class Codes: G06F 16/3334 Selection or weighting of terms from queries, including natural language queries
• CPC Class Codes: G06F 16/3347 using vector based model
• CPC Class Codes: G06F 40/30 Semantic analysis
• CPC Class Codes: G10L 15/18 using natural language modelling
• CPC Class Codes: G10L 15/1815 Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning

Abstract

A computer receives a plurality of text and determines a concept is present in the plurality of text. The computer determines a set of hypotheses for the determined concept, wherein the set of hypotheses is a plurality of natural language representations of the determined concept. The computer substitutes the determined concept in the plurality of text with a hypothesis from the determined set of hypotheses. The computer determines the hypothesis is valid based on analyzing the plurality of text with a neural network, wherein the neural network is trained for hypothesis validation. Based on determining that the hypothesis is valid, the computer storing the plurality of text with the determined hypothesis in place of the substituted concept and displays the stored plurality of text.

Claims

1. A processor-implemented method to improve determination of a valid sense of a concept by concept disambiguation in a natural language input, the method comprising:

receiving a plurality of text;

determining a concept is present in the plurality of text;

determining a set of hypotheses for the determined concept, wherein the set of hypotheses is a plurality of natural language representations of the determined concept, and wherein determining the set of hypotheses for the determined concept comprises;

analyzing a plurality of concept ontology data for the determined set of hypotheses corresponding to the determined concept; and

associating the determined set of hypotheses with the determined concept, wherein associating the determined set of hypotheses with the determined concept is based on converting the determined concept into a word embedded vector and determining a similar vector in the analyzed plurality of concept ontology data;

substituting the determined concept in the plurality of text with a hypothesis from the determined set of hypotheses;

determining the hypothesis is valid based on analyzing the plurality of text with a neural network, wherein the neural network is trained for hypothesis validation;

storing the plurality of text with the determined hypothesis in place of the substituted concept based on determining that the hypothesis is valid; and

displaying the stored plurality of text.

2. The method of claim 1, further comprising:

determining a next hypothesis from the determined set of hypotheses for the determined concept;

based on determining the next hypothesis, determining that the next hypothesis is valid based on analyzing the plurality of text with the neural network;

storing the plurality of text with the determined next hypothesis based on determining that the next hypothesis is valid; and

displaying the plurality of text with the next hypothesis.

3. The method of claim 1, further comprising:

displaying the plurality of text and the determined set of hypotheses;

determining a correct hypothesis based on a user input; and

training the neural network with the plurality of text and the determined correct hypothesis.

4. The method of claim 1, wherein determining that the hypothesis is valid further comprises:

analyzing the plurality of text with the neural network;

receiving an output from the neural network;

based on determining that the received output from the neural network is above a threshold value, determining that the hypothesis is valid; and

based on determining that the received output from the neural network is below the threshold value, determining that the hypothesis is invalid.

5. The method of claim 4, wherein the received output comprises a weighted average of a local context parameter and a document context parameter, and wherein the local context parameter establishes a syntactic validity of the hypothesis, and wherein the document context parameter establishes a topic validity of the hypothesis in a context of the plurality of text.

6. A computer system to improve determination of a valid sense of a concept by concept disambiguation in a natural language input, the computer system comprising:

one or more processors, one or more computer-readable memories, one or more computer-readable tangible storage medium, and program instructions stored on at least one of the one or more tangible storage medium for execution by at least one of the one or more processors via at least one of the one or more memories, wherein the computer system is capable of performing a method comprising;

receiving a plurality of text;

determining a concept is present in the plurality of text;

determining a set of hypotheses for the determined concept, wherein the set of hypotheses is a plurality of natural language representations of the determined concept, and wherein determining the set of hypotheses for the determined concept comprises;

analyzing a plurality of concept ontology data for the determined set of hypotheses corresponding to the determined concept; and

associating the determined set of hypotheses with the determined concept, wherein associating the determined set of hypotheses with the determined concept is based on converting the determined concept into a word embedded vector and determining a similar vector in the analyzed plurality of concept ontology data;

substituting the determined concept in the plurality of text with a hypothesis from the determined set of hypotheses;

determining the hypothesis is valid based on analyzing the plurality of text with a neural network, wherein the neural network is trained for hypothesis validation;

storing the plurality of text with the determined hypothesis in place of the substituted concept based on determining that the hypothesis is valid; and

displaying the stored plurality of text.

7. The computer system of claim 6, further comprising:

determining a next hypothesis from the determined set of hypotheses for the determined concept;

based on determining the next hypothesis, determining that the next hypothesis is valid based on analyzing the plurality of text with the neural network;

storing the plurality of text with the determined next hypothesis based on determining that the next hypothesis is valid; and

displaying the plurality of text with the next hypothesis.

8. The computer system of claim 6, further comprising:

displaying the plurality of text and the determined set of hypotheses;

determining a correct hypothesis based on a user input; and

training the neural network with the plurality of text and the determined correct hypothesis.

9. The computer system of claim 6, wherein determining that the hypothesis is valid further comprises:

analyzing the plurality of text with the neural network;

receiving an output from the neural network;

based on determining that the received output from the neural network is above a threshold value determining that the hypothesis is valid; and

based on determining that the received output from the neural network is below the threshold value, determining that the hypothesis is invalid.

10. The computer system of claim 9, wherein the received output comprises a weighted average of a local context parameter and a document context parameter, and wherein the local context parameter establishes a syntactic validity of the hypothesis, and wherein the document context parameter establishes a topic validity of the hypothesis in a context of the plurality of text.

11. A non-transitory computer program product to improve determination of a valid sense of a concept by concept disambiguation in a natural language input, the computer program product comprising:

one or more computer-readable tangible storage medium and program instructions stored on at least one of the one or more tangible storage medium, the program instructions executable by a processor, the program instructions comprising;

program instructions to receive a plurality of text;

program instructions to determine a concept is present in the plurality of text;

program instructions to determine a set of hypotheses for the determined concept, wherein the set of hypotheses is a plurality of natural language representations of the determined concept, and wherein program instructions to determine the set of hypotheses for the determined concept comprises;

program instructions to analyze a plurality of concept ontology data for the determined set of hypotheses corresponding to the determined concept; and

program instructions to associate the determined set of hypotheses with the determined concept, wherein program instructions to associate the determined set of hypotheses with the determined concept is based on program instructions to convert the determined concept into a word embedded vector and determining a similar vector in the analyzed plurality of concept ontology data;

program instructions to substitute the determined concept in the plurality of text with a hypothesis from the determined set of hypotheses;

program instructions to determine the hypothesis is valid based on analyzing the plurality of text with a neural network, wherein the neural network is trained for hypothesis validation;

program instructions to store the plurality of text with the determined hypothesis in place of the substituted concept based on determining that the hypothesis is valid; and

program instructions to display the stored plurality of text.

12. The non-transitory computer program product of claim 11, further comprising:

program instructions to determine a next hypothesis from the determined set of hypotheses for the determined concept;

based on determining the next hypothesis, program instructions to determine that the next hypothesis is valid based on analyzing the plurality of text with the neural network;

program instructions to store the plurality of text with the determined next hypothesis based on determining that the next hypothesis is valid; and

program instructions to display the plurality of text with the next hypothesis.

13. The non-transitory computer program product of claim 11, further comprising:

program instructions to display the plurality of text and the determined set of hypotheses;

program instructions to determine a correct hypothesis based on a user input; and

program instructions to train the neural network with the plurality of text and the determined correct hypothesis.

14. The non-transitory computer program product of claim 11, wherein program instructions to determine that the hypothesis is valid further comprises:

program instructions to analyzing the plurality of text with the neural network;

program instructions to receive an output from the neural network;

based on determining that the received output from the neural network is above a threshold value, program instructions to determine that the hypothesis is valid; and

based on determining that the received output from the neural network is below the threshold value, program instructions to determine that the hypothesis is invalid.