Semi-supervised learning of word embeddings

• US 9,947,314 B2
• Filed: 02/21/2017
• Issued: 04/17/2018
• Est. Priority Date: 05/08/2015
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Cao, Liangliang, Fan, James J., Wang, Chang, Xiang, Bing, Zhou, Bowen
• Primary Examiner(s): SINGH, SATWANT K
• Application Number:
  US15/437,490
• Publication Number:
  US 20170162189A1
• Time in Patent Office: 420 Days
• Field of Search: None
• CPC Class Codes: G06F 16/3347 using vector based model
• CPC Class Codes: G06F 40/169 Annotation, e.g. comment data or footnotes
• CPC Class Codes: G06F 40/40 Processing or translation of natural language natural language analysis G06F40/20; semantic analysis G06F40/30
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 3/04 Architecture, e.g. interconnection topology
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06N 3/084 Backpropagation, e.g. using gradient descent
• CPC Class Codes: G06N 3/088 Non-supervised learning, e.g. competitive learning
• CPC Class Codes: G06N 5/022 Knowledge engineering; Knowledge acquisition
• CPC Class Codes: G10L 15/063 Training
• CPC Class Codes: G10L 15/16 using artificial neural networks
• CPC Class Codes: G10L 15/18 using natural language modelling

Abstract

Software that trains an artificial neural network for generating vector representations for natural language text, by performing the following steps: (i) receiving, by one or more processors, a set of natural language text; (ii) generating, by one or more processors, a set of first metadata for the set of natural language text, where the first metadata is generated using supervised learning method(s); (iii) generating, by one or more processors, a set of second metadata for the set of natural language text, where the second metadata is generated using unsupervised learning method(s); and (iv) training, by one or more processors, an artificial neural network adapted to generate vector representations for natural language text, where the training is based, at least in part, on the received natural language text, the generated set of first metadata, and the generated set of second metadata.

Claims

1. A method comprising:

receiving, by one or more processors, a set of natural language text;

generating, by one or more processors, a set of first metadata for the set of natural language text, where the first metadata is generated using supervised learning method(s);

generating, by one or more processors, a set of second metadata for the set of natural language text, where the second metadata is generated using unsupervised learning method(s);

training, by one or more processors, an artificial neural network adapted to generate vector representations for natural language text, where the training is based, at least in part, on the received natural language text, the generated set of first metadata, and the generated set of second metadata

generating, by one or more processors, a set of at least two vector representations for the set of natural language text using the trained artificial neural network, where each vector representation of the set of at least two vector representations pertains to a respective subset of natural language text from the set of natural language text;

generating, by one or more processors, a set of third metadata for the generated set of at least two vector representations, where the third metadata is generated using supervised learning method(s);

generating, by one or more processors, a set of fourth metadata for the set of at least two vector representations, where the fourth metadata is generated using unsupervised learning method(s);

training, by one or more processors, the artificial neural network based, at least in part, on the generated set of at least two vector representations, the generated set of third metadata for the set of at least two vector representations, and the generated set of fourth metadata for the set of at least two vector representations; and

storing, by one or more processors, one or more vector representations generated using the trained artificial neural network for use by a natural language processing system.

2. The method of claim 1, further comprising:

determining, by one or more processors, an amount of similarity between at least two subsets of natural language text from the set of natural text by comparing their respectively generated vector representations.

3. The method of claim 2, wherein each of the at least two subsets of natural language text is a word.

4. The method of claim 1, further comprising:

generating, by one or more processors, a vector representation pertaining to the set of natural language text by adding each of the vector representations in the generated set of at least two vector representations.

5. The method of claim 1, further comprising:

generating, by one or more processors, a set of initial vector representations for the set of natural language text;

generating, by one or more processors, a set of first metadata vector representations for the generated set of first metadata; and

generating, by one or more processors, a set of second metadata vector representations for the generated set of second metadata;

wherein the training of the artificial neural network is further based, at least in part, on the generated set of initial vector representations, the generated set of first metadata vector representations, and the generated set of second metadata vector representations.

6. The method of claim 1, wherein the supervised learning methods utilize a natural language processing annotator.

7. The method of claim 1, wherein the unsupervised learning methods are based on reconstruction error.

8. A computer program product comprising a computer readable storage medium having stored thereon:

program instructions programmed to receive a set of natural language text;

program instructions programmed to generate a set of first metadata for the set of natural language text, where the first metadata is generated using supervised learning method(s);

program instructions programmed to generate a set of second metadata for the set of natural language text, where the second metadata is generated using unsupervised learning method(s);

program instructions programmed to train an artificial neural network adapted to generate vector representations for natural language text, where the training is based, at least in part, on the received natural language text, the generated set of first metadata, and the generated set of second metadata

program instructions programmed to generate a set of at least two vector representations for the set of natural language text using the trained artificial neural network, where each vector representation of the set of at least two vector representations pertains to a respective subset of natural language text from the set of natural language text;

program instructions programmed to generate a set of third metadata for the generated set of at least two vector representations, where the third metadata is generated using supervised learning method(s);

program instructions programmed to generate a set of fourth metadata for the set of at least two vector representations, where the fourth metadata is generated using unsupervised learning method(s);

program instructions programmed to train the artificial neural network based, at least in part, on the generated set of at least two vector representations, the generated set of third metadata for the set of at least two vector representations, and the generated set of fourth metadata for the set of at least two vector representations; and

program instructions programmed to store one or more vector representations generated using the trained artificial neural network for use by a natural language processing system.

9. The computer program product of claim 8, further comprising:

program instructions programmed to determine an amount of similarity between at least two subsets of natural language text from the set of natural text by comparing their respectively generated vector representations.

10. The computer program product of claim 9, wherein each of the at least two subsets of natural language text is a word.

11. The computer program product of claim 8, further comprising:

program instructions programmed to generate a vector representation pertaining to the set of natural language text by adding each of the vector representations in the generated set of at least two vector representations.

12. The computer program product of claim 8, further comprising:

program instructions programmed to generate a set of initial vector representations for the set of natural language text;

program instructions programmed to generate a set of first metadata vector representations for the generated set of first metadata; and

program instructions programmed to generate a set of second metadata vector representations for the generated set of second metadata;

wherein the training of the artificial neural network is further based, at least in part, on the generated set of initial vector representations, the generated set of first metadata vector representations, and the generated set of second metadata vector representations.

13. The computer program product of claim 8, wherein the supervised learning methods utilize a natural language processing annotator.

14. The computer program product of claim 8, wherein the unsupervised learning methods are based on reconstruction error.

15. A computer system comprising:

a processor(s) set; and

a computer readable storage medium;

wherein;

the processor set is structured, located, connected and/or programmed to run program instructions stored on the computer readable storage medium; and

the program instructions include;

program instructions programmed to receive a set of natural language text;

program instructions programmed to generate a set of first metadata for the set of natural language text, where the first metadata is generated using supervised learning method(s);

program instructions programmed to generate a set of second metadata for the set of natural language text, where the second metadata is generated using unsupervised learning method(s);

program instructions programmed to train an artificial neural network adapted to generate vector representations for natural language text, where the training is based, at least in part, on the received natural language text, the generated set of first metadata, and the generated set of second metadata

program instructions programmed to generate a set of at least two vector representations for the set of natural language text using the trained artificial neural network, where each vector representation of the set of at least two vector representations pertains to a respective subset of natural language text from the set of natural language text;

program instructions programmed to generate a set of third metadata for the generated set of at least two vector representations, where the third metadata is generated using supervised learning method(s);

program instructions programmed to generate a set of fourth metadata for the set of at least two vector representations, where the fourth metadata is generated using unsupervised learning method(s);

program instructions programmed to train the artificial neural network based, at least in part, on the generated set of at least two vector representations, the generated set of third metadata for the set of at least two vector representations, and the generated set of fourth metadata for the set of at least two vector representations; and

program instructions programmed to store one or more vector representations generated using the trained artificial neural network for use by a natural language processing system.

16. The computer system of claim 15, wherein the program instructions further include:

program instructions programmed to determine an amount of similarity between at least two subsets of natural language text from the set of natural text by comparing their respectively generated vector representations.

17. The computer system of claim 16, wherein each of the at least two subsets of natural language text is a word.

18. The computer system of claim 15, wherein the program instructions further include:

program instructions programmed to generate a vector representation pertaining to the set of natural language text by adding each of the vector representations in the generated set of at least two vector representations.

19. The computer system of claim 15, wherein the program instructions further include:

program instructions programmed to generate a set of initial vector representations for the set of natural language text;

program instructions programmed to generate a set of first metadata vector representations for the generated set of first metadata; and

program instructions programmed to generate a set of second metadata vector representations for the generated set of second metadata;

wherein the training of the artificial neural network is further based, at least in part, on the generated set of initial vector representations, the generated set of first metadata vector representations, and the generated set of second metadata vector representations.

20. The computer system of claim 15, wherein the supervised learning methods utilize a natural language processing annotator, and wherein the unsupervised learning methods are based on reconstruction error.