WORD VECTOR PROCESSING FOR FOREIGN LANGUAGES

• US 20200134262A1
• Filed: 09/30/2019
• Published: 04/30/2020
• Est. Priority Date: 01/22/2017
• Status: Active Grant

• Current Assignee: Advanced New Technologies Company Limited (Ant Group Co., Ltd.)
• Original Assignee: Alibaba Group Holding Ltd.
• Inventors: Cao, Shaosheng, Li, Xiaolong
• Granted Patent: US 10,878,199 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 40/30 Semantic analysis
• CPC Class Codes: G06F 40/53 Processing of non-Latin text kana-to-kanji conversion G06F40/129; vowelisation G06F40/232
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 3/084 Backpropagation, e.g. using gradient descent

Abstract

A word vector processing method is provided. Word segmentation is performed on a corpus to obtain words, and n-gram strokes corresponding to the words are determined. Each n-gram stroke represents n successive strokes of a corresponding word. Word vectors of the words and stroke vectors of the n-gram strokes are initialized corresponding to the words. After performing the word segmentation, the n-gram strokes are determined, and the word vectors and stroke vectors are determined, training the word vectors and the stroke vectors.

Claims

1. A word vector processing method, comprising:

performing word segmentation on a corpus to obtain words;

determining n-gram strokes corresponding to the words, the n-gram stroke representing n successive strokes of a corresponding word;

initializing word vectors of the words and stroke vectors of the n-gram strokes corresponding to the words; and

after performing the word segmentation, determining the n-gram strokes, and initializing the word vectors and stroke vectors, training the word vectors and the stroke vectors.

2. The method of claim 1, wherein determining n-gram strokes corresponding to the words comprises:

determining, according to a result of the word segmentation on the corpus, words occurring at least once in the corpus; and

performing the following operation on each determined word;

determining n-gram strokes corresponding to the word, wherein each n-gram stroke corresponding to the word represents n successive strokes of the word, and n is one positive integer or multiple different positive integers.

3. The method of claim 2, wherein determining, according to a result of the word segmentation on the corpus, words occurring at least once in the corpus specifically comprises:

determining, according to the result of the word segmentation on the corpus, a word that occurs in the corpus for not less than a set number of times, the set number of times being not less than 1.

4. The method of claim 1, wherein initializing word vectors of the words and stroke vectors of the n-gram strokes corresponding to the words specifically comprises:

initializing the word vectors of the words and the stroke vectors of the n-gram strokes corresponding to the words in a random initialization manner or in a manner of initializing according to a specified probability distribution, wherein stroke vectors of the same n-gram strokes are also the same.

5. The method of claim 1, wherein training the word vectors and the stroke vectors according to the corpus comprises:

determining a designated word in the corpus, and one or more context words of the designated word in the corpus;

determining a degree of similarity between the designated word and the context word according to stroke vectors of n-gram strokes corresponding to the designated word as well as a word vector of the context word; and

updating the word vector of the context word and the stroke vectors of the n-gram strokes corresponding to the designated word according to the degree of similarity between the designated word and the context word.

6. The method of claim 5, wherein updating the word vector of the context word and the stroke vectors of the n-gram strokes corresponding to the designated word according to the degree of similarity between the designated word and the context word specifically comprises:

selecting one or more words from the words as a negative sample word;

determining a degree of similarity between the designated word and each negative sample word;

determining a loss characterization value corresponding to the designated word according to a designated loss function, the degree of similarity between the designated word and the context word, and the degree of similarity between the designated word and each negative sample word; and

updating the word vector of the context word and the stroke vectors of the n-gram strokes corresponding to the designated word according to the loss characterization value.

7. The method of claim 6, wherein updating the word vector of the context word and the stroke vectors of the n-gram strokes corresponding to the designated word according to the loss characterization value specifically comprises:

determining a gradient corresponding to the loss function according to the loss characterization value; and

updating the word vector of the context word and the stroke vectors of the n-gram strokes corresponding to the designated word according to the gradient.

8. The method of claim 6, wherein selecting one or more words from the words as a negative sample word specifically comprises:

randomly selecting one or more words from the words as the negative sample word.

9. The method of claim 1, wherein training the word vectors and the stroke vectors according to the corpus comprises:

traversing the corpus, and performing the following operation on each word in the corpus;

determining one or more context words of the word in the corpus; and

performing the following operation according to each context word;

determining a degree of similarity between the word and the context word according to stroke vectors of n-gram strokes corresponding to the word and a word vector of the context word; and

updating the word vector of the context word and the stroke vectors of the n-gram strokes corresponding to the word according to the degree of similarity between the word and the context word.

10. The method of claim 9, wherein determining one or more context words of the word in the corpus comprises:

sliding, with this word as a center, leftward and/or rightward by a distance of a designated number of words in the corpus, to establish a window; and

determining words excluding this word in the window as the context words of this word.

11. The method of claim 1, wherein the words are Chinese words, and the word vectors are word vectors of the Chinese words.

12. A non-transitory, computer-readable medium storing one or more instructions executable by a computer system to perform operations comprising:

performing word segmentation on a corpus to obtain words;

determining n-gram strokes corresponding to the words, the n-gram stroke representing n successive strokes of a corresponding word;

initializing word vectors of the words and stroke vectors of the n-gram strokes corresponding to the words; and

after performing the word segmentation, determining the n-gram strokes, and initializing the word vectors and stroke vectors, training the word vectors and the stroke vectors.

13. The non-transitory, computer-readable medium of claim 12, wherein determining n-gram strokes corresponding to the words comprises:

determining, according to a result of the word segmentation on the corpus, words occurring at least once in the corpus; and

performing the following operation on each determined word;

determining n-gram strokes corresponding to the word, wherein each n-gram stroke corresponding to the word represents n successive strokes of the word, and n is one positive integer or multiple different positive integers.

14. The non-transitory, computer-readable medium of claim 13, wherein determining, according to a result of the word segmentation on the corpus, words occurring at least once in the corpus specifically comprises:

determining, according to the result of the word segmentation on the corpus, a word that occurs in the corpus for not less than a set number of times, the set number of times being not less than 1.

15. The non-transitory, computer-readable medium of claim 12, wherein initializing word vectors of the words and stroke vectors of the n-gram strokes corresponding to the words specifically comprises:

initializing the word vectors of the words and the stroke vectors of the n-gram strokes corresponding to the words in a random initialization manner or in a manner of initializing according to a specified probability distribution, wherein stroke vectors of the same n-gram strokes are also the same.

16. The non-transitory, computer-readable medium of claim 12, wherein training the word vectors and the stroke vectors according to the corpus comprises:

determining a designated word in the corpus, and one or more context words of the designated word in the corpus;

determining a degree of similarity between the designated word and the context word according to stroke vectors of n-gram strokes corresponding to the designated word as well as a word vector of the context word; and

updating the word vector of the context word and the stroke vectors of the n-gram strokes corresponding to the designated word according to the degree of similarity between the designated word and the context word.

17. A computer-implemented system, comprising:

one or more computers; and

one or more computer memory devices interoperably coupled with the one or more computers and having tangible, non-transitory, machine-readable media storing one or more instructions that, when executed by the one or more computers, perform one or more operations comprising;

performing word segmentation on a corpus to obtain words;

determining n-gram strokes corresponding to the words, the n-gram stroke representing n successive strokes of a corresponding word;

initializing word vectors of the words and stroke vectors of the n-gram strokes corresponding to the words; and

after performing the word segmentation, determining the n-gram strokes, and initializing the word vectors and stroke vectors, training the word vectors and the stroke vectors.

18. The computer-implemented system of claim 17, wherein determining, according to a result of the word segmentation on the corpus, words occurring at least once in the corpus specifically comprises:

determining, according to the result of the word segmentation on the corpus, a word that occurs in the corpus for not less than a set number of times, the set number of times being not less than 1.

19. The computer-implemented system of claim 18, wherein initializing word vectors of the words and stroke vectors of the n-gram strokes corresponding to the words specifically comprises:

initializing the word vectors of the words and the stroke vectors of the n-gram strokes corresponding to the words in a random initialization manner or in a manner of initializing according to a specified probability distribution, wherein stroke vectors of the same n-gram strokes are also the same.

20. The computer-implemented system of claim 17, wherein training the word vectors and the stroke vectors according to the corpus comprises:

determining a designated word in the corpus, and one or more context words of the designated word in the corpus;

determining a degree of similarity between the designated word and the context word according to stroke vectors of n-gram strokes corresponding to the designated word as well as a word vector of the context word; and

updating the word vector of the context word and the stroke vectors of the n-gram strokes corresponding to the designated word according to the degree of similarity between the designated word and the context word.