Method for text processing

• US 9,898,448 B2
• Filed: 10/23/2015
• Issued: 02/20/2018
• Est. Priority Date: 08/29/2014
• Status: Active Grant

• Current Assignee: YE Hub Armenia LLC
• Original Assignee: Yandex Europe AG (Yandex N.V.)
• Inventors: Melnikov, Ilia Alekseevich
• Primary Examiner(s): SPOONER, LAMONT M
• Application Number:
  US14/921,189
• Publication Number:
  US 20160232142A1
• Time in Patent Office: 851 Days
• Field of Search: 704 1, 704 9, 704 10
• CPC Class Codes: G06F 40/151 Transformation
• CPC Class Codes: G06F 40/247 Thesauruses; Synonyms
• CPC Class Codes: G06F 40/253 Grammatical analysis; Style critique
• CPC Class Codes: G06F 40/30 Semantic analysis
• CPC Class Codes: G06F 40/44 Statistical methods, e.g. probability models
• CPC Class Codes: G10L 13/08 Text analysis or generation of parameters for speech synthesis out of text, e.g. grapheme to phoneme translation, prosody generation or stress or intonation determination

Abstract

Method for text processing executable at a computing device, comprising appreciating a source phrase comprised of source words; appreciating a target phrase comprised of target words; associating a respective source word feature set with each one of the source words; associating a respective target word feature set with each one of the target words; analyzing source word feature sets and target word feature sets; and based on the analysis, generating one or more phrase transformation rules for transforming the source phrase into the target phrase. Also a server and non-transitory computer-readable medium storing program instructions for carrying out the method.

Claims

1. A computer-implemented method for text processing, the method being executable at a computing device, the method comprising:

at a training phase;

acquiring one or more source phrases, each of the source phrase comprising a first set of sequential words, each word of the first set of sequential words being a source word;

acquiring one or more target phrases, each of the target phrase being in a same language as the source phrases, each of the target phrase comprising a second set of sequential words being at least partially different from the first set of sequential words of a respective source phrase, each word of the second set of sequential words being a target word;

associating, for a given source phrase, a respective source word feature set with each one of the source words, the respective source word feature set for a given source word comprising;

one or more grammatical features of the given source word; and

a meaning of the given source word;

associating, for a respective target phrase, a respective target word feature set with each one of the target words, the respective target word feature set for a given target word comprising;

one or more grammatical features of the given target word; and

a meaning of the given target word;

analyzing the respective source word feature set of each source words of the given source phrase and the respective target word feature set of each target words of the respective target phrase;

mapping the given source word of the given source phrase to a corresponding target word of the respective target phrase based on a similarity of the source word feature set of the given source word to the target word feature set of the corresponding target word;

based on the mapping, generating one or more phrase transformation rules applicable to the given source phrase to transform the first set of sequential words into the second set of sequential words of the respective target phrase;

storing the one or more source phrases and the associated one or more generated phrase transformation rules in a memory of the computing device;

at an in-use phase;

acquiring a text phrase, the text phrase comprising a third set of sequential words being at least partially different from the first set of sequential words; and

retrieving the one or more source phrases from the memory;

performing at least one of a grammatical analysis and a semantic analysis of the text phrase and the one or more stored source phrases, to determine similarity of the text phrase to the one or more stored source phrases;

upon determining that the text phrase has the similarity to the given stored source phrase greater than a threshold, applying the associated one or more phrase transformation rules to the text phrase to generate a transformed text phrase, the transformed text phrase comprising a fourth set of sequential words being at least partially similar to the second set of sequential words of the respective target phrase.

2. The method of claim 1, wherein:

determining the similarity of each one of the source word feature sets to each one of the target word feature sets comprises comparing the respective grammatical features of each one of the source words to the respective grammatical features of each one of the target words;

determining the similarity of each one of the source word feature sets to each one of the target word feature sets comprises determining a similarity of the respective meaning of each one of the source words to the respective meaning of each one of the target words.

3. The method of claim 1, wherein:

analyzing the source word feature sets and the target word feature sets to determine a mapping from the source words to the target words further comprises selecting a most similar one of the target word feature sets to one of the source word feature sets; and

generating the one or more phrase transformation rules applicable to the given source phrase to transform the given source phrase into the respective target phrase comprises generating a first phrase transformation rule applicable to the given source phrase to transform a source word associated with the one of the source word feature sets into a target word associated with the most similar one of the target word feature sets.

4. The method of claim 3, wherein:

the source word associated with the one of the source word feature sets and the target word associated with the most similar one of the target word feature sets share a same root; and

the first phrase transformation rule is applicable to the given source phrase to modify a form of the source word to a form of the target word.

5. The method of claim 1, wherein generating the one or more phrase transformation rules applicable to the given source phrase to transform the given source phrase into the respective target phrase comprises generating a second phrase transformation rule applicable to the source phrase to delete at least one word from the given source phrase.

6. The method of claim 1, wherein generating the one or more phrase transformation rules applicable to the given source phrase to transform the given source phrase into the respective target phrase comprises generating a third phrase transformation rule applicable to the given source phrase to add at least one word to the given source phrase.

7. A method for text processing, the method being executable at a computing device comprising a memory, the memory storing one or more source phrases, and one or more phrase transformation rules associated with each of the one or more source phrases, the one or more phrase transformation rules having been generated based on an analysis of feature sets including (i) a source word feature set associated with each source word comprising a first set of sequential words forming a given source phrase, the source word feature set for a given source word comprising one or more grammatical features of the given source word, and a meaning of the given source word, and (ii) a target word feature set associated with each target word comprising a second set of sequential words forming a respective target phrase, the target word feature set for a given target word comprising one or more grammatical features of the given target word, and a meaning of the given target word, the target phrase being in a same language as the source phrase, the analysis comprising mapping the given source word to a corresponding target word based on a similarity of the source word feature set of the given source word to the target word feature set of the corresponding target word, the method comprising:

acquiring a text phrase, the text phrase comprising a third set of sequential words being at least partially different from the first set of sequential words;

retrieving the one or more source phrases;

performing at least one of a grammatical analysis and a semantic analysis of the text phrase and the one or more stored source phrases to determine similarity of the text phrase to the one or more stored source phrases;

upon determining that the similarity of the text phrase to the given source phrase is greater than a threshold applying the associated one or more phrase transformation rules to the text phrase to generate a transformed text phrase, the transformed text phrase being at least partially similar to the second set of sequential words of the respective target phrase.

8. A non-transitory computer-readable medium storing program instructions for text processing, the program instructions being executable by a computing device to effect:

at a training phase;

acquisition of one or more source phrases, each of the source phrase comprising a first set of sequential words, each word of the first set of sequential words being a source word;

acquisition of one or more target phrases, each of the target phrase being in a same language as the source phrases, each of the target phrase comprising of a second set of sequential words being at least partially different from the first set of sequential words of a respective source phrase, each word of the second set of sequential words being a target word;

association, for a given source phrase, of a respective source word feature set with each one of the source words, the respective source word feature set for a given source word comprising;

one or more grammatical features of the given source word; and

a meaning of the given source word;

association, for a respective target phrase, of a respective target word feature set with each one of the target words, the respective source word feature set for a given target word comprising;

one or more grammatical features of the given target word; and

a meaning of the given target word;

analysis of the respective source word feature set of each source words of the given source phrase and the respective target word feature set of each target words of the respective target phrase;

map the given source word of the given source phrase to a corresponding target word of the respective target phrase based on a similarity of the source word feature set of the given source word to the target word feature set of the corresponding target word;

based on the mapping, generation of one or more phrase transformation rules applicable to the given source phrase to transform the first set of sequential words into the second set of sequential words of the respective target phrase;

storing the one or more source phrases and the associated one or more generated phrase transformation rules in a memory of the computing device;

at an in-use phase;

acquisition of a text phrase, the text phrase comprising a third set of sequential words being at least partially different from the first set of sequential words;

retrieving the one or more source phrases from the memory;

perform at least one of a grammatical analysis and a semantic analysis of the text phrase and the one or more stored source phrases to determine similarity of the text phrase to the one or more stored source phrases;

upon determining that the text phrase has the similarity to the given source phrase greater than a threshold, applying the associated one or more phrase transformation rules to the text phrase to generate a transformed text phrase, the transformed text phrase comprising a fourth set of sequential words being at least partially similar to the second set of sequential words of the respective target phrase.

9. The medium of claim 8, wherein:

determination of the similarity of each one of the source word feature sets to each one of the target word feature sets comprises comparison of the respective grammatical features of each one of the source words to the respective grammatical features of each one of the target words; and

determination of the similarity of each one of the source word feature sets to each one of the target word feature sets comprises determination of a similarity of the respective meaning of each one of the source words to the respective meaning of each one of the target words.

10. The medium of claim 9, wherein generation of the one or more phrase transformation rules applicable to the given source phrase to transform the given source phrase into the respective target phrase comprises generation of a second phrase transformation rule applicable to the source phrase to delete at least one word from the given source phrase.

11. The medium of claim 9, wherein generation of the one or more phrase transformation rules applicable to the given source phrase to transform the given source phrase into the respective target phrase comprises generation of a third phrase transformation rule applicable to the source phrase to add at least one word to the given source phrase.

12. The medium of claim 8, wherein:

analysis of the source word feature sets and the target word feature sets to determine a mapping from the source words to the target words further comprises selection of a most similar one of the target word feature sets to one of the source word feature sets; and

generation of the one or more phrase transformation rules applicable to the given source phrase to transform the given source phrase into the respective target phrase comprises generation of a first phrase transformation rule applicable to the given source phrase to transform a source word associated with the one of the source word feature sets into a target word associated with the most similar one of the target word feature sets.

13. The medium of claim 12, wherein:

the source word associated with the one of the source word feature sets and the target word associated with the most similar one of the target word feature sets share a same root; and

the first phrase transformation rule is applicable to the given source phrase to modify a form of the source word to a form of the target word.

14. A non-transitory computer-readable medium storing one or more source phrases, and one or more phrase transformation rules associated with each of the one or more source phrases, and program instructions, the one or more phrase transformation rules having been generated based on an analysis of feature sets including (i) a source word feature set associated with each source word comprising a first set of sequential words forming a given source phrase, the source word feature for a given source word comprising one or more grammatical features of the given source word and a meaning of the given source word, and (ii) a target word feature set associated with each target word comprising a second set of sequential words forming a respective target phrase, the target word feature for a given target word comprising one or more grammatical features of the given target word and a meaning of the given target word, the target phrase being in a same language as the source phrase, the analysis comprising mapping the given source word to a corresponding target word based on a similarity of the source word feature set of the given source word to the target word feature set of the corresponding target word, the program instructions executable by a computing device to effect:

acquisition of a text phrase, the text phrase comprising a third set of sequential words being at least partially different from the first set of sequential words;

retrieve the one or more source phrases;

perform at least one of a grammatical analysis and a semantic analysis of the text phrase and the one or more stored source phrases to determine similarity of the text phrase to the one or more stored source phrases;

upon determination that the similarity of the text phrase to the given source phrase is greater than a threshold, applying of the associated one or more phrase transformation rules to the text phrase to generate a transformed text phrase, the transformed text phrase being at least partially similar to the second set of sequential words of the respective target phrase.