Automatic context sensitive language generation, correction and enhancement using an internet corpus

• US 8,645,124 B2
• Filed: 02/04/2009
• Issued: 02/04/2014
• Est. Priority Date: 08/01/2007
• Status: Active Grant

• Current Assignee: Ginger Software, Inc.
• Original Assignee: Ginger Software, Inc.
• Inventors: Karov Zangvil, Yael
• Primary Examiner(s): Godbold, Douglas
• Assistant Examiner(s): ESTES, ERNEST LANGSTON
• Application Number:
  US13/056,563
• Publication Number:
  US 20110184720A1
• Time in Patent Office: 1,826 Days
• Field of Search: 704 1- 10
• US Class Current: 704/9
• CPC Class Codes: G06F 40/10 Text processing natural language analysis G06F40/20; semantic analysis G06F40/30; processing or translation of natural language G06F40/40
• CPC Class Codes: G06F 40/232 Orthographic correction, e.g. spell checking or vowelisation
• CPC Class Codes: G06F 40/253 Grammatical analysis; Style critique
• CPC Class Codes: G06F 40/263 Language identification
• CPC Class Codes: G06F 40/40 Processing or translation of natural language natural language analysis G06F40/20; semantic analysis G06F40/30
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence

Abstract

A computer-assisted language generation system including sentence retrieval functionality, operative on the basis of an input text containing words, to retrieve from an internet corpus a plurality of sentences containing words which correspond to the words in the input text and sentence generation functionality operative using a plurality of sentences retrieved by the sentence retrieval functionality from the internet corpus to generate at least one correct sentence giving expression to the input text.

Claims

1. A computer-assisted language generation system comprising:

sentence retrieval functionality, operative on the basis of an input text containing words, to retrieve from an internet corpus a plurality of sentences containing words which correspond to said words in the input text; and

sentence generation functionality operative using said plurality of sentences to generate at least one correct sentence giving expression to the input text,

said sentence generation functionality comprising;

sentence simplification functionality operative to simplify said sentences retrieved from said internet corpus by;

classifying all words of each of said sentences retrieved from said internet corpus as being one of unnecessary and mandatory, said unnecessary words comprising words which are at least one of adjectives, adverbs, articles, prepositions, predefined list of unnecessary words, said mandatory words comprising words which are not unnecessary words;

deleting unnecessary words from each of said sentences;

ascertaining, for each phrase of each of said sentences, whether said phrase comprises at least one of at least one word in said input text and at least one alternative word of at least one word in said input text; and

responsive to ascertaining that said phrase does not comprise at least one of at least one word in said input text and at least one alternative word of at least one word in said input text, deleting said phrase from said sentence;

simplified sentence grouping functionality for grouping similar simplified sentences provided by said sentence simplification functionality;

simplified sentence group ranking functionality for ranking groups of said similar simplified sentences; and

sentence selection functionality operative to select at least one of said similar simplified sentences as said at least one correct sentence based on said ranking.

2. A computer-assisted language generation system according to claim 1 and wherein said sentence retrieval functionality comprises:

an independent phrase generator splitting said input text into one or more independent phrases;

a word stem generator and classifier, operative for each independent phrase to generate word stems for words appearing therein and to assign importance weights thereto; and

an alternatives generator for generating alternative word stems corresponding to said word stems.

3. A computer-assisted language generation system according to claim 2 and also comprising a stem to sentence index which interacts with said internet corpus for retrieving said plurality of sentences containing words which correspond to said words in the input text.

4. A computer-assisted language generation system according to claim 1 and wherein said simplified sentence group ranking functionality operates using at least some of the following criteria:

A. the number of simplified sentences contained in a group;

B. degree to which the word stems of the words in the group correspond to the word stems in an independent phrase and their alternatives;

C. the extent to which the group includes words which do not correspond to the words in the independent phrase and their alternatives.

5. A computer-assisted language generation system according to claim 4 and wherein said simplified sentence group ranking functionality operates using at least part of the following procedure:

defining the weight of a word stem, to indicate the importance of the word in the language;

calculating a Positive Match Rank corresponding to criterion B;

calculating a Negative Match Rank corresponding to criterion C;

calculating a Composite Rank based on;

said number of simplified sentences contained in a group and corresponding to criterion A;

said Positive Match Rank; and

said Negative Match Rank.

6. A computer assisted language generation system according to claim 5 and also comprising:

machine translation functionality providing said input text.

7. A computer-assisted language generation system according to claim 1 and also comprising text correction functionality.

8. A computer-assisted language generation system according to claim 1 and also comprising:

sentence search functionality providing said input text based on user-entered query words.

9. A computer-assisted language generation system according to claim 1 and also comprising:

speech-to-text conversion functionality providing said input text.

10. A system according to claim 1 and also comprising:

at least one of;

spelling correction functionality;

misused word correction functionality;

grammar correction functionality; and

vocabulary enhancement functionality; and

contextual feature-sequence functionality cooperating with at least one of said spelling correction functionality;

said misused word correction functionality, grammar correction functionality and said vocabulary enhancement functionality and employing an internet corpus.

11. A system according to claim 1 and also comprising:

a misused-word suspector evaluating, words in said input text;

an alternatives generator, generating multiple alternatives for at least some of the words in said input text evaluated as suspect words by said suspector, at least one of said multiple alternatives for a word in said input text being consistent with a contextual feature of said word in said input text in an internet corpus;

a selector for selecting among at least said multiple alternatives; and

a correction generator operative to provide a correction output based at least partially on a selection made by said selector.

12. A system according to claim 1 and also comprising:

at least one of a spelling correction module, a misused-word correction module, a grammar correction module and a vocabulary enhancement module receiving said input text and providing a correction output, each of said at least one of a spelling correction module, a misused-word correction module, a grammar correction module and a vocabulary enhancement module including;

an alternative word candidate generator including;

phonetic similarity functionality operative to propose alternative words based on phonetic similarity to a word in said input text and to indicate a metric of phonetic similarity; and

character string similarity functionality operative to propose alternative words based on character string similarity to a word in said input text and to indicate a metric of character string similarity for each alternative word; and

a selector operative to select either a word in said output or an alternative word candidate proposed by said alternative word candidate generator by employing said phonetic similarity and character string similarity metrics together with context-based selection functionality.

13. A system according to claim 1 and also comprising:

suspect word identification functionality, receiving said input text and providing a suspect word output which indicates suspect words;

feature identification functionality operative to identify features including said suspect words;

an alternative selector identifying alternatives to said suspect words;

feature occurrence functionality employing a corpus and providing an occurrence output, ranking various features including said alternatives as to their frequency of use in the corpus; and

a selector employing said occurrence output to provide a correction output,

said feature identification functionality comprising feature filtration functionality including at least one of;

functionality for eliminating features containing suspected errors;

functionality for negatively biasing features which contain words introduced in an earlier correction iteration of said input text and which have a confidence level below a confidence level predetermined threshold; and

functionality for eliminating features which are contained in another feature having an frequency of occurrence above a predetermined frequency threshold.

14. A system according to claim 1 and also comprising:

suspect word identification functionality, receiving said input text and providing a suspect word output which indicates suspect words;

feature identification functionality operative to identify features including said suspect words;

an alternative selector identifying alternatives to said suspect words;

occurrence functionality employing a corpus and providing an occurrence output, ranking features including said alternatives as to their frequency of use in the corpus; and

a correction output generator, employing said occurrence output to provide a correction output,

said feature identification functionality comprising;

at least one of;

N-gram identification functionality; and

co-occurrence identification functionality; and

at least one of;

skip-gram identification functionality;

switch-grain identification functionality; and

previously used by user feature identification functionality.

15. A system according to claim 1 and also comprising:

a grammatical error suspector evaluating words in said input text;

an alternatives generator, generating multiple alternatives for at least some of the words in said input text evaluated as suspect words by said suspector, at least one of said multiple alternatives for a word in input text being consistent with a contextual feature of said word in said input text;

a selector for selecting among at least said multiple alternatives; and

a correction generator operative to provide a correction output based at least partially on a selection made by said selector.

16. A system according to claim 15 and also comprising context based scoring of various alternative corrections, based at least partially on contextual feature-sequence (CFS) frequencies of occurrences in an internet corpus.

17. A system according to claim 1 and also comprising vocabulary enhancement functionality including:

vocabulary-challenged words identification functionality;

alternative vocabulary enhancements generation functionality; and

context based scoring functionality, based at least partially on contextual feature-sequence (CFS) frequencies of occurrences in an internet corpus,

said alternative vocabulary enhancements generation functionality including thesaurus pre-processing functionality operative to generate alternative vocabulary enhancements.

18. A system according to claim 1 and also comprising:

a punctuation error suspector evaluating at least some of the words and punctuation in said input text on the basis of their fit within a context of the language input based on frequency of occurrence of feature-grams of said input text in an internet corpus; and

a correction generator operative to provide a correction output based at least partially on an evaluation performed by said suspector.