Enhancing QA system cognition with improved lexical simplification using multilingual resources

• US 10,303,765 B2
• Filed: 06/08/2017
• Issued: 05/28/2019
• Est. Priority Date: 01/02/2017
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Dandala, Bharath, Sinha, Ravi S.
• Primary Examiner(s): Guerra-Erazo, Edgar X
• Application Number:
  US15/617,407
• Publication Number:
  US 20180189264A1
• Time in Patent Office: 719 Days
• CPC Class Codes: G06F 16/243 Natural language query formulation
• CPC Class Codes: G06F 40/157 using dictionaries or tables
• CPC Class Codes: G06F 40/247 Thesauruses; Synonyms
• CPC Class Codes: G06F 40/284 Lexical analysis, e.g. tokenisation or collocates
• CPC Class Codes: G06F 40/55 Rule-based translation

Abstract

An approach is provided that returns a simplified set of text to a user of a natural language processing (NLP) system with the simplified set of text having a complexity appropriate to the reading level of the user. The approach receives a word that belongs to a first natural language and retrieves a first set of complexity data pertaining to the word in the first natural language. The approach translates the word to one or more translated words, with each of the translated words corresponding to one or more second natural languages. The approach then retrieves sets of complexity data, with the sets of complexity data corresponding to a different translated word. The approach determines a complexity of the word in the first natural language based on an analysis of the first and second sets of complexity data.

Claims

1. A method implemented by an information handling system that includes a processor and a memory accessible by the processor, the method comprising:

returning a simplified set of text to a user of a natural language processing (NLP) system, wherein the simplified set of text comprises text appropriate to a reading level of the user, wherein a lexical simplification process selects the simplified set of text from a corpus of plurality of words that have a complexity level appropriate to the reading level, and wherein the complexity level is based on a multi-language word mapping performed on at least a selected one of the plurality of words using a process comprising;

creating the multi-language word mapping by a multi-language word mapping generator executing on the information handling system, wherein the creating further comprises;

retrieving the selected word that belongs to a first natural language;

retrieving a first set of complexity data pertaining to the selected word in the first natural language, wherein the first set of complexity data comprises a first word length and a first word frequency;

translating the selected word to one or more translated words, wherein each of the translated words corresponds to one or more second natural languages;

retrieving one or more second sets of complexity data, wherein each of the second sets of complexity data correspond to a different one of the translated words, and wherein the one or more second sets of complexity data comprises one or more second word lengths and one or more second word frequencies;

computing a complexity of the selected word in the first natural language based on an overall word length and an overall word frequency, wherein the overall word length is based on the first word length and the one or more second word lengths, and wherein the overall word frequency is based on the first word frequency and the one or more second word frequencies; and

storing the computed complexity of the word in the multi-language word mapping; and

wherein the lexical simplification process selects, based on the computed complexity of the word, one of the one or more translated words to replace the selected word.

2. The method of claim 1 wherein the first set of complexity data includes a first word n-gram of the selected word in the first natural language, wherein the second sets of complexity data includes one or more second word n-grams of the selected word in each of the second natural languages, and wherein the method further comprises:

determining an overall word n-gram based on the first word n-gram and the second one or more word n-grams, wherein the complexity of the selected word is based on the overall word n-gram.

3. The method of claim 1 wherein the first set of complexity data includes a first word encyclopedia entry of the selected word in the first natural language, wherein the second sets of complexity data includes one or more second encyclopedia entries of the selected word in each of the second natural languages, and wherein the method further comprises:

determining an overall word n-gram based on the first word encyclopedia entry and the second one or more encyclopedia entries, wherein the complexity of the selected word is based on the overall word n-gram.

4. The method of claim 1 wherein the complexity of the selected word is based on an average length of characters of the selected word and the translated words in each of the first and second natural languages, a total number of translated words, a frequency of the selected word in the first natural language, a sum of the normalized frequencies of the one or more translated words in the second natural languages, an existence of an encyclopedia entry of the selected word, a number of encyclopedia entries of the translated words in the second natural languages, and a vector value of possible character n-grams in the second natural languages collectively.

5. The method of claim 1 wherein the translated words include synonyms of the translated words in the second natural languages.