Linguistically-adapted structural query annotation

• US 8,812,301 B2
• Filed: 09/26/2011
• Issued: 08/19/2014
• Est. Priority Date: 09/26/2011
• Status: Active Grant

• Current Assignee: Xerox Corporation (Xerox Holdings Corp.)
• Original Assignee: Xerox Corporation (Xerox Holdings Corp.)
• Inventors: Brun, Caroline, Nikoulina, Vassilina, Lagos, Nikolaos
• Primary Examiner(s): GUERRA-ERAZO, EDGAR X
• Application Number:
  US13/245,147
• Publication Number:
  US 20130080152A1
• Time in Patent Office: 1,058 Days
• Field of Search: 704/1, 704/4, 704/5, 704/9, 704/10, 704/270, 704/270.1, 704/275
• US Class Current: 704/9
• CPC Class Codes: G06F 16/3335 Syntactic pre-processing, e.g. stopword elimination, stemming
• CPC Class Codes: G06F 16/3337 Translation of the query language, e.g. Chinese to English
• CPC Class Codes: G06F 16/3344 using natural language analysis
• CPC Class Codes: G06F 40/211 Syntactic parsing, e.g. based on context-free grammar [CFG] or unification grammars
• CPC Class Codes: G06F 40/242 Dictionaries
• CPC Class Codes: G06F 40/284 Lexical analysis, e.g. tokenisation or collocates
• CPC Class Codes: G06F 40/289 Phrasal analysis, e.g. finite state techniques or chunking
• CPC Class Codes: G06F 40/295 Named entity recognition
• CPC Class Codes: G06F 40/44 Statistical methods, e.g. probability models

Abstract

A system and method for natural language processing of queries are provided. A lexicon includes text elements that are recognized as being a proper noun when capitalized. A natural language query includes a sequence of text elements including words. The query is processed. The processing includes a preprocessing step, in which part of speech features are assigned to the text elements in the query. This includes identifying, from a lexicon, a text element in the query which starts with a lowercase letter and assigning recapitalization information to the text element in the query, based on the lexicon. This information includes a part of speech feature of the capitalized form of the text element. Then parts of speech for the text elements in the query are disambiguated, which includes applying rules for recapitalizing text elements based on the recapitalization information.

Claims

1. A method for processing queries, comprising:

providing access to a lexicon in which a set of text elements that each start with a lowercase letter are each recognized in the lexicon as being a proper noun when in a capitalized form, each of text elements in the set of text elements starting with a lowercase letter being linked in the lexicon to a respective capitalized form;

receiving a natural language query to be processed, the query comprising a sequence of text elements, the text elements comprising words;

with a computer processor, processing the query comprising;

assigning part of speech (POS) features from a list of POS features to the text elements in the query, including;

for a text element in the query which starts with a lowercase letter and which is among the set of text elements in the lexicon that are recognized as being a proper noun when in a capitalized form, assigning recapitalization information to the query text element, the recapitalization information comprising a part of speech feature of the capitalized form;

disambiguating part of speech features for the text elements in the query including applying rules for recapitalizing text elements based on the recapitalization information; and

chunking the disambiguated query; and

outputting the processed query.

2. The method of claim 1, further comprising translating the chunked, disambiguated query and wherein the out putting comprises outputting the translated query.

3. The method of claim 1, wherein the outputting of the processed query includes outputting the processed query to a client device for querying a document collection.

4. The method of claim 1 wherein the recapitalization information comprises information that the capitalized form has one of a person name and place name feature.

5. The method of claim 1, wherein the assigning includes applying rules for associating recapitalization information with a word which is unknown in the lexicon.

6. The method of claim 5, wherein the rules include a rule which, for word that is unknown in the lexicon and which starts with a lower case letter which is preceded by a word recognized in the lexicon as being a first name, the unknown word is associated with feature corresponding to a last name.

7. The method of claim 5, wherein the rules include a rule which, for word that is unknown in the lexicon and which starts with a lower case letter which is preceded by a first name and thereafter by a particle, the unknown word is associated with feature corresponding to a last name.

8. The method of claim 1, wherein the disambiguating part of speech features for the text elements in the query comprises applying the rules for recapitalization, whereby a text element starting with a lowercase letter is capitalized in the query and associated with a part of speech when one of the rules is satisfied.

9. The method of claim 8, wherein at least one of the recapitalization rules is language specific.

10. The method of claim 1, wherein the part of speech disambiguation includes applying contextual rules which favor assigning nouns rather than verbs as part of speech features.

11. The method of claim 1, wherein the query natural language is English and the part of speech disambiguation includes applying at least one contextual rule selected from:

a. a rule which in the case of ambiguity between a verb and a noun for a first word of the query, selects the noun as the part of speech for the first word;

b. a rule which in the case of ambiguity between a verb and a noun for a second word of the query, selects the noun as the part of speech for the second word if the query starts with an adjective or a noun;

c. a rule which in the case of ambiguity between a verb and a noun for a word of the query, selects the noun as the part of speech for the word if the word is followed by a coordinating conjunction and a noun or if it is preceded by a noun and a coordinating conjunction; and

d. a rule which in the case of ambiguity between an adjective and a past participle verb, selects the adjective as the part of speech if the word is followed by a noun.

12. The method of claim 1, wherein the query natural language is French and the part of speech disambiguation includes applying at least one contextual rule selected from:

a. a rule which in the case of ambiguity between a verb and a noun for the first word of the query, selects the noun as the part of speech for the first word;

b. a rule which in the case of ambiguity between a verb and a noun for a word of the query other than a first word of the query, selects the noun as the part of speech for the word if there is no person agreement with one of the previous nouns in the query; and

c. a rule which in the case of ambiguity between a verb and a noun for a word of the query, wherein the verb is neither in the past participle form nor the infinitive form, selects the noun as the part of speech for the word if it is not followed by a determiner.

13. The method of claim 1, wherein the chunking comprises, for a text element which is recognized in the lexicon as being a named entity, applying a rule whereby a chunk includes solely the named entity, without other words.

14. The method of claim 1, wherein the chunking comprises providing a rule for attaching a prepositional phrase or adjectival phrase to the closest antecedent that is morphologically compatible with it.

15. The method of claim 1, wherein the providing of the lexicon includes providing a lexicon in which text elements are each linked to a lemma form of the text element and in which lowercase text elements which are known to be a proper noun when capitalized are linked to the same lemma form as their capitalized form.

16. A system for performing the method of claim 1 comprising memory which stores instructions for performing the method and a processor in communication with the memory for executing the instructions.

17. A computer program product comprising a non-transitory recording medium storing instructions, which when executed by a computer, perform the method of claim 1.

18. A system for processing queries, comprising:

a lexicon which stores a set of text elements that start with a lowercase letter that are indexed as being proper nouns when in a capitalized form in which the first letter is capitalized;

memory which receives a natural language query to be processed, the query comprising a sequence of text elements, the text elements comprising words;

a linguistic analysis component for processing the query, comprising;

a preprocessing component for preprocessing the query, the preprocessing component assigning part of speech features to the text elements in the query, the assigning including assigning recapitalization information to the text elements which are in the set of text elements, the recapitalization information comprising proper noun information of the capitalized form of the text element;

a part of speech disambiguation component for disambiguating part of speech features for the text elements in the query, the part of speech disambiguation component applying rules for recapitalizing text elements based on the recapitalization information; and

a chunking component for chunking the disambiguated query; and

a processing component for implementing the linguistic analysis component.

19. The system of claim 18, further comprising a translation component for translating the processed query into a natural language other than the natural language of the input query.

20. The system of claim 19, further comprising memory which stores a list of named entities and their translations into the other natural language.

21. The system of claim 18, further comprising an information retrieval component which retrieves the processed query and retrieves documents that are responsive to the processed query.

22. A method for processing queries, comprising:

generating a lexicon comprising a set of text elements that start with a lowercase letter and which are recognized as being proper nouns when in a capitalized form of the text element, each of text elements in the set of text elements starting with a lowercase letter being linked in the lexicon to a respective capitalized form;

receiving a natural language query to be processed, the query comprising a sequence of text elements, the text elements comprising words;

with a computer processor, processing the query comprising;

assigning part of speech features to the text elements in the query, the assigning including;

for a text element in the query which is among the set of text elements in the lexicon which are recognized as being proper nouns when in a capitalized form, assigning recapitalization information to the text element, the recapitalization information comprising a part of speech feature of the capitalized form of the text element;

disambiguating any ambiguous part of speech features for the text elements in the query, comprising applying rules for recapitalizing text elements based on the recapitalization information; and

outputting the processed query.