Multi-language document search and retrieval system

• US 7,369,987 B2
• Filed: 12/29/2006
• Issued: 05/06/2008
• Est. Priority Date: 11/30/1998
• Status: Expired due to Fees

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Cásseres, David, Loofbourrow, Wayne
• Primary Examiner(s): MCFADDEN, SUSAN IRIS
• Application Number:
  US11/647,229
• Publication Number:
  US 20070106500A1
• Time in Patent Office: 494 Days
• Field of Search: 704/9, 715/536
• US Class Current: 704/9
• CPC Class Codes: G06F 16/313 Selection or weighting of terms for indexing
• CPC Class Codes: G06F 16/3335 Syntactic pre-processing, e.g. stopword elimination, stemming
• CPC Class Codes: G06F 40/268 Morphological analysis
• CPC Class Codes: G06F 40/284 Lexical analysis, e.g. tokenisation or collocates
• CPC Class Codes: G06F 40/289 Phrasal analysis, e.g. finite state techniques or chunking

Abstract

A multi-lingual indexing and search system is presented that performs tokenization and stemming in a manner which is independent of whether index entries and search terms appear as words in a dictionary. The system includes a tokenizer that separates a string of text into individual word tokens, and eliminates predetermined types of tokens from further processing. The system also includes a stemmer that reduces words to grammatical stems by removing known word-endings associated with the various languages to be supported. The stemmer removes known word endings from the word tokens without any effort to guarantee that the remaining stem is contained in a dictionary. In an embodiment, the stemmer only removes those word endings which are associated with nouns. The system further includes an indexer that stores the stems in an index.

Claims

1. A system for indexing textual content in any of a plurality of languages for searching purposes, comprising:

a processing device, comprising;

a tokenizer which separates a string of text into individual word tokens,

a stemmer which reduces the word tokens to grammatical stems by removing word endings which are associated with any one or more of the languages, without regard to whether the remaining stem is a recognized word in any combination of the plurality of languages, and

an indexer which creates an index from the stems; and

a computer-readable medium which stores the created index.

2. The system of claim 1 wherein the word endings which are removed are limited to only those endings which are associated with nouns.

3. The system of claim 1 wherein a word ending is not removed if the resulting stem is less than a predetermined length.

4. The system of claim 3 wherein said predetermined length is four characters.

5. The system of claim 1 wherein the stemmer reduces only once per word token.

6. The system of claim 5 wherein said stemmer reduces by first examining each word token for the longest known endings, and examining the token for successively shorter endings until a known ending is identified in the word token and removed.

7. The system of claim 1 wherein the stemmer and index disregard stopwords, wherein stopwords are words which occur with relatively high frequency in at least one of said languages and which are not also significant nouns in another one of said languages.