System and method for managing a textual archive using semantic units

• US 7,092,870 B1
• Filed: 09/15/2000
• Issued: 08/15/2006
• Est. Priority Date: 09/15/2000
• Status: Expired due to Term

• Current Assignee: Nuance Communications, Inc. (Microsoft Corporation)
• Original Assignee: International Business Machines Corporation
• Inventors: Chen, Julian C., Zadrozny, Wlodek W., Kanevsky, Dimitri
• Primary Examiner(s): Dorvil, Richemond
• Assistant Examiner(s): ALBERTALLI, BRIAN LOUIS
• Application Number:
  US09/663,812
• Time in Patent Office: 2,160 Days
• Field of Search: 704/10, 704/7, 704/9, 715/530, 707/2
• US Class Current: 704/9
• CPC Class Codes: G06F 16/313 Selection or weighting of terms for indexing
• CPC Class Codes: G06F 16/3334 Selection or weighting of terms from queries, including natural language queries
• CPC Class Codes: G06F 40/268 Morphological analysis
• CPC Class Codes: G06F 40/284 Lexical analysis, e.g. tokenisation or collocates
• CPC Class Codes: G06F 40/53 Processing of non-Latin text kana-to-kanji conversion G06F40/129; vowelisation G06F40/232
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence

Abstract

A system and method for indexing and searching textual archives using semantic units such as syllables and morphemes. In one aspect, a system for indexing a textual archive comprises an AHR (automatic handwriting recognition) system and/or OCR (optical character recognition) system for transcribing (decoding) textual input data (handwritten or typed text) into a string of semantic units (e.g., syllables or morphemes) using a statistical language model and vocabulary based on semantic units (such as syllables or morphemes). The string of semantic units that result from a decoding process are stored in a semantic unit database and indexed with pointers to the corresponding textual data in the textual archive. In another aspect, a system for searching a textual archive is provided, wherein a word (or words) to be searched is rendered into a string of semantic units (e.g., syllables or morphemes) depending on the application. A search engine then compares the string of semantic units (resulting from the input query) against the decoded semantic unit database, and then identifies textual data stored in the textual archive using the indexes that were generated during a semantic unit-based indexing process.

Claims

1. A method for managing a textual database, comprising the steps of:

receiving textual data;

identifying a data type of the textual data;

transcribing the textual data into corresponding semantic units of words using a recognition system for the identified data type, wherein the recognition system performs transcription by decoding the textual data using a language model and phonetic dictionary of semantic units;

storing the textual data in a textual database; and

generating an index based on semantic units of words for indexing the stored textual data with the corresponding semantic units.

2. The method of claim 1, wherein the semantic units comprise syllables.

3. The method of claim 1, wherein the semantic units comprise morphemes.

4. The method of claim 1, wherein the textual data is associated with audio data, and wherein the step of indexing further comprises indexing the audio data with the semantic units.

5. The method of claim 1, wherein the step of transcribing comprises the step of time-stamping the semantic units.

6. The method of claim 1, wherein the step of generating an index comprises generating a hierarchical index wherein a semantic unit index points to one or more data modes.

7. The method of claim 6, wherein the one or more modes of data comprises words and pictures.

8. The method of claim 1, further comprising the step of converting the index into a universal index which cross-references characters of different fonts.

9. The method of claim 1, further comprising the step of searching the textual database for target textual data using the semantic unit index.

10. The method of claim 9, further comprising the step of converting a target word into a string of semantic units to perform the searching step.

11. The method of claim 10, wherein the step of converting a target word is performed automatically using a character-to-semantic unit mapping table.

12. The method of claim 9, further comprising the step of displaying search results, wherein the target textual data is displayed starting from a corresponding semantic unit in a user query and commencing one of forward and backward for a given length based on a user request.

13. The method of claim 1, wherein identifying a data type of the textual data comprises identifying data types including handwritten text and typed text.

14. The method of claim 13, wherein the different data types include handwritten text or typed text of different font or styles of a given language.

15. The method of claim 13, wherein transcribing textual data comprises decoding handwritten text using an AHR (automatic handwriting recognition) system and decoding typed text using an OCR (optical character recognition) system.

16. The method of claim 13, wherein generating an index based on semantic units of words comprises indexing the semantic units to stored handwritten textual data based on handwriting biometric data.

17. The method of claim 1, wherein generating an index based on semantic units of words comprises generating separate indexes for indexing the semantic units to stored textual data of different data types.

18. The method of claim 17, further comprising generating a universal index for indexing the semantic units to stored textual data of a plurality of different data types.

19. A program storage device readable by a machine, tangibly embodying a program of instructions executable by the machine to perform method steps for managing a textual database, the method comprising the steps of:

receiving textual data;

identifying a data type of the textual data;

transcribing the textual data into corresponding semantic units of words using a recognition system for the identified data type, wherein the recognition system performs transcription by decoding the textual data using a language model and phonetic dictionary of semantic units;

storing the textual data in a textual database; and

generating an index based on semantic units of words for indexing the stored textual data with the corresponding semantic units.

20. A system for managing a textual database, comprising:

a data type identification system for identifying a data type of textual data;

a recognition system for transcribing the textual data into corresponding semantic units of words based on the identified data type, wherein the text recognition system performs transcription by decoding the textual data using a language model and phonetic dictionary of semantic units, wherein the recognition system comprises an OCR (optical character recognition) system for transcribing typed text, and an AHR (automatic handwriting recognition) system for transcribing handwritten text;

a textual database for storing the textual data; and

an index generator adapted to generate an index based on semantic units of words, wherein the textual data stored in the textual database is indexed with the corresponding semantic units.

21. The system of claim 20, further comprising an index converter adapted to convert the index into a universal index which cross-references characters of different fonts for a given language.

22. The system of claim 20, further comprising:

a query processor adapted to transform a search query for target textual data into corresponding semantic units; and

a search engine for searching the textual database based on the semantic units corresponding to the search query.

23. The system of claim 22, further comprising an automatic word boundary marking system that is applied to a search query.