UNSUPERVISED STEMMING SCHEMA LEARNING AND LEXICON ACQUISITION FROM CORPORA

• US 20090150415A1
• Filed: 12/10/2007
• Published: 06/11/2009
• Est. Priority Date: 12/10/2007
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Dayan, Yigal Shai, Cohen, Daniel, Mazel, Victoria, Magdalen, Josemina Marcella
• Granted Patent: US 7,912,703 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/36 Creation of semantic tools, e.g. ontology or thesauri

Abstract

Illustrated embodiments provide a computer implemented method, an apparatus, and a computer program product for unsupervised stemming schema learning and lexicon acquisition from corpora. In one illustrative embodiment, the computer implemented method obtains a corpus from corpora, analyzes the corpus to deduce a set of possible stemming schema and reviews and revises the set of possible stemming schema, to create a pruned set of stemming schema. The computer implemented method further deduces a lexicon from the corpus using the pruned set of stemming schema.

Claims

1. A computer implemented method for unsupervised stemming schema learning and lexicon acquisition from corpora, the computer implemented method comprising;

obtaining a corpus from a corpora;

analyzing the corpus to deduce a set of possible stemming schema;

reviewing and revising the set of possible stemming schema to create a pruned set of stemming schema; and

deducing a lexicon from the corpus using the pruned set of stemming schema.

2. The computer implemented method of claim 1, wherein analyzing the corpus to deduce a set of possible stemming schema further comprises:

identifying frequent concepts;

identifying frequent affixes from the frequent concepts;

identifying word splits using the frequent affixes to create affix and kernel pairs; and

creating stemming schema for each unique affix and kernel pair.

3. The computer implemented method of claim 1, wherein analyzing the corpus to deduce a set of possible stemming schema further comprises:

generating an affix count in concepts;

generating an affix count in schemas; and

generating schema scores, from a combination of affix counts, for the stemming schemas to identify useful stemming schema.

4. The computer implemented method of claim 1, wherein analyzing the corpus to deduce a set of possible stemming schema further comprises:

identifying a best stemming schema for each kernel in the set of possible stemming schema.

5. The computer implemented method of claim 1, wherein reviewing and revising the set of possible stemming schema to create a pruned set of stemming schema further comprises:

analyzing each stemming schema of the set of possible stemming schema; and

identifying stemming schema having a kernel that has a best schema matching a current stemming schema and keeping the kernel, otherwise deleting the stemming schema.

6. The computer implemented method of claim 2, wherein reviewing and revising the set of possible stemming schema to create a pruned set of stemming schema further comprises:

recalculating the schema scores using a reverse index of the stemming schema for each unique affix and kernel pair.

7. A data processing system for unsupervised stemming schema learning and lexicon acquisition from corpora, the data processing system comprising;

a bus;

a memory connected to the bus;

a persistent storage connected to the bus, wherein the persistent storage having computer executable program code embodied therein;

a communications unit connected to the bus;

a display connected to the bus; and

a processor unit connected to the bus, wherein the processor unit executes the computer executable program code directing the data processing to;

obtain a corpus from a corpora;

analyze the corpus to deduce a set of possible stemming schema;

review and revise the set of possible stemming schema to create a pruned set of stemming schema; and

deduce a lexicon from the corpus using the pruned set of stemming schema.

8. The data processing system of claim 7, wherein the processor unit executes the computer executable program code directing the data processing to analyze the corpus to deduce a set of possible stemming schema further comprising:

identifying frequent concepts;

identifying frequent affixes from the frequent concepts;

identifying word splits using the frequent affixes to create affix and kernel pairs; and

creating stemming schema for each unique affix and kernel pair.

9. The data processing system of claim 7, wherein the processor unit executes the computer executable program code directing the data processing to analyze the corpus to deduce a set of possible stemming schema further comprising:

generating an affix count in concepts;

generating an affix count in schemas; and

generating schema scores, from a combination of affix counts, for the stemming schemas to identify useful stemming schema.

10. The data processing system of claim 7, wherein the processor unit executes the computer executable program code directing the data processing system to analyze the corpus to deduce a set of possible stemming schema further comprising:

identifying a best stemming schema for each kernel in the set of possible stemming schema.

11. The data processing system of claim 7, wherein the processor unit executes the computer executable program code directing the data processing to review and revise the set of possible stemming schema to create a pruned set of stemming schema further comprising:

analyzing each stemming schema of the set of possible stemming schema; and

identifying stemming schema having a kernel that has a best schema matching a current stemming schema and keeping the kernel, otherwise deleting the stemming schema.

12. The data processing system of claim 8, wherein the processor unit executes the computer executable program code directing the data processing system to review and revise the set of possible stemming schema to create a pruned set of stemming schema further comprising:

recalculating the schema scores using a reverse index of the stemming schema for each unique affix and kernel pair.

13. A computer program product for unsupervised stemming schema learning and lexicon acquisition from corpora, the computer program product comprising a computer usable recordable medium having computer executable program code tangibly embodied thereon, the computer executable program code comprising;

computer executable program code for obtaining a corpus from a corpora;

computer executable program code for analyzing the corpus to deduce a set of possible stemming schema;

computer executable program code for reviewing and revising the set of possible stemming schema to create a pruned set of stemming schema; and

computer executable program code for deducing a lexicon from the corpus using the pruned set of stemming schema.

14. The computer program product of claim 13, wherein computer executable program code for analyzing the corpus to deduce a set of possible stemming schema further comprises:

computer executable program code for identifying frequent concepts;

computer executable program code for identifying frequent affixes from the frequent concepts;

computer executable program code for identifying word splits using the frequent affixes to create affix and kernel pairs; and

computer executable program code for creating stemming schema for each unique affix and kernel pair.

15. The computer program product of claim 13, wherein computer executable program code for analyzing the corpus to deduce a set of possible stemming schema further comprises:

computer executable program code for generating an affix count in concepts;

computer executable program code for generating an affix count in schemas; and

computer executable program code for generating schema scores, from a combination of affix counts, for the stemming schemas to identify useful stemming schema.

16. The computer program product of claim 13, wherein computer executable program code for analyzing the corpus to deduce a set of possible stemming schema further comprises:

computer executable program code for identifying a best stemming schema for each kernel in the set of possible stemming schema.

17. The computer program product of claim 13, wherein computer executable program code for reviewing and revising the set of possible stemming schema to create a pruned set of stemming schema further comprises:

computer executable program code for analyzing each stemming schema of the set of possible stemming schema; and

computer executable program code for identifying stemming schema having a kernel that has a best schema matching a current stemming schema and keeping the kernel, otherwise deleting the stemming schema.

18. The computer program product of claim 14, wherein computer executable program code for reviewing and revising the set of possible stemming schema to create a pruned set of stemming schema further comprises:

computer executable program code for recalculating the schema scores using a reverse index of the stemming schema for each unique affix and kernel pair.