Text segmentation with multiple granularity levels

• US 8,892,420 B2
• Filed: 11/17/2011
• Issued: 11/18/2014
• Est. Priority Date: 11/22/2010
• Status: Active Grant

• Current Assignee: Alibaba Group Holding Ltd.
• Original Assignee: Alibaba Group Holding Ltd.
• Inventors: Sun, Jian, Hou, Lei, Tang, Jing Ming, Chu, Min, Liao, Xiao Ling, Xu, Bing Jing, Peng, Ren Gang, Yang, Yang
• Primary Examiner(s): Vo, Huyen X.
• Application Number:
  US13/298,941
• Publication Number:
  US 20120130705A1
• Time in Patent Office: 1,097 Days
• Field of Search: 704 1- 10, 704/251, 704/255, 704/257, 704/270, 704/270.1, 704/254, 707/739
• US Class Current: 704/9
• CPC Class Codes: G06F 40/211 Syntactic parsing, e.g. based on context-free grammar [CFG] or unification grammars
• CPC Class Codes: G06F 40/284 Lexical analysis, e.g. tokenisation or collocates
• CPC Class Codes: G06F 40/30 Semantic analysis
• CPC Class Codes: G06F 40/40 Processing or translation of natural language natural language analysis G06F40/20; semantic analysis G06F40/30
• CPC Class Codes: G06F 40/53 Processing of non-Latin text kana-to-kanji conversion G06F40/129; vowelisation G06F40/232

Abstract

Text processing includes: segmenting received text based on a lexicon of smallest semantic units to obtain medium-grained segmentation results; merging the medium-grained segmentation results to obtain coarse-grained segmentation results, the coarse-grained segmentation results having coarser granularity than the medium-grained segmentation results; looking up in the lexicon of smallest semantic units respective search elements that correspond to segments in the medium-grained segmentation results; and forming fine-grained segmentation results based on the respective search elements, the fine-grained segmentation results having finer granularity than the medium-grained segmentation results.

Claims

1. A method of text processing, comprising:

training, using a processor, a classifier for classifying text, wherein;

the training is based on a plurality of training sample entries;

a training sample entry in the plurality of training sample entries includes;

a character count;

an independent use rate;

a phrase structure rule value indicating whether the training sample entry complies with phrase structure rules;

a semantic attribute value indicating an inclusion state of the training sample entry in a predetermined set of enumerated entries;

an overlap attribute value indicating overlap of the training sample entry with another entry in the predetermined set of enumerated entries; and

a classification result indicating whether the training sample entry is a compound semantic unit or a smallest semantic unit;

building, using the processor, a lexicon of smallest semantic units, comprising;

receiving an entry to be classified;

using the trained classifier to determine whether the entry to be classified is a smallest semantic unit or a compound semantic unit; and

in the event that the entry is determined to be a smallest semantic unit, adding the entry to the lexicon of smallest semantic units;

segmenting, using the processor, received text based on the lexicon of smallest semantic units to obtain medium-grained segmentation results;

merging, using the processor, the medium-grained segmentation results to obtain coarse-grained segmentation results, the coarse-grained segmentation results having coarser granularity than the medium-grained segmentation results;

looking up, using the processor, in the lexicon of smallest semantic units respective search elements that correspond to segments in the medium-grained segmentation results; and

forming, using the processor, fine-grained segmentation results based on the respective search elements, the fine-grained segmentation results having finer granularity than the medium-grained segmentation results.

2. The method of claim 1, wherein the received text is in a non-divider marked language.

3. The method of claim 1 further comprising, in the event that the entry is determined to be a compound semantic unit, adding the entry to a lexicon of compound semantic units.

4. The method of claim 1, wherein using the trained classifier to determine whether the entry is a smallest semantic unit or a compound semantic unit includes inputting into the trained classifier:

a character count of the entry, an independent use rate of the entry, a phrase structure rule indicator indicating whether the entry complies with phrase structure rules, a semantic attribute indicating an inclusion state of the entry in the predetermined set of enumerated entries, and an overlap attribute of the entry.

5. The method of claim 1, further comprising:

determining a search element that corresponds to the entry; and

saving the search element in the lexicon of smallest semantic units.

6. The method of claim 5, wherein determining a search element that corresponds to the entry includes:

determining whether the entry is segmentable;

in the event that the entry is segmentable, setting the search element to a fine-grained word included within the entry; and

in the event that the entry is not segmentable, setting the search element to the entry.

7. The method of claim 1, wherein segmenting received text based on the lexicon of smallest semantic units to obtain medium-grained segmentation results include resolving an ambiguity of the medium-grained segmentation results.

8. A system for text processing, comprising:

one or more processors configured to;

train a classifier for classifying text, wherein;

the training is based on a plurality of training sample entries;

a training sample entry in the plurality of training sample entries includes;

a character count;

an independent use rate;

a phrase structure rule value indicating whether the training sample entry complies with phrase structure rules;

a semantic attribute value indicating an inclusion state of the training sample entry in a predetermined set of enumerated entries;

an overlap attribute value indicating overlap of the training sample entry with another entry in the predetermined set of enumerated entries; and

a classification result indicating whether the training sample entry is a compound semantic unit or a smallest semantic unit; and

build a lexicon of smallest semantic units, comprising;

receiving an entry to be classified;

using the trained classifier to determine whether the entry to be classified is a smallest semantic unit or a compound semantic unit; and

in the event that the entry is determined to be a smallest semantic unit, adding the entry to the lexicon of smallest semantic units;

segment received text based on the lexicon of smallest semantic units to obtain medium-grained segmentation results;

merge the medium-grained segmentation results to obtain coarse-grained segmentation results, the coarse-grained segmentation results having coarser granularity than the medium-grained segmentation results;

look up in the lexicon of smallest semantic units respective search elements that correspond to segments in the medium-grained segmentation results; and

form fine-grained segmentation results based on the respective search elements, the fine-grained segmentation results having finer granularity than the medium-grained segmentation results; and

one or more memories coupled to the one or more processors, configured to provide the one or more processors with instructions.

9. The system of claim 8, wherein the text is in a non-divider marked language.

10. The system of claim 8, wherein the one or more processors are further configured to, in the event that the entry is determined to be a compound semantic unit, add the entry to a lexicon of compound semantic units.

11. The system of claim 8, wherein using the trained classifier to determine whether the entry is a smallest semantic unit or a compound semantic unit includes inputting into the trained classifier:

a character count of the entry, an independent use rate of the entry, a phrase structure rule indicator indicating whether the entry complies with phrase structure rules, a semantic attribute indicating an inclusion state of the entry in the predetermined set of enumerated entries, and an overlap attribute of the entry.

12. The system of claim 8, wherein the one or more processors are further configured to:

determine a search element that corresponds to the entry; and

save the search element in the lexicon of smallest semantic units.

13. The system of claim 12, wherein determining a search element that corresponds to the entry includes:

determining whether the entry is segmentable;

in the event that the entry is segmentable, setting the search element to a fine-grained word included within the entry; and

in the event that the entry is not segmentable, setting the search element to the entry.

14. The system of claim 8, wherein segmenting received text based on the lexicon of smallest semantic units to obtain medium-grained segmentation results includes resolving an ambiguity of the medium-grained segmentation results.

15. A computer program product for text processing, the computer program product being embodied in a non-transitory computer readable storage medium and comprising computer instructions for:

training a classifier for classifying text, wherein;

the training is based on a plurality of training sample entries;

a training sample entry in the plurality of training sample entries includes;

a character count;

an independent use rate;

a phrase structure rule value indicating whether the training sample entry complies with phrase structure rules;

a semantic attribute value indicating an inclusion state of the training sample entry in a predetermined set of enumerated entries;

an overlap attribute value indicating overlap of the training sample entry with another entry in the predetermined set of enumerated entries; and

a classification result indicating whether the training sample entry is a compound semantic unit or a smallest semantic unit; and

building a lexicon of smallest semantic units, comprising;

receiving an entry to be classified;

using the trained classifier to determine whether the entry to be classified is a smallest semantic unit or a compound semantic unit; and

in the event that the entry is determined to be a smallest semantic unit, adding the entry to a lexicon of smallest semantic units;

segmenting received text based on the lexicon of smallest semantic units to obtain medium-grained segmentation results;

merging the medium-grained segmentation results to obtain coarse-grained segmentation results, the coarse-grained segmentation results having coarser granularity than the medium-grained segmentation results;

looking up in the lexicon of smallest semantic units respective search elements that correspond to segments in the medium-grained segmentation results; and

forming fine-grained segmentation results based on the respective search elements, the fine-grained segmentation results having finer granularity than the medium-grained segmentation results.

16. A system for text processing, comprising:

one or more processors configured to;

train a classifier for classifying text, wherein;

the training is based on a plurality of training sample entries;

a training sample entry in the plurality of training sample entries includes;

a character count;

an independent use rate;

a phrase structure rule value indicating whether the training sample entry complies with phrase structure rules;

a semantic attribute value indicating an inclusion state of the training sample entry in a predetermined set of enumerated entries;

an overlap attribute value indicating overlap of the training sample entry with another entry in the predetermined set of enumerated entries; and

a classification result indicating whether the training sample entry is a compound semantic unit or a smallest semantic unit; and

build a lexicon of smallest semantic units, comprising;

receiving an entry to be classified;

using the trained classifier to determine whether the entry to be classified is a smallest semantic unit or a compound semantic unit; and

in the event that the entry is determined to be a smallest semantic unit, adding the entry to the lexicon of smallest semantic units; and

one or more memories coupled to the one or more processors, configured to provide the one or more processors with instructions.