Linguistically intelligent text compression

• US 20060184351A1
• Filed: 04/04/2006
• Published: 08/17/2006
• Est. Priority Date: 01/26/2001
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Mathur, Sharad, Corston-Oliver, Simon H.
• Granted Patent: US 7,398,203 B2
• Time in Patent Office: Days
• US Class Current: 704/1
• CPC Class Codes: G06F 40/211 Syntactic parsing, e.g. based on context-free grammar [CFG] or unification grammars
• CPC Class Codes: G06F 40/268 Morphological analysis
• CPC Class Codes: G06F 40/284 Lexical analysis, e.g. tokenisation or collocates
• CPC Class Codes: G06F 40/295 Named entity recognition
• CPC Class Codes: G06F 40/30 Semantic analysis

Abstract

A text processor processes text in a message. The text processor generates a plurality of compressed forms of components of the message. The processor performs a linguistic analysis on the body of text to obtain a linguistic output indicative of linguistic components of the body of text. The processor then generates the plurality of compressed forms that can be used to compress the body of text. The plurality of compressed forms are generated based on the linguistic output. The invention can be implemented as a method of generating the compressed forms and as an apparatus.

Claims

1. A method of processing a body of text to generate compression options, comprising:

performing a linguistic analysis on the body of text to obtain a linguistic output indicative of linguistic components of the body of text;

after performing the linguistic analysis, automatically generating a plurality of correct compression options for each of a plurality of different portions of the body of text to compress the body of text based on the linguistic output, each of the correct compression options comprising a different, correct compressed form of an instance of the portion in the body of text; and

selecting one of the plurality of correct compression options for each of the plurality of different portions of the body of text to output a compressed form of the body of text.

2. The method of claim 1 wherein automatically generating a plurality of correct compression options comprises:

automatically subjecting the portions of the body of text to different sets of compression rules to obtain the plurality of correct compression options.

3. The method of claim 2 wherein automatically subjecting the portions of the body of text to different sets of compression rules, comprises:

subjecting each portion of the body of text to the different sets of compression rules in a predetermined order such that the correct compression options reflect varying degrees of compression of a same portion of the body of text.

4. The method of claim 3 wherein automatically generating a plurality of correct compression options comprises:

generating a compression identifier attribute indicative of at least one of the sets of compression rules to which the portion of the body of text is subjected.

5. The method of claim 4 wherein automatically generating a plurality of correct compression options comprises:

generating a ShortForm attribute indicative of a compressed form of the portion of the body of text after application of the set of compression rules.

6. The method of claim 5 wherein automatically generating a plurality of correct compression options comprises:

generating a case normalized attribute, based on the ShortForm attribute, indicative of a CaseNormalizedForm of the ShortForm attribute.

7. The method of claim 6 wherein automatically generating a plurality of correct compression options comprises:

generating a compression attribute indicative of a further compressed form of the case normalized attribute.

8. The method of claim 7 wherein generating a compression attribute comprises:

applying letter removal rules to the case normalized attribute to remove letters based on a predetermined location of the letters in the CaseNormalizedForm.

9. The method of claim 8 wherein automatically generating a plurality of correct compression options comprises:

generating a LongForm attribute that reflects substantially no compression of the portion of the body of text.

10. The method of claim 9 wherein one ShortForm attribute comprises a word substitution based on a dictionary look-up and wherein automatically generating a plurality of correct compression options comprises:

setting the case normalized attribute and the compression attribute to the ShortForm attribute.

11. The method of claim 5 wherein performing a linguistic analysis comprises performing a syntactic analysis on the portion of the body of text and wherein generating the ShortForm attribute comprises:

applying the set of compression rules based on the syntactic analysis.

12. The method of claim 11 wherein the linguistic analysis further comprises, prior to performing the syntactic analysis:

performing a lexical analysis on the body of text; and

performing a morphological analysis on the body of text.

13. The method of claim 5 wherein generating the ShortForm attribute comprises:

normalizing dates to a numerical form.

14. The method of claim 5 wherein generating the ShortForm attribute comprises:

normalizing offset dates to a numerical form, based on a date that the body of text was authored.

15. The method of claim 5 wherein generating the ShortForm attribute comprises:

maintaining symbol-sensitive text fragments in uncompressed form.

16. The method of claim 15 wherein maintaining symbol-sensitive text fragments comprises:

maintaining text fragments that, cannot be accurately understood unless maintained fully in-tact, in uncompressed form.

17. The method of claim 16 wherein maintaining text fragments comprises:

maintaining uniform resource locators and electronic mail addresses in uncompressed form.

18. A computer readable data structure formed from a linguistic analysis of a body of text to be compressed indicative of a plurality of correct compressed forms of the body of text, the data structure comprising:

a plurality of different sections, each section corresponding to a textual term in the body of text, each section further comprising a plurality of selectable data fields, selectable to represent in subsequent operations one of a plurality of different, correct compressed forms of the corresponding textual term in the body of text, the different compressed forms representing different levels of compression of the corresponding textual term.

19. The computer readable data structure of claim 18 and further comprising:

a compression type attribute indicative of a type of compression applied to the textual term in the body of text in generating at least one of the plurality of correct compressed forms.

20. A message handler receiving a message and generating compression options indicative of different forms of a portion of a body of text in the message, the message handler comprising:

a linguistic analyzer linguistically configured to analyze the body of text and provide a linguistic analysis;

a compression form generator configured to automatically generate a plurality of different compressed forms of a plurality of individual textual segments in the body of text based on the linguistic analysis, the plurality of different compressed forms each representing a correct compressed form of a corresponding individual text segment; and

a compressor configured to generate an output indicative of selected ones of the plurality of different compressed forms for the individual textual segments in the body of text.