Linguistically intelligent text compression

• US 7,398,203 B2
• Filed: 04/04/2006
• Issued: 07/08/2008
• Est. Priority Date: 01/26/2001
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Mathur, Sharad, Corston-Oliver, Simon H.
• Primary Examiner(s): Edouard; Patrick N.
• Assistant Examiner(s): Wozniak; James S
• Application Number:
  US11/397,503
• Publication Number:
  US 20060184351A1
• Time in Patent Office: 826 Days
• Field of Search: 704/1, 704 9- 10, 715/534, 715/254
• US Class Current: 704/9
• CPC Class Codes: G06F 40/211 Syntactic parsing, e.g. based on context-free grammar [CFG] or unification grammars
• CPC Class Codes: G06F 40/268 Morphological analysis
• CPC Class Codes: G06F 40/284 Lexical analysis, e.g. tokenisation or collocates
• CPC Class Codes: G06F 40/295 Named entity recognition
• CPC Class Codes: G06F 40/30 Semantic analysis

Abstract

A text processor processes text in a message. The text processor generates a plurality of compressed forms of components of the message. The processor performs a linguistic analysis on the body of text to obtain a linguistic output indicative of linguistic components of the body of text. The processor then generates the plurality of compressed forms that can be used to compress the body of text. The plurality of compressed forms are generated based on the linguistic output. The invention can be implemented as a method of generating the compressed forms and as an apparatus.

Claims

1. A method of processing a body of text to generate compression options, comprising:

performing a linguistic analysis on the body of text to obtain a plurality of different tokens, comprising one of a word and a number, in the body of text;

after performing the linguistic analysis, automatically generating a plurality of correct compression options for each of a plurality of different corresponding tokens in the body of text to compress the body of text, each of the correct compression options comprising a different, correct compressed form of the corresponding tokens in the body of text wherein generating at least one of the compressed forms comprises reducing a number of letters or numbers in the corresponding token so the compressed form includes some, but not all, letters or numbers in the corresponding tokens;

wherein automatically generating a plurality of correct compression options comprises automatically subjecting each of the tokens in the body of text to different sets of compression rules in a predetermined order to obtain the plurality of correct compression options, such that the plurality of correct compression options reflect varying degrees of compression of a same token in the body of text; and

selecting one of the plurality of correct compression options for each of the plurality of different tokens in the body of text and outputting a compressed form of the body of text as the selected compression option for each token.

2. The method of claim 1 wherein automatically generating a plurality of correct compression options comprises:

generating a compression identifier attribute indicative of at least one of the sets of compression rules to which the tokens in the body of text is subjected.

3. The method of claim 2 wherein automatically generating a plurality of correct compression options comprises:

generating a ShortForm attribute indicative of a compressed form of the tokens in the body of text after application of the set of compression rules.

4. The method of claim 3 wherein automatically generating a plurality of correct compression options comprises:

generating a case normalized attribute, based on the ShortForm attribute, indicative of a CaseNormalizedForm of the ShortForm attribute.

5. The method of claim 4 wherein automatically generating a plurality of correct compression options comprises:

generating a compression attribute indicative of a further compressed form of the case normalized attribute.

6. The method of claim 5 wherein generating a compression attribute comprises:

applying letter removal rules to the case normalized attribute to remove letters based on a predetermined location of the letters in the CaseNormalizedForm.

7. The method of claim 6 wherein automatically generating a plurality of correct compression options comprises:

generating a LongForm attribute that reflects substantially no compression of the tokens in the body of text.

8. The method of claim 7 wherein one ShortForm attribute comprises a word substitution based on a dictionary look-up and wherein automatically generating a plurality of correct compression options comprises:

setting the case normalized attribute and the compression attribute to the ShortForm attribute.

9. The method of claim 3 wherein performing a linguistic analysis comprises performing a syntactic analysis on the body of text and wherein generating the ShortForm attribute comprises:

applying the set of compression rules based on the syntactic analysis.

10. The method of claim 9 wherein the linguistic analysis further comprises, prior to performing the syntactic analysis:

performing a lexical analysis on the body of text; and

performing a morphological analysis on the body of text.

11. The method of claim 3 wherein generating the ShortForm attribute comprises:

normalizing dates to a numerical form.

12. The method of claim 3 wherein generating the ShortForm attribute comprises:

normalizing offset dates to a numerical form, based on a date that the body of text was authored.

13. The method of claim 3 wherein generating the ShortForm attribute comprises:

maintaining symbol-sensitive text fragments in uncompressed form.

14. The method of claim 13 wherein maintaining symbol-sensitive text fragments comprises:

maintaining text fragments that, cannot be accurately understood unless maintained fully in-tact, in uncompressed form.

15. The method of claim 14 wherein maintaining text fragments comprises:

maintaining uniform resource locators and electronic mail addresses in uncompressed form.

16. A message handler receiving a message and generating compression options indicative of different forms of a portion of a body of text in the message, the message handler comprising:

a linguistic analyzer linguistically configured to analyze the body of text and provide a linguistic analysis having leaf nodes representing individual tokens in the body of text;

a compression form generator configured to automatically generate a plurality of different compressed forms of at least a plurality of tokens represented by the leaf nodes in the linguistic analysis, the plurality of different compressed forms each representing a correct compressed form of a corresponding individual token,

wherein generating at least one of the compressed forms comprises reducing a number of letters or numbers in the corresponding token so the compressed form includes some, but not all, letters or numbers in the corresponding tokens,

wherein the compression form generator generates the plurality of different compressed forms by automatically subjecting each of the tokens in the body of text to different sets of compression rules in a predetermined order to obtain the plurality of correct compressed forms, such that the plurality of correct compressed forms reflect varying degrees of compression of a same token in the body of text; and

a compressor configured to generate an output indicative of selected ones of the plurality of different compressed forms for the individual tokens in the body of text.