Lingustically intelligent text compression

• US 20020138248A1
• Filed: 01/26/2001
• Published: 09/26/2002
• Est. Priority Date: 01/26/2001
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Mathur, Sharad, Corston-Oliver, Simon H.
• Granted Patent: US 7,069,207 B2
• Time in Patent Office: Days
• US Class Current: 704/1
• CPC Class Codes: G06F 40/211 Syntactic parsing, e.g. based on context-free grammar [CFG] or unification grammars
• CPC Class Codes: G06F 40/268 Morphological analysis
• CPC Class Codes: G06F 40/284 Lexical analysis, e.g. tokenisation or collocates
• CPC Class Codes: G06F 40/295 Named entity recognition
• CPC Class Codes: G06F 40/30 Semantic analysis

Abstract

A text processor processes text in a message. The text processor generates a plurality of compressed forms of components of the message. The processor performs a linguistic analysis on the body of text to obtain a linguistic output indicative of linguistic components of the body of text. The processor then generates the plurality of compressed forms that can be used to compress the body of text. The plurality of compressed forms are generated based on the linguistic output. The invention can be implemented as a method of generating the compressed forms and as an apparatus.

Claims

1. A method of processing a body of text to generate compression options, comprising:

performing a linguistic analysis on the body of text to obtain a linguistic output indicative of linguistic components of the body of text; and

generating a plurality of compression options to compress the body of text based on the linguistic output.

2. The method of claim 1 wherein generating a plurality of compression options comprises:

subjecting a portion of the body of text to different sets of compression rules to obtain the plurality of compression options.

3. The method of claim 2 wherein subjecting the body of text to different sets of compression rules, comprises:

subjecting the portion of the body of text to the different sets of compression rules in a predetermined order such that the compression options reflect varying degrees of compression of a same portion of the body of text.

4. The method of claim 4 wherein generating a plurality of compression options comprises:

generating a compression identifier attribute indicative of at least one of the sets of compression rules to which the portion of the body of text is subjected.

5. The method of claim 4 wherein generating a plurality of compression options comprises:

generating a ShortForm attribute indicative of a compressed form of the portion of the body of text after application of the set of compression rules.

6. The method of claim 5 wherein generating a plurality of compression options comprises:

generating a case normalized attribute, based on the ShortForm attribute, indicative of a CaseNormalizedForm of the ShortForm attribute.

7. The method of claim 6 wherein generating a plurality of compression options comprises:

generating a compression attribute indicative of a further compressed form of the case normalized attribute.

8. The method of claim 7 wherein generating a compression attribute comprises:

applying letter removal rules to the case normalized attribute to remove letters based on a predetermined location of the letters in the CaseNormalizedForm.

9. The method of claim 8 wherein generating a plurality of compression options comprises:

generating a LongForm attribute that reflects substantially no compression of the portion of the body of text.

10. The method of claim 9 wherein one ShortForm attribute comprises a word substitution based on a dictionary look-up and wherein generating a plurality of compression options comprises:

setting the case normalized attribute and the compression attribute to the ShortForm attribute.

11. The method of claim 5 wherein performing a linguistic analysis comprises performing a syntactic analysis on the portion of the body of text and wherein generating the ShortForm attribute comprises:

applying the set of compression rules based on the syntactic analysis.

12. The method of claim 11 wherein the linguistic analysis further comprises, prior to performing the syntactic analysis:

performing a lexical analysis on the body of text; and

performing a morphological analysis on the body of text.

13. The method of claim 5 wherein generating the ShortForm attribute comprises:

normalizing dates to a numerical form.

14. The method of claim 5 wherein generating the ShortForm attribute comprises:

normalizing offset dates to a numerical form, based on a date that the body of text was authored.

15. The method of claim 5 wherein generating the ShortForm attribute comprises:

maintaining symbol-sensitive text fragments in uncompressed form.

16. The method of claim 15 wherein maintaining symbol-sensitive text fragments comprises:

maintaining text fragments that, cannot be accurately understood unless maintained fully in-tact, in uncompressed form.

17. The method of claim 16 wherein maintaining text fragments comprises:

maintaining uniform resource locators and electronic mail addresses in uncompressed form.

18. The method of claim 11 wherein the syntactic analysis includes a tree having non-terminal nodes representing multi-word portions of the body of text and terminal nodes indicative of words in the body of text, and wherein both the non-terminal nodes and the terminal nodes are examined for application of compression rules.

20. The data structure of claim 19 and further comprising:

a compression type attribute indicative of a type of compression applied to the portion of the body of text in generating at least one of the plurality of compressed forms.

21. The data structure of claim 20 wherein the plurality of compressed forms comprises:

a ShortForm attribute indicative of a compressed form of the portion of the body of text after application of the type of compression identified by the compression type attribute.

22. The data structure of claim 21 wherein the plurality of compressed forms comprises:

a case normalized attribute, based on the ShortForm attribute, indicative of a CaseNormalizedForm of the ShortForm attribute.

23. The data structure of claim 22 wherein the plurality of compressed forms comprises:

a compression attribute indicative of a further compressed form of the case normalized attribute.

24. The data structure of claim 23 and further comprising:

a LongForm attribute indicative of substantially no compression of the portion of the body of text.

26. The message handler of claim 25 wherein the compression form generator is configured to apply a plurality of different sets of compression rules to the portion of the body of text obtain the plurality of compressed forms.

27. The message handler of claim 26 wherein the compression form generator is further configured to apply the different sets of compression rules in a predetermined order such that the plurality of compressed forms reflect varying degrees of compression of a same portion of the body of text.

28. The message handler of claim 27 wherein the compression form generator is further configured to generate a compression identifier attribute indicative of at least one of the sets of compression rules applied to the portion of the body of text.

29. The message handler of claim 27 wherein the compression form generator is configured to provide, at its output, a data structure containing a plurality of attributes indicative of the plurality of compressed forms, and the compression identifier attribute.

30. The message handler of claim 29 wherein the plurality of attributes includes:

a ShortForm attribute indicative of a compressed form of the portion of the body of text after application of the set of compression rules;

a case normalized attribute, based on the ShortForm attribute, indicative of a CaseNormalizedForm of the ShortForm attribute; and

a compression attribute indicative of a further compressed form of the case normalized attribute.

31. The message handler of claim 30 wherein the plurality of attributes further comprises:

a LongForm attribute that reflects substantially no compression of the portion of the body of text.

19. A data structure formed from an analysis of a portion of a body of text indicative of a plurality of compressed forms of the portion of the body of text, the data structure comprising:

a plurality of data fields, representing a plurality of compressed forms of the portion of the body of text.

25. A message handler receiving a message and generating compression options indicative of different forms a portion of a body of text in the message, the message handler comprising:

a linguistic analyzer linguistically configured to analyze the body of text and provide a linguistic analysis; and

a compression form generator configured to generate a plurality of compressed forms of a portion of the body of text based on the linguistic analysis.