Information processing analysis system for sorting and scoring text

• US 5,371,673 A
• Filed: 04/07/1993
• Issued: 12/06/1994
• Est. Priority Date: 04/06/1987
• Status: Expired due to Term

• Current Assignee: Infotrend, Inc.
• Original Assignee: David P. Fan
• Inventors: Fan, David P.
• Primary Examiner(s): Hayes, Gail O.
• Application Number:
  US08/045,317
• Time in Patent Office: 608 Days
• Field of Search: 364/419, 364/419.1, 364/419.01, 364/419.13, 364/419.19
• US Class Current: 704/1
• CPC Class Codes: G06Q 30/02 Marketing; Price estimation or determination; Fundraising

Abstract

A method and system for text analysis provides that text messages perceived by a population can be scored to determine the extent to which the messages favor one or more specified positions on a specified issue. A method and system for predicting public opinion based on message scores provides that the extent to which messages favor one or more specified positions can be used to determine the effect on the opinions of a specified population and to determine changes in the percentages of the percent of subpopulations within said specified population which favor said one or more specified positions.

Claims

1. A system for determining the presence within a text message of one or more predetermined ideas wherein the text of said message is in digital form and includes components in a human language, said system comprising:

a) means for searching said message for a plurality of predetermined scoring words and identifying words in said message which match with said scoring words;

b) means for determining the sequence within said message of matching scoring words and the distance between certain ones of said matching scoring words; and

c) means for identifying which said ideas are present in said message according to one or more predetermined rules, each of said rules specifying a relationship between one or more of said matching scoring words.

2. A system according to claim 1 further including means for assigning numerical scores for said message according to said one or more predetermined rules.

3. A system according to claim 1 further including communication means for electronically connecting to an electronic data base containing text stored in digital form and for obtaining said text.

4. A system according to claim 1, wherein said system further includes:

d) means for dividing the text of said message into specified blocks of text, wherein said blocks can include all or a subset of the text of said message;

e) means for making summary representations of each said block of text wherein each important concept conveyed by said message is reduced to a single concept symbol, and wherein each important concept is identified from;

1) one or more specified words within said block of text;

2) a quantity of text between said one or more specified words within said block of text;

3) the sequence within said message of said one or more specified words; and

f) means for altering specified blocks of text.

5. The system according to claim 4 further wherein said system further includes means for determining, based on said summary representations, scores for each said message representing the presence and absence of text favoring said ideas.

6. The system according to claim 4 wherein said means for making summary representations further includes:

b)(1) a dictionary containing a limited number of words likely to be contained in the text of said specified blocks of text;

b)(2) a set of concept symbols corresponding to the concepts to be used in evaluating the presence of said ideas within said specified blocks of text;

b)(3) means for assigning each word in said dictionary to one of said concept symbols;

b)(4) means for scanning the text of said specified block of text using pattern matching for the occurrence of matching words wherein matching words are those also found in said dictionary;

b)(5) a set of transformation rules for identifying said important concepts within said text wherein an important concept is identified from;

(i) one or more specified words within said specified block of text,

(ii) a quantity of text between said one or more specified words within said specified block of text, and

b)(6) means for transforming said summary representations to obtain a reduced summary representations wherein said important concepts conveyed by said text, relevant to said ideas, are each reduced to a single concept symbol.

7. The system according to claim 6 wherein each said reduced summary representation is in the form of a text equivalent listing comprising:

b)(6)(i) a symbol representing a beginning of said specified block of text wherein said symbol has the same form as that of said concept symbols;

b)(6)(ii) a distance symbol representing a quantity of text between the beginning of said specified block of text and its next nearest neighbor wherein said next nearest neighbor is a said matching word, or the end of said block of text;

b)(6)(iii) the concept symbol corresponding to each said matching word in said specified block of text;

b)(6)(iv) a distance symbol corresponding to said distance between each said matching word and its next nearest neighbor or said end of said specified block of text; and

b)(6)(v) sequence information specifying the sequence of said beginning, each said matching word, and said end of said specified block of text.

8. The system according to claim 6 wherein said means for transforming said summary representations to obtain said reduced summary representations further includes:

b)(6)(i) means for performing evaluations of pairs of said concept symbols in said text equivalent of said specified block of text wherein said evaluations are based on said symbols, said sequence information, and said distance symbols in said text equivalent listing; and

b)(6)(ii) means for performing transformations on said text equivalent listing to reduce said text equivalent listing to important concept symbols wherein said transformations are based on said evaluations of pairs of said concept symbols.

9. The system according to claim 8 wherein said means for performing transformations further includes, based on relationships defined in said transformation rules:

b)(6)(ii)(A) means for inserting a specified distance symbol and a specified concept symbol into said specified text equivalent listing; and

b)(6)(ii)(B) means for modifying said sequence information of said specified text equivalent listing.

10. The system according to claim 8 wherein said means for performing transformations further includes, based on relationships defined in said transformation rules:

b)(6)(ii)(A) means for deleting a specified concept symbol and a specified distance symbol from a specified text equivalent listing; and

b)(6)(ii)(B) means for modifying said sequence information of said text equivalent.

11. The system according to claim 4 wherein said means for making summary representations includes:

b)(1) means for altering said summary representations based on a said specified block of text and specified input rules, wherein said input rules specify one or more concept symbols to insert and the method for insertion there into; and

b)(2) means for altering said specified block of text, based on other blocks of text and specified input rules, wherein said input rules specify one or more words to insert and locations within said individual text blocks for their insertion.

12. The system according to claim 11 further including means for insertion of one or more specified words into the text of said message at locations specified by said input rules and based on said summary representation, wherein said one or more specified words are quantities of text defined in said input rules.

13. The system according to claim 11 further including means for deletions of above specified words from the text of said message at locations specified by said input rules and based on said summary representation, wherein said specified words are quantities of text defined in said input rules.

14. The system according to claim 11 further including means for replacement of one or more specified words in the text of said message by one or more specified replacement words at locations specified by input rules, based on said summary representation, wherein said one or more specified words and said one or more replacement words are quantities of text defined in said input rules.

15. A system for reducing a text message into its essential message components and identifying the presence within said message of one or more predetermined ideas wherein the text of said message is stored in digital form and includes components in a human language, said system comprising:

computer means including;

a) a listing of predetermined concept categories and a dictionary of predetermined identifying words wherein each of said identifying words is a text representation corresponding to one of said concept categories, and wherein said concept categories represent a predetermined concept;

b) a set of predetermined text analysis rules wherein said text analysis rules define relationships between one or more concept categories;

c) means for dividing the text of said message into specified blocks of text, wherein said blocks can include all or a subset of the text of said message;

d) means for searching each said block of text for a first plurality of words in said message which match with said predetermined identifying words, such instance of a plurality being a set of matching words;

e) means for determining the sequence of said matching words in said block of text and the distance between pairs of said matching words wherein distance is a numeric representation of the quantity of text between the said pair of matching words in said text;

f) means for analyzing said matching words, said sequence of matching words, and said distances between pairs of matching words to select blocks of text in said message according to one or more of said text analysis rules, wherein said text analysis rules define a relationship between one or more matching words that identifies said blocks of text as relevant to a said idea;

g) means for searching each said relevant block of text in said message for a plurality of words which match with said predetermined identifying words;

h) means for determining the sequence within said block of text of matching identifying words and the distance between each pair of matching identifying words; and

i) means for determining within each relevant block of text using said predetermined text analysis rules, the quantities of text favoring each of said ideas.

16. A system according to claim 15 wherein said computer means further includes means for assigning numerical scores for each relevant block of text according to said predetermined text analysis rules, each of said rules further specifying a relationship between said concept categories each corresponding to one said matching identifying word and identifying which said ideas the block of text favors.

17. A system according to claim 16 wherein said computer means further includes means for assigning a numerical score to said message according to the numerical scores for the relevant blocks of text found therein.

18. A system according to claim 15 further including communication means for electronically connecting said computer means to an electronic data base containing text stored in digital form and for transferring said text to said computer means.

19. The system according to claim 15 wherein said computer means further includes:

j) means for making summary representations of said specified blocks of text wherein each important concept conveyed by said block of text is reduced to a single concept symbol, and wherein each important concept is identified from;

(i) one or more specified words within said specified block of text;

(ii) a quantity of text between said one or more specified words within said block of text; and

(iii) the sequence within said specified block of text of said one or more specified words; and

k) means for altering specified blocks of text.

20. The system of claim 19 further including means for determining, based on said summary representations:

1) which blocks of text to use to determine said numerical scores for each said message; and

2) numerical scores for each said message.

21. The system according to claim 19 wherein said means for making said summary representations from said specified blocks of text further includes:

a)(1) means for altering said summary representations based on said specified blocks of text and specified input rules, wherein said input rules specify one or more concept symbols to insert and the method for insertion there into; and

a)(2) means for altering individual said specified blocks of text, based on other blocks of text and specified input rules, wherein said input rules specify one or more words to insert and locations within said individual text blocks for their insertion.

22. The system according to claim 21 further including means for insertion of one or more specified words into the text of said message at locations specified by said input rules and based on said summary representation, wherein said one or more specified words are quantities of text defined in said input rules.

23. The system according to claim 21 further including means for deletions of above specified words from the text of said message at locations specified by said input rules and based on said summary representation, wherein said specified words are quantities of text defined in said input rules.

24. The system according to claim 21 further including means for replacing one or more specified words in the text of said message by one or more specified replacement words at locations specified by input rules, based on said summary representation, wherein said one or more specified words and said one or more replacement words are quantities of text defined in said input rules.

25. The system according to claim 19 wherein said means for making summary representations of said specified blocks of text further includes:

j)(1) a dictionary containing a limited number of words likely to be contained in the text of said specified blocks of text;

j)(2) a set of concept symbols corresponding to the concepts to be used in evaluating presence of said ideas within said specified blocks of text;

j)(3) means for assigning each word in said dictionary to one of said concept symbols;

j)(4) means for scanning the text of said specified block of text using pattern matching for the occurrence of matching words wherein matching words are those also found in said dictionary;

j)(5) a set of transformation rules for identifying said important concepts within said text wherein an important concept is identified from;

(i) one or more specified words within said specified block of text,

(ii) a quantity of text between said one or more specified words within said specified block of text, and

j((6) means for transforming said summary representations to obtain a reduced summary representations wherein said important concepts conveyed by said text, relevant to said ideas, are each reduced to a single concept symbol.

26. The system according to claim 25 wherein each said reduced summary representation is in the form of a text equivalent listing comprising:

j)(6)(i) a symbol representing a beginning of said specified block of text wherein said symbol has the same form as that of said concept symbols;

j)(6)(ii) a distance symbol representing a quantity of text between the beginning of said specified block of text and its next nearest neighbor wherein said next nearest neighbor is a said matching word, or the end of said block of text;

j)(6)(iii) the concept symbol corresponding to each said matching word in said specified block of text;

j)(6)(iv) a distance symbol corresponding to said distance between each said matching word and its next nearest neighbor or said end of said specified block of text; and

j)(6)(v) sequence information specifying the sequence of said beginning, each said matching word, and said end of said specified block of text.

27. The system according to claim 25 wherein said means for transforming said summary representations to obtain said reduced summary representations further includes:

j)(6)(i) means for performing evaluations of pairs of said concept symbols in said text equivalent of said specified block of text wherein said evaluations are based on said symbols, said sequence information, and said distance symbols in said text equivalent listing; and

j)(6)(ii) means for performing transformations on said text equivalent listing to reduce said text equivalent listing to important concept symbols wherein said transformations are based on said evaluations of pairs of said concept symbols.

28. The system according to claim 27 wherein said means for transforming further includes, based on relationships defined in said transformation rules:

j)(6)(i) means for inserting a specified distance symbol and a specified concept symbol into said specified text equivalent; and

j)(6)(ii) means for modifying said sequence information of said specified text equivalent.

29. The system according to claim 27 wherein said means for transforming further includes, based on relationships defined in said transformation rules:

j)(6)(i) means for deleting a specified concept symbol and a specified distance symbol from a specified text equivalent; and

j)(6)(ii) means for modifying said sequence information of said text equivalent.

30. The system according to claim 19 wherein said computer means further includes:

(l) means for altering said summary representations based on said specified blocks of text and specified input rules, wherein said input rules specify one or more concept symbols to insert and the method for insertion there into; and

(m) means for altering individual said specified blocks of text, based on other blocks of text and specified input rules, wherein said input rules specify one or more words to insert and locations within said individual text blocks for their insertion.

31. The system according to claim 30 further including means for insertion of one or more specified words into the text of said message at locations specified by said input rules and based on said summary representation, wherein said one or more specified words are quantities of text defined in said input rules.

32. The system according to claim 30 further including means for deletions of above specified words from the text of said message at locations specified by said input rules and based on said summary representation, wherein said specified words are quantities of text defined in said input rules.

33. The system according to claim 30 further including means for the replacement of one or more specified words in the text of said message by one or more specified replacement words at locations specified by input rules, based on said summary representation, wherein said one or more specified words and said one or more replacement words are quantities of text defined in said input rules.

34. A system for determining the presence within a text message of one or more predetermined ideas wherein the text of said message is in digital form and includes components in a human language, said system comprising:

a) means for searching said message for a plurality of predetermined scoring words and identifying words in said message which match with said scoring words;

b) means for determining the sequence within said message of matching scoring words and the distance between certain ones of said matching scoring words; and

c) means for identifying which said ideas are present in said message according to one or more predetermined rules, each of said rules specifying a relationship between one or more of said matching scoring words;

said means further including;

d) means for dividing the text of said message into specified blocks of text, wherein said blocks can include all or a subset of the text of said message;

e) means for making summary representations of each said block of text wherein each important concept conveyed by said message is reduced to a single concept symbol, and wherein each important concept is identified from;

1) one or more specified words within said block of text;

2) a quantity of text between said one or more specified words within said block of text;

3) the sequence within said message of said one or more specified words; and

f) means for altering specified blocks of text.

35. A system for determining the presence within a text message of one or more predetermined ideas wherein the text of said message is in digital form and includes components in a human language, said system comprising:

a) means for searching said message for a plurality of predetermined scoring words and identifying words in said message which match with said scoring words;

b) means for determining the sequence within said message of matching scoring words and the distance between certain ones of said matching scoring words;

c) means for transforming said message to make a summary representation thereof;

d) means for identifying which said ideas are present in said message according to one or more predetermined rules, each of said rules specifying a relationship between one or more of said matching scoring words.

36. The system according to claim 35 wherein said transformation reduces said message to a concept symbol.

37. A system for reducing a text message into its essential message components and identifying the presence within said message of one or more predetermined ideas wherein the text of said message is stored in digital form and includes components in a human language, said system comprising:

computer means including;

a) a listing of predetermined concept categories and a dictionary of predetermined identifying words wherein each of said identifying words is a text representation corresponding to one of said concept categories, and wherein each said concept category represents a predetermined concept;

b) a set of predetermined text analysis rules wherein said text analysis rules define relationships between one or more concept categories;

c) means for dividing the text of said message into specified blocks of text, wherein said blocks can include all or a subset of the text of said message;

d) means for searching each said block of text for a first plurality of words in said message which match with said predetermined identifying words, such instance of a plurality being a set of matching words;

e) means for determining the sequence of said matching words in said block of text and the distance between pairs of said matching words wherein distance is a numeric representation of the quantity of text between the said pair of matching words in said text;

f) means for analyzing said matching words, said sequence of matching words, and said distances between pairs of matching words to select blocks of text in said message according to one or more of said text analysis rules, wherein said text analysis rules define a relationship between one or more matching words and identify relevant blocks of text as relevant to said ideas;

g) means for searching each said relevant block of text in said message for a plurality of words which match with said predetermined identifying words;

h) means for determining the sequence within said block of text of matching identifying words and the distance between each pair of matching identifying words;

i) means for transforming said block of text to make a summary representation thereof;

j) means for determining within said relevant block of text using said predetermined text analysis rules, the quantities of text favoring each of said ideas.

38. The system according to claim 37 wherein said transformation reduces each block of text to a concept symbol.