SYSTEM AND METHOD FOR DETECTING PERSONAL EXPERIENCE EVENT REPORTS FROM USER GERNERATED INTERNET CONTENT

• US 20120089616A1
• Filed: 10/05/2011
• Published: 04/12/2012
• Est. Priority Date: 10/06/2010
• Status: Active Grant

• Current Assignee: Treato Ltd.
• Original Assignee: First Life Ltd.
• Inventors: Sa'adon, Roee Robert
• Granted Patent: US 8,612,455 B2
• Time in Patent Office: Days
• US Class Current: 707/748
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation

Abstract

A method for retrieving Internet posts, implementable on a computing device, includes analyzing Internet posts to define segments in the Internet posts, where the segments at least contain terms consistent with user generation of a personal experience event report associated with a pre-defined search subject, and scoring each of the segments, where the score indicates a likelihood that the Internet post associated with the segment represents a user generated the personal experience report associated with the pre-defined search subject. A method for detecting personal experience event reports from user generated content on the Internet includes filtering a collection of Internet posts to include only Internet posts containing personal experience terms, and further filtering the filtered Internet posts by removing the Internet posts with non-personal experience terms.

Claims

1. A method, implementable on a computing device, for detecting personal experience event reports from user generated content on the Internet, the method comprising:

filtering a collection of Internet posts to include only said Internet posts containing personal experience terms; and

further filtering said filtered Internet posts by removing said Internet posts with non-personal experience terms.

2. The method according to claim 1 and also comprising:

analyzing said Internet posts to define segments in said Internet posts, wherein said segments at least contain terms consistent with user generation of a personal experience event report associated with a pre-defined search subject; and

scoring each of said segments, wherein said score indicates a likelihood that said Internet post associated with said segment represents a user generated said personal experience report associated with said pre-defined search subject.

3. The method according to claim 2 and also comprising:

storing at least indications of said Internet posts with associated said scores above a pre-defined threshold in a searchable personal experience database.

4. The method according to claim 2 and also comprising:

compiling a list of post collection websites; and

collecting said Internet posts according to said list of websites for said analyzing on a periodic basis.

5. The method according to claim 4 and wherein said compiling comprises at least one of:

detecting “

good”

textual patterns indicative of an authentic said user generated personal experience event report from a training set of authenticated said user generated personal experience event reports; and

detecting “

bad”

textual patterns indicative of a non-authentic said user generated personal experience event report from a training set of non-valid said user generated personal experience event reports.

6. The method according to claim 5 and also comprising:

assigning weights to each of said “

good” and

“

bad”

textual patterns to reflect a likelihood of said user generated personal experience event reports including each of said “

good” and

“

bad”

textual patterns.

7. The method according to claim 6 and also comprising:

assigning weights to predictive factors associated with said authentic and non-authentic user generated personal experience event reports in said training sets to reflect a likelihood of said user generated personal experience event reports being associated with at least some of said predictive factors, wherein said predictive factors include at least one of external website/page rankings and factors derived from said training sets.

8. The method according to claim 7 and wherein said external website/page rankings include at least one of Google website ranks, Google page ranks, query ranks, and Alexa ranks.

9. The method according to claim 7 and wherein said derived factors include at least one of website metadata, number of images per page, number of links per page, ratio of authentic user generated personal experience event reports per discussion thread, number of authentic user generated personal experience event reports per website, total anchor terms detected and total terms detected.

10. The method according to claim 7 and also comprising:

calculating a set of weighted indicators for assessing a likelihood that candidate websites have authentic said user generated personal experience event reports, wherein said set of weighted indicators include at least said weighted good and bad textual patterns and said weighted predictive factors.

11. The method according to claim 10 and also comprising:

identifying said candidate websites with Internet posts including terms from at least one of two “

anchor”

categories, wherein said anchor categories represent two essential components of said user generated personal experience reports;

collecting at least a sample of said Internet posts from said identified candidate websites; and

scoring each candidate website according to a cumulative weighted score as per said set of weighted indicators, wherein a pre-defined score threshold indicates a website with user generated personal experience event reports; and

adding said website with user generated personal experience event reports to said list of post collection websites.

12. The method according to claim 3 and wherein said pre-defined subject is pharmaceuticals.

13. The method according to claim 11 and wherein said anchor categories are a drug name and a symptom.

14. The method according to claim 3 and wherein said analyzing also comprises:

filtering said Internet posts to remove said Internet posts that do not at least contain said terms from each of a minimum number of term categories associated with said pre-defined subject;

detecting a pair of anchors from two anchor categories, wherein said anchor categories are also term categories and represent two essential components of said user generated personal experience reports; and

defining a basic said segment as a shortest section of text between said pair of anchors; and

when said shortest section of text does not include at least one said term from each of said minimum number of term categories, expanding said basic segment to extend beyond said shortest section of text to include at least one said term from each of said minimum number of term categories.

15. The method according to claim 14 and also comprising:

calculating a density value for said terms in said basic segment;

expanding said basic segment to include a nearest said term not included in said basic segment;

recalculating said density value for said expanded basic segment;

iteratively repeating said expanding and recalculating until said recalculated density value is less than a previously calculated said density value.

defining said expanded basic segment as a final segment.

16. The method according to claim 15 and wherein said calculating comprises:

defining weights to be associated with each of said terms, wherein each defined weight reflects a value for an associated said term as a predictor of said basic segment representing a user generated personal experience event report; and

multiplying a ratio of said weighted terms, divided by an overall number of words in said basic segment.

17. The method according to claim 3 and wherein said scoring comprises:

defining a set of indicating factors, wherein each said indicating factor is associated with a possible feature in said segments, wherein said possible features affects said likelihood that said Internet post associated with said segment represents a user generated said personal experience event report associated with said pre-defined search subject; and

weighting said indicating factors in accordance with said likelihood, wherein each of said indicating factors is at least one of negative and positive.

18. The method according to claim 17 and wherein said defining and weighting is at least in accordance with linear regression of a training set of said Internet posts known to represent said user generated said personal experience event reports.

19. The method according to claim 17 and wherein said defining and weighting is at least in accordance with linear regression of a training set of said Internet posts known not to represent said user generated said personal experience event reports.

20. The method according to claim 17 and also comprising:

multiplying a value for each of said indicating factors; and

summing the products of said multiplying to determine a final score, wherein said final scores exceeding a pre-defined threshold score indicate said user generated said personal experience event reports.

21. An Internet post retrieval system, implementable on a computing device, comprising:

a segment analyzer module to define segments in said Internet posts, wherein said segments at least contain terms consistent with user generation of a personal experience event report associated with a pre-defined search subject; and

a scoring engine to calculate a score for each said segment, wherein said score indicates a likelihood that said Internet post associated with said segment represents a user generated said personal experience report associated with said pre-defined search subject.

22. The system according to claim 21 and also comprising a post collector to collect posts from a collection list of Internet websites.

23. The system according to claim 21 and also comprising:

a searchable personal experience database to store at least indications of said Internet posts with associated said scores above a pre-defined threshold.

24. The system according to claim 21 and wherein said segment analyzer comprises:

a post filtering module to access a filter database of terms to filter out said Internet posts which do no not contain defined categories of said terms that are associated with said user generated personal experience event reports for said pre-defined search subject;

an anchor detection module to access an anchor database to detect at least a pair of anchors in said Internet posts that pass through said post filtering module, wherein said pair of anchors represents one term from a name category associated with said pre-defined search subject, and one term from a second category associated with said pre-defined search subject; and

a basic segmentation unit to define basic post segments, wherein said basic post segments contain at least said pair of anchors and expressions from additional categories of terms located in a term database.

25. The system according to claim 24 and wherein said segment analyzer also comprises:

a density calculator to calculate a cumulative density of said expressions in said basic post segments, wherein each of said expressions is weighted to indicate its contribution to said density; and

a segment optimizer to expand said basic post segments in accordance with a maximum said cumulative density as calculated by said density calculator.

26. The system according to claim 25 and wherein said scoring engine is configurable to calculate said score in accordance with a series of positively and negatively weighted factors associated with said search subject.

27. The system according to claim 21 and also comprising:

a website selection utility to generate a collection list of websites, wherein said websites are determined by said website selection utility to contain user generated personal experience event reports.

28. The system according to claim 26 and wherein said utility comprises:

a pattern recognizer to at least detect textual patterns associated with user generated personal experience event reports in a training set of “

good”

said Internet posts;

a training set scoring engine to calculate weighted indicators based on at least said detected textual patterns; and

a candidate scoring engine to at least apply said weighted indicators to Internet posts from candidate websites to determine, for each of said candidate websites, if they contain user generated personal experience event reports.

29. A method for compiling a list of Internet post collection websites, implementable on a computing device and comprising:

detecting “

good”

textual patterns indicative of an authentic user generated personal experience event report from a training set of authenticated said user generated personal experience event reports; and

detecting “

bad”

textual patterns indicative of a non-authentic said user generated personal experience event report from a training set of non-valid said user generated personal experience event reports.

30. The method according to claim 29 and also comprising:

assigning weights to each of said “

good” and

“

bad”

textual patterns to reflect a likelihood of said user generated personal experience event reports including each of said “

good” and

“

bad”

textual patterns.

31. The method according to claim 30 and also comprising:

assigning weights to predictive factors associated with said authentic and non-authentic user generated personal experience event reports in said training sets to reflect a likelihood of said user generated personal experience event reports being associated with at least some of said predictive factors, wherein said predictive factors include at least one of external website/page rankings and factors derived from said training sets.

32. The method according to claim 31 and wherein said external website/page rankings include at least one of Google website ranks, Google page ranks, query ranks, and Alexa ranks.

33. The method according to claim 31 and wherein said derived factors include at least one of website metadata, number of images per page, number of links per page, ratio of authentic user generated personal experience event reports per discussion thread, number of authentic user generated personal experience event reports per website, total anchor terms detected and total terms detected.

34. The method according to claim 31 and also comprising:

calculating a set of weighted indicators for assessing a likelihood that candidate websites have authentic said user generated personal experience event reports, wherein said set of weighted indicators include at least said weighted good and bad textual patterns and said weighted predictive factors.

35. The method according to claim 34 and also comprising:

identifying said candidate websites with Internet posts including terms from at least one of two “

anchor”

categories, wherein said anchor categories represent two essential components of said user generated personal experience reports;

collecting at least a sample of said Internet posts from said identified candidate websites; and

scoring each candidate website according to a cumulative weighted score as per said set of weighted indicators, wherein a pre-defined score threshold indicates a website with user generated personal experience event reports; and

adding said website with user generated personal experience event reports to said list of post collection websites.

36. A method for isolating segments from Internet posts, implementable on a computing device, comprising:

filtering said Internet posts to remove said Internet posts that do not at least contain terms from each of a minimum number of term categories that are associated with a user generated personal experience report associated with a pre-defined subject;

detecting a pair of anchors from two anchor categories, wherein said anchor categories are among said term categories and represent two essential components of said user generated personal experience event reports;

defining a basic said segment as a shortest section of text between said pair of anchors; and

when said shortest section of text does not include at least one said term from each of said minimum number of term categories, expanding said basic segment to extend beyond said shortest section of text to include at least one said term from each of said minimum number of term categories.

37. The method according to claim 36 and also comprising:

calculating a density value for said terms in said basic segment;

expanding said basic segment to include a nearest said term not included in said basic segment;

recalculating said density value for said expanded basic segment;

iteratively repeating said expanding and recalculating until said recalculated density value is less than a previously calculated said density value.

defining said expanded basic segment as a final segment.

38. The method according to claim 37 and wherein said calculating comprises:

defining weights to be associated with each of said terms, wherein each defined weight reflects a value for an associated said term as a predictor of said basic segment representing a user generated personal experience event report; and

multiplying a ratio of said weighted terms, divided by an overall number of words in said basic segment.

39. A method for scoring segments of Internet posts, implementable on a computing device, comprises:

defining a set of indicating factors, wherein each said indicating factor is associated with a possible feature in said segments, wherein said possible features affect a likelihood that said Internet post associated with said segment represents a user generated said personal experience event report associated with a pre-defined search subject; and

weighting said indicating factors in accordance with said likelihood, wherein each of said indicating factors is at least one of negative and positive.

40. The method according to claim 39 and wherein said defining and weighting is at least in accordance with linear regression of a training set of said Internet posts known to represent said user generated said personal experience event reports.

41. The method according to claim 39 and wherein said defining and weighting is at least in accordance with linear regression of a training set of said Internet posts known not to represent said user generated said personal experience event reports.

42. The method according to claim 39 and also comprising:

multiplying a value for each of said indicating factors; and

summing the products of said multiplying to determine a final score, wherein said final scores exceeding a pre-defined threshold score indicate said user generated said personal experience event reports.