SYSTEMS AND METHODS OF DE-DUPLICATING SIMILAR NEWS FEED ITEMS

• US 20160103916A1
• Filed: 10/10/2014
• Published: 04/14/2016
• Est. Priority Date: 10/10/2014
• Status: Active Grant

• Current Assignee: Salesforce.com, Inc.
• Original Assignee: Salesforce.com, Inc.
• Inventors: Even-Zohar, Yair, Tsur, Elad
• Granted Patent: US 9,984,166 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation

Abstract

The technology disclosed relates to de-duplicating contextually similar news feed items. In particular, it relates to assembling a set of news feed items from a plurality of electronic sources and preprocessing the set to generate normalized news feed items that share common company-name mentions and token occurrences. The normalized news feed items are used to calculate one or more resemblance measures based on a sequence alignment score and/or a hyperlink score. The sequence alignment score determines contextual similarity between news feed item pairs, arranged as sequences, based on a number of matching elements in the news feed item sequences and a number of edit operations, such as insertion, deletion, and substitution, required to match the news feed item sequences. The hyperlink score determines contextual similarity between news feed item pairs by comparing the respective search results retrieved in response to supplying the news feed item pairs to a search engine.

Claims

1. A method of de-duplicating similar news feed items, the method including:

assembling a set of news feed items from a plurality of electronic sources;

preprocessing the set to qualify some of the news feed items to return based on common company-name mentions and common token occurrences;

pairwise determining a resemblance measure for the qualified news feed items based on sequence alignment between news feed item pairs;

constructing a graph of news feed item pairs with the resemblance measure above a threshold and representing the resemblance measure as edges between nodes representing the news feed item pairs, thereby forming connected node pairs; and

determining similar news feed items by clustering the connected node pairs into strongly connected components.

2. The method of claim 1, wherein the news feed items are published within a predetermined time window prior to a current time.

3. The method of claim 1, further including determining representative news feed items for the similar news feed items by identifying cluster heads of respective strongly connected components, wherein the cluster heads have highest degree of connectivity in the respective strongly connected components.

4. The method of claim 1, further including pairwise determining the resemblance measure for the news feed items based on results returned in response to supplying the news feed item pairs as search criteria.

5. The method of claim 4, wherein the results returned include at least one of:

unified resource locators (URLs) of web pages;

content of the web pages; and

metadata about the web pages.

6. The method of claim 1, wherein preprocessing the set further includes removing stop tokens from the news feed items.

7. A method of de-duplicating similar news feed items, the method including:

assembling a set of news feed items from a plurality of electronic sources;

preprocessing the set to qualify some of the news feed items to return based on common company-name mentions and common token occurrences;

pairwise determining a resemblance measure for the qualified news feed items based on results returned in response to supplying news feed item pairs as search criteria;

constructing a graph of news feed item pairs with the resemblance measure above a threshold and representing the resemblance measure as edges between nodes representing the news feed item pairs, thereby forming connected node pairs; and

determining similar news feed items by clustering the connected node pairs into strongly connected components.

8. The method of claim 7, wherein the news feed items are published within a predetermined time window prior to a current time.

9. The method of claim 7, further including determining representative news feed items for the similar news feed items by identifying cluster heads of respective strongly connected components, wherein the cluster heads have highest degree of connectivity in the respective strongly connected components.

10. The method of claim 7, wherein the results returned include at least one of:

unified resource locators (URLs) of web pages;

content of the web pages; and

metadata about the web pages.

11. The method of claim 7, wherein the results returned in response to supplying a first news feed item as a search criteria include a second news feed item, further including augmenting the resemblance measure for the first and second news feed time pairs.

12. The method of claim 7, further including pairwise determining the resemblance measure for the news feed items based on sequence alignment between the news feed item pairs.

13. The method of claim 7, wherein preprocessing the set further includes removing stop word tokens from the news feed items.

14. A system of de-duplicating similar news feed items, the system including:

a processor and a computer readable storage medium storing computer instructions configured to cause the processor to;

assemble a set of news feed items from a plurality of electronic sources;

preprocess the set to qualify some of the news feed items to return based on common company-name mentions and common token occurrences;

pairwise determine a resemblance measure for the qualified news feed items based on sequence alignment between news feed item pairs;

construct a graph of news feed item pairs with the resemblance measure above a threshold and representing the resemblance measure as edges between nodes representing the news feed item pairs, thereby forming connected node pairs; and

determine similar news feed items by clustering the connected node pairs into strongly connected components.

15. The system of claim 14, wherein the news feed items are published within a predetermined time window prior to a current time.

16. The system of claim 14, further configured to determine representative news feed items for the similar news feed items by identifying cluster heads of respective strongly connected components, wherein the cluster heads have highest degree of connectivity in the respective strongly connected components.

17. The system of claim 14, further configured to pairwise determine the resemblance measure for the news feed items based on results returned in response to supplying the news feed item pairs as search criteria.

18. The system of claim 17, wherein the results returned include at least one of:

unified resource locators of web pages;

content of the web pages; and

metadata about the web pages.

19. The system of claim 17, wherein the results returned in response to supplying a first news feed item as a search criteria include a second news feed item, further including augmenting the resemblance measure for the first and second news feed time pairs.

20. The system of claim 14, wherein preprocessing the set further includes removing stop word tokens from the news feed items.