Detection and handling of aggregated online content using decision criteria to compare similar or identical content items

• US 9,191,291 B2
• Filed: 09/09/2013
• Issued: 11/17/2015
• Est. Priority Date: 09/14/2012
• Status: Active Grant

• Current Assignee: Salesforce.com, Inc.
• Original Assignee: Salesforce.com, Inc.
• Inventors: Doan, Dai Duong
• Primary Examiner(s): Moorthy, Aravind
• Application Number:
  US14/021,977
• Publication Number:
  US 20140082755A1
• Time in Patent Office: 799 Days
• Field of Search: 726/32, 713/165, 713/168, 713/176, 713/180, 380/44
• US Class Current: 1/1
• CPC Class Codes: G06F 16/24568 Data stream processing; Continuous queries
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 21/10 Protecting distributed programs or content, e.g. vending or licensing of copyrighted material protection in video systems or pay television H04N7/16 ; Digital rights management [DRM]
• CPC Class Codes: G06F 21/60 Protecting data
• CPC Class Codes: G06F 21/64 Protecting data integrity, e.g. using checksums, certificates or signatures
• CPC Class Codes: G11B 20/0021 involving encryption or decryption of contents recorded on or reproduced from a record carrier
• CPC Class Codes: H04L 43/08 Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
• CPC Class Codes: H04L 67/02 based on web technology, e.g. hypertext transfer protocol [HTTP]
• CPC Class Codes: H04L 67/568 Storing data temporarily at an intermediate stage, e.g. caching
• CPC Class Codes: H04L 69/28 Timers or timing mechanisms used in protocols
• CPC Class Codes: H04L 9/08 Key distribution or management, e.g. generation, sharing or updating, of cryptographic keys or passwords network architectures or network communication protocols for supporting key management in a packet data network H04L63/06
• CPC Class Codes: H04L 9/3247 involving digital signatures

Abstract

A computer-implemented method is presented herein. The method obtains a first content item from an online source, and then generates a characterizing signature of the first content item. The method continues by finding a previously-saved instance of the characterizing signature and retrieving data associated with a second content item (the second content item is characterized by the characterizing signature). The method continues by analyzing the data associated with the second content item, corresponding data associated with the first content item, and decision criteria. Thereafter, either the first content item or the second content item is identified as an original content item, based on the analyzing. The other content item can be flagged as an aggregated content item.

Claims

1. A computer-implemented method comprising:

obtaining, at a computer system, a first content item from an online source, wherein the first content item is obtained via network connection;

generating a characterizing signature of the first content item, by;

selecting a quantity of text for analysis, the first content item comprising the quantity of text;

eliminating filler words from the quantity of text to identify a plurality of significant words;

arranging a predetermined number of the plurality of significant words from the quantity of text to create a document key;

applying a hash function to the document key to obtain a hashed document key; and

appending a language identifier to the hashed document key to create the characterizing signature;

finding a previously-saved instance of the characterizing signature in a cache memory architecture of the computer system;

retrieving, from the cache memory architecture, data associated with a second content item, in response to finding the previously-saved instance of the characterizing signature, wherein the second content item is characterized by the characterizing signature;

analyzing the data associated with the second content item, corresponding data associated with the first content item, and decision criteria; and

identifying either the first content item or the second content item as an original content item, based on the analyzing.

2. The computer-implemented method of claim 1, wherein:

the identifying identifies the second content item as the original content item; and

the method further comprises flagging the first content item as an aggregated content item.

3. The computer-implemented method of claim 1, wherein:

the identifying identifies the first content item as the original content item; and

the method further comprises flagging the second content item as an aggregated content item.

4. The computer-implemented method of claim 3, further comprising:

saving a source uniform resource locator (URL) of the first content item in association with the characterizing signature.

5. The computer-implemented method of claim 4, further comprising:

deleting a source URL of the second content item such that the characterizing signature is no longer saved in association with the second content item.

6. The computer-implemented method of claim 1, wherein the analyzing comprises:

comparing a first update frequency associated with a source of the first content item against predetermined update frequency criteria; and

comparing a second update frequency associated with a source of the second content item against the predetermined update frequency criteria.

7. The computer-implemented method of claim 1, wherein the analyzing comprises:

analyzing one or more first outbound links associated with a source of the first content item in view of predetermined outbound link criteria; and

analyzing one or more second outbound links associated with a source of the second content item in view of the predetermined outbound link criteria.

8. The computer-implemented method of claim 1, wherein the analyzing comprises:

checking a first publication date of the first content item relative to predetermined publication date criteria; and

checking a second publication date of the second content item relative to the predetermined publication date criteria.

9. The computer-implemented method of claim 1, wherein the analyzing comprises:

reviewing a first identified author of the first content item in view of predetermined authorship criteria; and

reviewing a second identified author of the second content item in view of the predetermined authorship criteria.

10. A computer-implemented method comprising:

generating, at a computer system, a first characterizing signature of a first content item, by;

selecting a quantity of text for analysis, the first content item comprising the quantity of text;

eliminating filler words from the quantity of text to identify a plurality of significant words;

arranging a predetermined number of the plurality of significant words from the quantity of text to create a document key;

applying a hash function to the document key to obtain a hashed document key; and

appending a language identifier to the hashed document key to create the first characterizing signature;

determining, at the computer system, that the first characterizing signature of a first content item matches a second characterizing signature of a second content item, wherein the second characterizing signature has been previously saved in a memory element of the computer system;

comparing an update frequency associated with a source of the first content item against predetermined update frequency criteria to determine whether the first content item is aggregated content;

when the first content item is not determined to be aggregated content based on the predetermined update frequency criteria, analyzing one or more outbound links associated with a source of the first content item in view of predetermined outbound link criteria, to determine whether the first content item is aggregated content;

when the first content item is not determined to be aggregated content based on the predetermined outbound link criteria, checking a publication date of the first content item relative to predetermined publication date criteria, to determine whether the first content item is aggregated content; and

when the first content item is not determined to be aggregated content based on the predetermined publication date criteria, reviewing an identified author of the first content item in view of predetermined authorship criteria, to determine whether the first content item is aggregated content.

11. The computer-implemented method of claim 10, wherein:

the first content item is determined to be aggregated content; and

the method further comprises flagging the first content item as an aggregated content item.

12. The computer-implemented method of claim 11, further comprising:

filtering the aggregated content from stored webcrawler data to obtain filtered webcrawler data; and

outputting the filtered webcrawler data for presentation to a user.

13. A computing system comprising a processor and a memory having computer-executable instructions stored thereon that, when executed by the processor, cause the computing system to:

obtain a first content item from an online source, wherein the first content item is obtained via network connection;

generate a characterizing signature of the first content item, by;

selecting a quantity of text for analysis, the first content item comprising the quantity of text;

eliminating filler words from the quantity of text to identify a plurality of significant words;

arranging a predetermined number of the plurality of significant words from the quantity of text to create a document key;

applying a hash function to the document key to obtain a hashed document key; and

appending a language identifier to the hashed document key to create the characterizing signature;

find a previously-saved instance of the characterizing signature in a cache memory architecture of the computing system;

retrieve, from the cache memory architecture, data associated with a second content item, in response to finding the previously-saved instance of the characterizing signature, wherein the second content item is characterized by the characterizing signature;

analyze the data associated with the second content item, corresponding data associated with the first content item, and decision criteria; and

identify either the first content item or the second content item as an original content item, based on the analyzing.

14. The computing system of claim 13, wherein the computer-executable instructions, when executed by the processor, cause the computing system to:

save a source uniform resource locator (URL) of the first content item in association with the characterizing signature.

15. The computing system of claim 13, wherein the data associated with the second content item, the corresponding data associated with the first content item, and the decision criteria is analyzed by:

comparing a first update frequency associated with a source of the first content item against predetermined update frequency criteria; and

comparing a second update frequency associated with a source of the second content item against the predetermined update frequency criteria.

16. The computing system of claim 13, wherein the data associated with the second content item, the corresponding data associated with the first content item, and the decision criteria is analyzed by:

analyzing one or more first outbound links associated with a source of the first content item in view of predetermined outbound link criteria; and

analyzing one or more second outbound links associated with a source of the second content item in view of the predetermined outbound link criteria.

17. The computing system of claim 13, wherein the data associated with the second content item, the corresponding data associated with the first content item, and the decision criteria is analyzed by:

checking a first publication date of the first content item relative to predetermined publication date criteria; and

checking a second publication date of the second content item relative to the predetermined publication date criteria.

18. The computing system of claim 13, wherein the data associated with the second content item, the corresponding data associated with the first content item, and the decision criteria is analyzed by:

reviewing a first identified author of the first content item in view of predetermined authorship criteria; and

reviewing a second identified author of the second content item in view of the predetermined authorship criteria.

19. A tangible and non-transitory computer readable medium having computer-executable instructions stored thereon that, when executed by a processor, perform a method comprising:

obtaining, at a computer system, a first content item from an online source, wherein the first content item is obtained via network connection;

generating a characterizing signature of the first content item, by;

selecting a quantity of text for analysis, the first content item comprising the quantity of text;

eliminating filler words from the quantity of text to identify a plurality of significant words;

arranging a predetermined number of the plurality of significant words from the quantity of text to create a document key;

applying a hash function to the document key to obtain a hashed document key; and

appending a language identifier to the hashed document key to create the characterizing signature;

finding a previously-saved instance of the characterizing signature in a cache memory architecture of the computer system;

retrieving data associated with a second content item from the cache memory architecture, in response to finding the previously-saved instance of the characterizing signature, wherein the second content item is characterized by the characterizing signature;

analyzing the data associated with the second content item, and corresponding data associated with the first content item; and

identifying either the first content item or the second content item as an original content item, based on the analyzing.

20. The computer readable medium of claim 19, wherein the analyzing comprises:

analyzing an update frequency associated with a source of the first content item, and an update frequency associated with a source of the second content item, to determine whether the first content item or the second content item is likely to be aggregated content;

analyzing one or more outbound links associated with a source of the first content item, and one or more outbound links associated with a source of the second content item, to determine whether the first content item or the second content item is likely to be aggregated content;

checking a publication date of the first content item, and a publication date of the second content item, to determine whether the first content item or the second content item is likely to be aggregated content; and

reviewing an identified author of the first content item, and an identified author of the second content item, to determine whether the first content item or the second content item is likely to be aggregated content.