Harvesting Data From Page

• US 20080189267A1
• Filed: 08/07/2007
• Published: 08/07/2008
• Est. Priority Date: 08/09/2006
• Status: Active Grant

• Current Assignee: Fiver LLC
• Original Assignee: Radar Networks Incorporated (EVRI Incorporated)
• Inventors: Spivack, Nova, Jones, Christopher W., Tucker, Lewis W.
• Granted Patent: US 8,924,838 B2
• Time in Patent Office: Days
• US Class Current: 707/5
• CPC Class Codes: G06F 16/958 Organisation or management of web site content, e.g. publishing, maintaining pages or automatic linking
• CPC Class Codes: G06F 40/143 Markup, e.g. Standard Generalized Markup Language [SGML] or Document Type Definition [DTD]

Abstract

Among other disclosure, computer-implemented methods and computer program products for obtaining data from a page. A method can include initiating a harvesting process for a page available in a computer system. The method can include identifying a feed representation that has been created for the page. The method can include retrieving and storing, as part of the harvesting process, at least a portion from the page based on information in the identified feed representation. The feed representation can include at least excerpts of content from the page. The feed representation can include at least one representation selected from: an RSS feed, an Atom feed, an XML feed, an RDF feed, a serialized data feed representation, and combinations thereof.

Claims

1. A computer-implemented method for obtaining data from a page, the method comprising:

initiating a harvesting process for a page available in a computer system;

identifying a feed representation that has been created for the page; and

retrieving and storing, as part of the harvesting process, at least a portion from the page based on information in the identified feed representation.

2. The computer-implemented method of claim 1, further comprising:

identifying, before retrieving the portion of the page, at least a part of the feed representation to be used for retrieving the portion from the page, the part of the feed representation including the information.

3. The computer-implemented method of claim 2, further comprising:

using the identified part of the feed representation to identify the portion of the page as matching the information in the feed representation.

4. The computer-implemented method of claim 3, wherein using the identified part of the feed representation comprises:

comparing the identified part of the feed representation with contents of the page.

5. The computer-implemented method of claim 1, wherein the feed representation includes at least excerpts of content from the page.

6. The computer-implemented method of claim 1, wherein the feed representation comprises at least one representation selected from:

an RSS feed, an Atom feed, an XML feed, an RDF feed, a serialized data feed representation, and combinations thereof

7. The computer-implemented method of claim 1, further comprising:

identifying at least one feed entry in the feed representation, the feed entry relating to a portion of the page that links to another page;

determining a URL of the other page;

retrieving page content from the other page using the determined URL; and

identifying, in the retrieved page content, the portion of the page as matching content from the identified feed entry.

8. The computer-implemented method of claim 7, wherein retrieving the portion from the page based on the information includes using a text recognition technique.

9. The computer-implemented method of claim 1, further comprising:

determining, using the identified feed representation, which content from the page not to retrieve in the harvesting process; and

identifying the portion from the page as not being included in the determined content of the page not to retrieve.

10. The computer-implemented method of claim 9, further comprising causing the determined content of the page not to be retrieved in the harvesting process.

11. A computer program product tangibly embodied in a computer-readable medium and comprising instructions that when executed by a processor perform a method for obtaining data from a page, the method comprising:

initiating a harvesting process for a page available in a computer system;

identifying a feed representation that has been created for the page; and

retrieving and storing, as part of the harvesting process, at least a portion from the page based on information in the identified feed representation.

12. A computer-implemented method for obtaining data from a page, the method comprising:

identifying a page as a target for content retrieval, the page including multiple content portions;

identifying a feed representation that has been created for the identified page, the identified feed representation including multiple feed entries each corresponding to at least some of the multiple content portions;

processing each of the multiple feed entries by;

accessing the identified page;

identifying any of the multiple content portions that match contents of the feed entry being processed; and

retrieving at least one of the multiple content portions based on the identified content portion; and

storing, as a result of the content retrieval, each retrieved content portion obtained from the processing of the multiple feed entries.

13. The computer-implemented method of claim 12, wherein at least a first content portion of the multiple content portions links to another page, and wherein processing a first feed entry of the multiple feed entries relating to the first content portion includes:

accessing the other page to which the first content portion links;

identifying contents of the accessed other page that match contents of the first feed entry; and

retrieving the identified contents of the accessed other page.

14. The computer-implemented method of claim 12, wherein the feed representation includes at least excerpts of content from the identified page.

15. The computer-implemented method of claim 12, wherein the feed representation comprises at least one representation selected from:

an RSS feed, an Atom feed, an XML feed, an RDF feed, a serialized data feed representation, and combinations thereof.

16. The computer-implemented method of claim 12, wherein identifying contents of the accessed page that match contents of the feed entry being processed includes using a text recognition technique.

17. The computer-implemented method of claim 12, further comprising:

determining, using the identified feed representation, which content from the identified page not to retrieve in the content retrieval; and

identifying the at least one of the multiple content portion as not being included in the determined content of the page not to retrieve.

18. The computer-implemented method of claim 17, further comprising causing any of the multiple content portions identified as matching contents of the feed entry being processed not to be retrieved in the content retrieval.

19. A computer program product tangibly embodied in a computer-readable medium and comprising instructions that when executed by a processor perform a method for obtaining data from a page, the method comprising:

identifying a page as a target for content retrieval, the page including multiple content portions;

identifying a feed representation that that has been created for the identified page, the identified feed representation including multiple feed entries each corresponding to at least some of the multiple content portions;

processing each of the multiple feed entries by;

accessing the identified page;

identifying any of the multiple content portions that match contents of the feed entry being processed; and

retrieving at least one of the multiple content portions based on the identified content portion; and

storing, as a result of the content retrieval, each retrieved content portion obtained from the processing of the multiple feed entries.