System for providing database functions for multiple internet sources

• US 6,826,553 B1
• Filed: 11/16/2000
• Issued: 11/30/2004
• Est. Priority Date: 12/18/1998
• Status: Expired due to Term

• Current Assignee: Stripe, Inc.
• Original Assignee: Knowmadic, Inc.
• Inventors: DaCosta, Gerson Francis, Bhide, Rahul, Ghaskadvi, Vijay
• Primary Examiner(s): Alam, Shahid
• Assistant Examiner(s): Chen, Chongshan
• Application Number:
  US09/715,424
• Time in Patent Office: 1,475 Days
• Field of Search: 707/1, 707/10, 707/101, 707/204, 717/143, 715/500, 715/513
• US Class Current: 1/1
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: G06F 2216/01 Automatic library building
• CPC Class Codes: Y10S 707/99931 Database or file accessing
• CPC Class Codes: Y10S 707/99942 Manipulating data structure, e.g. compression, compaction, compilation
• CPC Class Codes: Y10S 707/99953 Recoverability

Abstract

A system for automatically extracting data from at least one electronic document accessible through the Internet or other computer network. The system records a sequence of actions operable to electronically navigate to a target page of the electronic document, the target page including a plurality of elements each having contents and a structural definition wherein the structural definitions interrelate the plurality of elements to specify a target pattern for a select subset of the plurality of elements. After recording the navigation path and the target pattern, the system automatically accesses the target page according to the recorded sequence. When the target page is accessed, the system automatically identifies, copies and processes selections from the plurality of elements dependent upon the target pattern.

Claims

1. A system for automatically extracting data from at least one electronic document in any of a plurality of formats, said at least one electronic document including a target page being accessible over a computer network, said target page comprising a plurality of elements each having a contents or structural definition, wherein said structural definition interrelates said plurality of elements, said system comprising:

a navigation module to record a sequence of actions associated with an initial visit by a user to said target page operable to navigate to said target page of said electronic document;

an extraction recording module to receive user inputs from said user defining information of interest to said user to be extracted from said plurality of elements of said target page and generating a target pattern for automatically extracting said information of interest to said user from said target page;

a navigation playback module to automatically access said target page according to said recorded sequence for at least one subsequent visit to said target page; and

an extraction playback module to automatically identify and scrape select ones of said plurality of elements dependent upon said target pattern for each said at least one subsequent visit to said target page;

said extraction recording module remapping said target page by re-identifying any modified structural definitions of said plurality of elements thereby to enable access to an altered target page;

said extraction playback module identifies and scrapes said select ones of said plurality of elements dependent upon said target pattern and said re-identified structural definitions to thereby automatically identify and scrape said select ones of said plurality of elements from said altered target page dependent upon said target pattern;

wherein information of interest to said user is automatically extracted from said target page for each said at least one subsequent visit to said target page.

2. The system of claim 1 wherein said electronic document is a web site and said target page is a web page or other web-accessible document.

3. The system of claim 1 wherein said contents and structural definitions are specified by a said user through user interaction using an application program interface by filling in forms and activating HTTP links.

4. The system of claim 1 wherein said format comprises a format consisting of one of the group of search engine results, web pages, other web-accessible documents, e-mail, text feeds in any format, HTML, .txt, .pdf, Word, Excel, .ppt, .ftp text feeds, databases and XML.

5. The system of claim 1, further comprising:

a script for altering said sequence of actions, thereby generating a second sequence of actions; and

wherein

said navigation playback module automatically executes said second sequence of actions, thereby accessing a second target page and automatically identifies and scrapes select ones of said plurality of elements from said second target page dependent upon said target pattern.

6. The system of claim 1 wherein:

said extraction recording module automatically accesses said target page according to said recorded sequence and remaps said target page by re-identifying said structural definitions of said plurality of elements;

said extraction playback module automatically identifies and scrapes select ones of said plurality of elements dependent upon said target pattern and said re-identified structural definitions; and

said select ones of said plurality of elements are compared to predicted results to determine whether said mapping is functioning properly.

7. The system of claim 1, further comprising:

means for generating at least one file including data indicative of said actions and said target pattern; and

means for storing said at least one file so as to be accessible over said computer network.

8. The system of claim 7 wherein said file is in XML format.

9. The system of claim 1 wherein said computer network includes a global interconnection of computer networks.

10. The system of claim 1 wherein said extraction recording module ignores some of said structural definitions in identifying said target pattern.

11. The system of claim 1 wherein said modules are plug-ins in a browser.

12. A method for automatically extracting data from a target page of at least one electronic document being accessible over a computer network, said target page comprising a plurality of elements each having a contents or structural definition wherein said structural definitions interrelate said plurality of elements, said method comprising:

recording a sequence of actions associated with an initial visit to said target page by a user operable to electronically navigate to said target page of said electronic document;

receiving user inputs defining a user selected subset of said plurality of elements to be extracted based on at least one of said contents and structural definitions;

generating a target pattern to identify said user selected subset of said plurality of elements to be extracted in subsequent visits to said target page;

automatically accessing said target page according to said recorded sequence for at least one subsequent visit to said target page;

automatically identifying and scraping a subset of select ones of said plurality of elements dependent upon said target pattern for each said at least one subsequent visit;

remapping said target page by re-identifying any modified structural definitions of said plurality of elements thereby to enable access to an altered target page; and

identifying and scraping said select ones of said plurality of elements dependent upon said target pattern and said re-identified structural definitions to thereby automatically identify and scrape said select ones of said plurality of elements from said altered target page dependent upon said target pattern;

wherein information of interest to said user is automatically extracted from said target page for each said at least one subsequent visit to said target page.

13. The method of claim 12 wherein:

in a first mode, said target pattern is dependent upon said interrelation of said structural definitions for said select subset;

in a second mode, said target pattern is dependent upon contents of said select subset;

in a third mode, said target pattern is dependent upon said structural definitions for and contents of said select subset; and

in a fourth mode, said pattern is dependent upon formatting of said select subset; and

in a fifth mode, said pattern is dependent upon said structural definitions for, contents of and formatting of said select subset.

14. The method of claim 12 wherein said actions include user interaction with a plurality of electronic documents.

15. The method of claim 14 wherein each of said plurality of electronic documents comprises a web page or other web-accessible electronic document.

16. The method of claim 15 wherein said actions include activating HTTP links and electronically filling in and submitting forms.

17. The method of claim 12 wherein said format comprises a format consisting of one of the group of search engine results, web pages, other web-accessible documents, e-mail, text feeds in any format, HTML, .txt, pdf, Word, Excel, ppt, ftp text feeds, databases and XML.

18. The method of claim 12, further comprising the step of:

applying XML tags to said scraped subset of select ones of said plurality of elements.

19. A computerized system for automatically scraping select data from a web site, data associated with said web site including a plurality of elements each having contents or structural data associated therewith, and being stored an a server being accessible through the Internet or other computer network, said contents and structural data and elements defining a select web page or other web-accessible document of said web site, said system comprising:

a navigation module being operable on a microprocessor-based device electronically coupled to the Internet or other computer network, said navigational module being operable to;

record a sequence of actions of a user operable to electronically navigate to said select web page or other web-accessible document of said web site using the Internet or other computer network; and

automatically access said select web page or other web-accessible document according to said recorded sequence for at least one subsequent visit to said select web page or other web-accessible document of said web site; and

an extraction module being operable on said microprocessor-based device, said extraction module being operable to;

receive user inputs identifying information of interest and generating a pattern for a select subset of said plurality of elements on said select web page or other web-accessible document for extracting said information of interest to said user;

automatically identify and scrape select ones of said plurality of elements of said select web page, or other web-accessible document dependent upon said pattern for each said at least one subsequent visit to said select web page or other web-accessible document;

remap said web page or other web-accessible document by re-identifying any modified structural definitions of said plurality of elements thereby to enable access to an altered web page or other web-accessible document; and

identify and scrape said select ones of said plurality of elements dependent upon said pattern and said re-identified structural definitions to thereby automatically identify and scrape said select ones of said plurality of elements from said altered web page or other web-accessible document dependent upon said pattern;

wherein information of interest to said user is automatically extracted in each said at least one subsequent visit to said target page.

20. The system of claim 19 wherein said microprocessor-based device includes a browser and said modules are operable through browser software.

21. The system of claim 20 wherein said modules and browser software are mutually responsive to one another.

22. The system of claim 19 wherein:

in a first mode, said pattern is dependent upon said interrelation of said structural definitions for said select subset;

in a second mode, said pattern is dependent upon contents of said select subset;

in a third mode, said pattern is dependent upon said structural definitions for and contents of said select subset;

in a fourth mode, said pattern is dependent upon formatting of said select subset; and

in a fifth mode, said pattern is dependent upon said structural definitions for, contents of and formatting of said select subset.

23. The system of claim 19 wherein:

said navigation module is adapted to automatically alter said sequence of actions according to predetermined criteria and automatically access other web pages or other web-accessible documents according to said altered sequence; and

said extraction module is adapted to automatically alter said pattern according to predetermined criteria and automatically identify and scrape other select ones of said plurality of elements of said other web pages or other web-accessible documents dependent upon said altered pattern.

24. A system for automatically extracting data from at least one electronic document accessible over a computer network, comprising:

a navigation module to record a navigation path to a target page selected by a user, said target page comprising a plurality of elements each having a contents or structural definition, wherein said structural definition interrelates said plurality of elements;

an extraction recording module receiving at least one user input for identifying information of interest on said target page to be extracted and generating a target pattern for extracting said information of interest;

a navigation playback module to automatically access said target page according to said navigation path for at least one subsequent visit to said target page;

an extraction playback module using said target pattern to extract said information of interest from said target page for each said at least one subsequent visit to said target page;

said extraction recording module remapping said target page by re-identifying any modified structural definitions of said plurality of elements thereby to enable access to an altered target page; and

said extraction playback module identifies and scrapes said select ones of said plurality of elements dependent upon said target pattern and said re-identified structural definitions to thereby automatically identify and scrape said select ones of said plurality of elements from said altered target page dependent upon said target pattern;

wherein information of interest to said user is automatically extracted from said target page in each said at least one subsequent visit to said target page.

25. The system of claim 24 wherein said electronic document is a web site and said target page is a web page or other web-accessible document.

26. The system of claim 24 wherein:

said target page comprises a plurality of elements each having a contents or structural definition, wherein said structural definition interrelates said plurality of elements;

said extraction recording module remaps said target page by re-identifying any modified structural definitions of said plurality of elements thereby to enable access to an altered target page; and

said extraction playback module identifies and scrapes said select ones of said plurality of elements dependent upon said target pattern and said re-identified structural definitions to thereby automatically identify and scrape said select ones of said plurality of elements from said altered target page dependent upon said target pattern.

27. The system of claim 24 wherein:

said target page comprises a plurality of elements each having a contents or structural definition, wherein said structural definition interrelates said plurality of elements;

said extraction recording module automatically-accesses said target page according to said navigation path and remaps said target page by re-identifying said structural definitions of said plurality of elements;

said extraction playback module automatically identifies and scrapes select ones of said plurality of elements dependent upon said target pattern and said re-identified structural definitions; and

said select ones of said plurality of elements are compared to predicted results to determine whether said mapping is functioning properly.

28. A method for automatically extracting data from an electronic document being accessible over a computer network, comprising:

in an initial visit by a user to a target page of said electronic document, recording a navigation path to said target page and receiving at least one user input defining information of interest to said user in said target page to be extracted, said target page comprising a plurality of elements each having a contents or structural definition, wherein said structural definition interrelates said plurality of elements;

generating a target pattern for extracting said information of interest from said target page;

automatically accessing said target page according to said navigation path to return to said target page for at least one subsequent visit;

for each subsequent visit to said target page, extracting information from said target page based on said target pattern, said extracting information including remapping said target page by re-identifying any modified structural definitions of plurality of elements thereby to enable access to an altered target page and identifying and scraping said select ones of said plurality of elements dependent upon said target pattern and said re-identified structural definitions to thereby automatically identify and scrape said select ones of said plurality of elements from said altered target page dependent upon said target pattern;

wherein information of interest to said user is automatically extracted from said target page for each said at least one subsequent visit.

29. The method of claim 28 further comprising:

providing a graphical user interface for said user to input said at least one user input defining information of interest.

30. The method of claim 28, further comprising:

providing a graphical user interface for said user to select said target page, monitor said user'"'"'s navigation to said target page, and for said user to define information of interest on said target page.

31. The method of claim 28, further comprising:

providing a graphical user interface for said user to input user preferences for a subset of a plurality of elements on said target page to be scraped.

32. The method of claim 28 wherein said extracting information comprises:

scraping a subset of elements from said target page.

33. The method of claim 28 wherein generating said target pattern includes:

analyzing at least one of a structural attribute, a content attribute, and a formatting attribute associated with each element of said target page selected by said user.

34. The method of claim 28, further comprising:

storing extracted information in a repository.

35. The method of claim 28, further comprising:

formatting extracted information into a standard data format, thereby generated formatted extracted data.

36. The method of claim 35, further comprising:

exporting said formatted extracted data into a software application specified by said user.

37. The method of claim 35, further comprising:

processing said formatted extracted data according to a process selected by said user.

38. The method of claim 28, wherein said automatically accessing comprises periodically visiting said target page.

39. A computer implemented method for automatically extracting data from an electronic document being accessible over a computer network, comprising:

providing a graphical user interface for a user to identify a web page as a target page, said target page comprising a plurality of elements each having a contents or structural definition, wherein said structural definition interrelates said plurality of elements;

recording a navigation path to said target page;

providing a user interface for said user to identify a subset of a plurality of elements of said target page as being information of interest to said user to be extracted;

generating a target pattern for extracting said information of interest from said target page;

automatically accessing said target page according to said navigation path for at least one return visit to said target page;

for each return visit to said target page, extracting information from said target page based on said target pattern, said extracting information including remapping said target page by re-identifying any modified structural definitions of said plurality of elements thereby to enable access to an altered target page and identifying and scraping said select ones of said plurality of elements dependent upon said target pattern and said re-identified structural definitions to thereby automatically identify and scrape said select ones of said plurality of elements from said altered target page dependent upon said target pattern; and

transforming said extracted information into a standard format for further processing;

wherein information of interest to said user is automatically extracted from said target page and transformed into a format suitable for use by at least one other software application for each said at least one return visit to said target page.

40. The method of claim 39, further comprising:

storing said extracted information in said standard format into a repository.

41. The method of claim 39, further comprising:

importing said extracted information in said standard format into a software application specified by said user.

42. The method of claim 40, wherein said automatically accessing comprises:

periodically returning to said target web page, whereby information of interest is periodically extracted.

43. The method of claim 41, wherein said automatically accessing comprises:

periodically returning to said target web page, whereby information of interest is periodically extracted.