System and methods for easy-to-use periodic network data capture engine with automatic target data location, extraction and storage

• US 6,810,414 B1
• Filed: 04/20/2000
• Issued: 10/26/2004
• Est. Priority Date: 02/04/2000
• Status: Expired due to Fees

• Current Assignee: Dennis A. Brittain
• Original Assignee: Dennis A. Brittain
• Inventors: Brittain, Dennis A.
• Primary Examiner(s): VU, VIET DUY
• Application Number:
  US09/553,669
• Time in Patent Office: 1,650 Days
• Field of Search: 709/217, 709/219, 709/223, 709/224, 709/328, 709/329, 719/328, 719/329, 707/10, 707/104
• US Class Current: 709/219
• CPC Class Codes: G06F 16/86 Mapping to a database

Abstract

System and methods for automatically capturing and storing data from a network is disclosed. The system and methods include easy-to-use graphical user interfaces; integration with Web browsers; point-and-click selection of data targets; automatic input element parameter substitution to retrieve multiple pages from a single network address; periodic Web page retrieval from network servers at pre-specified intervals; dynamic tag string matching to locate text data on variable format HTML and XML pages; dynamic tree matching to locate data nodes in variable document object model representations of HTML and XML data; intelligent character recognition of graphical HTML or XML elements; graphical database, database table and table record creation; and automatic creation of formatted data files or direct storage to database.

Claims

1. A method for specifying, capturing, locating, extracting and storing data from a network, comprising:

specifying at least one target datum accessible from a network location addressable by a network address;

generating one or more sets of input parameters, wherein said “

input parameters”

normally refers to hypertext markup language (HTML) or extensible markup language (XML) <

INPUT>

elements and <

FORM method=“

get”

>

or <

FORM method=“

post”

>

elements, or to variables within <

SCRIPT>

elements, allowing capture of one or more sets of network data from said network location;

capturing one or more sets of said network data from said network location at specified dates and times;

locating at least one said target datum in variable format sets of said network data, wherein said “

variable format”

normally refers to possible insertions or deletions of HTML or XML elements as well as variable text strings;

extracting at least one said target datum from said variable format sets of said network data; and

storing at least one said target datum in a file, a database, or any storage medium. Wherein said capturing from said network location further comprises substitution of said input parameters to generate one or more sets of said input parameters and repeatedly capturing from said network location for each set of said input parameters. Wherein performing said substitution of said input Darameters further comprises said substitution of one or more nested said input parameters, wherein said “

nested”

has a similar meaning as in “

nested FOR loops”

such that variables within inner FOR loops run through their entire range of values for every value of a variable in an outer FOR loop, wherein said substitution of one or more said nested said input parameters includes generating one or more sets of said input parameters, where each of one or more sets of said input parameters may provide different target data from said network location.

2. The method of claim 1, wherein said network is an Internet and said network location corresponds to a web page.

3. The method of claim 1, wherein said specifying at least one target datum includes a graphical interface, wherein said graphical interface may include or be part of a web browser.

4. The method of claim 1, wherein said specifying at least one target datum further comprises:

specifying data capture times including, but not limited to, starting dates, starting times, ending dates, ending times, months of the year, weeks of the month, days of the month, days of the week; and

storing the specifications of said data capture times in a file, a database, or any storage medium.

5. The method of claim 1, wherein said specifying at least one target datum further comprises setting up to perform substitution of said input parameters, wherein said “

substitution”

normally refers to the creation of sets of values for <

INPUT>

elements or <

SCRIPT>

variables, wherein said setting up to perform said substitution of said input parameters includes;

parsing URL strings to locate and identify said input parameter names and types;

locating and extracting <

INPUT>

elements from HTML or XML data;

specifying said input parameter types and said input parameter values; and

storing the specifications of said input parameters in a file, a database, or any storage medium.

6. The method of claim 5, wherein said input parameter values are:

static text strings;

or automatically generated numeric sequences;

or retrieved from a file;

or retrieved from a database;

or retrieved from any storage medium.

7. The method of claim 1, wherein said specifying at least one target datum further comprises:

specifying target data types, wherein said target data types include, but are not limited to, text, hypertext, hyperlinks, date or time values, input parameter values, invariant reference data, image or graphical data; and

storing specifications of said target data types in a file, a database, or any storage medium.

8. The method of claim 1, wherein said specifying at least one target datum further comprises setting up to perform dynamic document object model (DOM) tree matching, wherein said setting up to setting up to perform dynamic DOM tree matching includes:

capturing one or more sets of said network data from said network location;

extracting HTML or XML elements from said network data;

generating a reduced DOM template of said network data using extracted HTML or XML elements, wherein said “

reduced”

normally refers to the removal text strings or other data contained within said network data that may change each time said network data is captured from said network location; and

storing said reduced DOM template in a file, a database, or any storage medium.

9. The method of claim 1, wherein said specifying at least one target datum further comprises setting up to perform dynamic tag string matching, wherein said setting up to perform dynamic tag string matching includes:

capturing one or more sets of said network data from said network location;

extracting HTML or XML elements from said network data;

generating a reduced tag string template of said network data using extracted HTML or XML elements, wherein said “

reduced”

normally refers to the removal text strings or other data contained within said network data that may change each time said network data is captured from said network location; and

storing said reduced tag string template in a file, a database, or any storage medium.

10. The method of claim 1, wherein said at least one target datum is extracted from said variable format said network data using:

character sequence bounding;

or dynamic DOM tree matching, wherein said dynamic DOM tree matching includes creating a said reduced DOM tree representation from said network data, comparing said reduced DOM tree representation with previously stored said reduced DOM tree template to identify most probable location within said network data of said target datum, and extracting said target datum from said most probable location within said network data;

or dynamic tag string matching, wherein said dynamic tag string matching includes creating a said reduced tag string representation from said network data, comparing said reduced tag string representation with previously stored said reduced tag string template to identify most probable location within said network data of said target datum, and extracting said target datum from said most probable location within said network data.

11. The method of claim 10, wherein said dynamic tag string matching or said dynamic DOM tree matching further comprises using invariant reference data to increase robustness of location and extraction of said at least one target datum, wherein said “

invariant reference data”

normally refers to text strings or other data contained within said network data that does not change each time said network data is captured from said network location.

12. The method of claim 1, wherein said extracting said at least one target datum further comprises:

locating and extracting possible text blocks from unconstrained color images;

performing character recognition on possible text blocks;

filtering the recognized text; and

storing the filtered text in a file, a database, or any storage medium.

13. The method of claim 1, wherein said capturing one or more sets of network data further comprises using a watchdog timer to detect hung data capture attempts, and said hung data capture attempts are automatically aborted and reattempted.

14. The method of claim 1, wherein said specifying said at least one target datum includes graphically highlighting and isolating HTML or XML elements corresponding to said at least one target datum, wherein said “

graphically highlighting and isolating”

normally refers to selection of any of the HTML or XML elements contained within said page data, not just to hyperlinks or text strings which may be selected with ordinary web browsers, and said graphically highlighting and isolating HTML or XML elements includes using a mouse, joystick, trackball, or any pointing device.

15. The method of claim 1, wherein said specifying said at least one target datum includes specifying input parameters values in a menu-driven graphical interface.