Method and system for extraction and organizing selected data from sources on a network

• US 7,418,440 B2
• Filed: 04/12/2001
• Issued: 08/26/2008
• Est. Priority Date: 04/13/2000
• Status: Expired due to Fees

• Current Assignee: QL2 Software LLC
• Original Assignee: QL2 Software LLC
• Inventors: Kubaitis, Edward Clifford
• Primary Examiner(s): Truong; Cam Y T
• Application Number:
  US09/833,846
• Publication Number:
  US 20010032205A1
• Time in Patent Office: 2,693 Days
• Field of Search: 707 1- 3, 707/10, 707/6, 707/5, 707/100, 707/102, 715/523
• US Class Current: 1/1
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/9532 Query formulation
• CPC Class Codes: G06F 16/9538 Presentation of query results
• CPC Class Codes: Y10S 707/99932 Access augmentation or optimizing
• CPC Class Codes: Y10S 707/99933 Query processing, i.e. searching
• CPC Class Codes: Y10S 707/99935 Query augmenting and refining, e.g. inexact access
• CPC Class Codes: Y10S 707/99936 Pattern matching access
• CPC Class Codes: Y10S 707/99939 Privileged access

Abstract

Described is a system and method for employing user created database-structured queries and data extraction engines to crawl through Websites extracting and organizing data from selected sources on a network, such as the Internet. The structure of a query processed by a Data Extraction engine enables a user to treat the network as a searchable database. The database-structured queries provide a user with tools to match patterns on selected sites on the network. A user may automate database-structured queries to be executed on a regular frequency. Output of the database-structured queries may be placed into a data log, displayed on a user display screen, or optionally reshaped for use by a plurality of data analysis tools. Additionally, an optional graphical user interface is provided.

Claims

1. A method for extracting data from a network by a server, comprising:

a) creating a database-structured query with at least one fundamental clause including a web domain address used for locating the data, based, in part, on a user input, wherein the database-structured query includes a regular expression used to determine the data to extract, and a conditional expression describing a number where to start searching and another number where to stop searching the data at the web domain address, wherein the number where to start searching, the another number where to stop searching and an amount to increment are defined by a user;

b) creating a template of the regular expression used to extract the data;

c) providing authentication data to the web domain address;

d) determining the web domain address on the network from which to extract the data;

e) extracting the data from the web domain address directly by retrieving a non-database structured arrangement of data from the determined web domain address and performing the database-structured query upon the retrieved non-database structured arrangement of data, wherein the extracting data from the web domain further comprises matching a plurality of patterns contained within the regular expression to retrieved data to determine the data to extract,

(f) repeating steps (d) and (e) in an iterative manner based on the at least one fundamental clause;

(g) reshaping the extracted data to a predetermined format; and

(h) providing the extracted data from the determined web domain address, wherein the extracted data is provided in a tab delimited data file, and wherein the tab delimited data file is provided directly to the user.

2. The method of claim 1, wherein creating the database-structured query, further comprises, including a network address within the database-structured query indicating a starting point.

3. The method of claim 2, wherein the determined web domain address, includes at least one universal resource locator (URL).

4. The method of claim 2, wherein directly extracting data from the determined web domain address, further comprises, following links contained within the web domain address until the links have been exhausted or following the links until a predetermined limit is reached.

5. The method of claim 1, wherein directly extracting the data from the web domain address, further comprises:

reducing the retrieved data to a region of interest; and

searching the region of interest for the data matching a predetermined regular expression.

6. The method of claim 5, wherein directly extracting the data from the web domain address, further comprises, storing data matching the regular expression.

7. The method of claim 1, further comprising, reshaping at least a portion of the extracted data for use by at least one data analysis software program.

8. The method of claim 1, wherein the web domain address further comprises at least one link address for locating at least a portion of the data.

9. The method of claim 1, wherein the at least one fundamental clause includes a request to parse an HTML table, and wherein extracting the data further comprises extracting data from the HTML table.

10. A computer-readable storage device having computer-executable instructions for extracting data from a network, the computer-executable instructions enabling actions comprising:

a) creating a database-structured query with at least one fundamental clause including a web domain address used for locating the data, based, in part, on a user input, wherein the database-structured query includes a regular expression used to determine the data to extract, and a conditional expression describing a number where to start searching and another number where to stop searching the data at the web domain address, wherein the number where to start searching, the another number where to stop searching and an amount to increment are defined by a user;

b) creating a template of the regular expression used to extract the data;

c) providing authentication data to the web domain address;

d) determining the web domain address on the network from which to extract the data;

e) extracting the data from the web domain address directly by retrieving a non-database structured arrangement of data from the determined web domain address and performing the database-structured query upon the retrieved non-database structured arrangement of data, wherein the extracting data from the web domain further comprises matching a plurality of patterns contained within the regular expression to retrieved data to determine the data to extract;

(f) repeating steps (d) and (e) in an iterative manner based on the at least one fundamental clause;

(g) reshaping the extracted data to a predetermined format; and

(h) providing the extracted data from the determined web domain address,

wherein the extracted data is provided in a tab delimited data file, and wherein the tab delimited data file is provided directly to the user.

11. The computer-readable storage device of claim 10, wherein the database-structured query, further comprises, a network address included within the database-structured query indicating a starting point to start to crawl the network.

12. The computer-readable storage device of claim 11, wherein the network address, further comprises at least one universal resource locator (URL) string generated by a sequence or list function.

13. The computer-readable storage device of claim 10, wherein the web domain address, further comprises at least one link to another web domain address for locating data to extract.

14. The computer-readable storage device of claim 10, wherein the web domain address further comprises at least one link address that is followed to locate data to extract until a user-specified number of links is reached.

15. A system for extracting data from a network comprising:

a client computer system having a hardware computing device and a client network connection to the network and communicating with a server computer system, the client computer system creating a database-structured query with at least one fundamental clause, based, in part, on a user input, wherein the database-structured query includes a regular expression used to determine the data to extract and a conditional expression describing a number where to start searching and another number where to stop searching the data at a web domain address, wherein the number where to start searching, the another number where to stop searching and an amount to increment are defined by a user;

an editor for creating a template of the regular expression used to extract the data;

the server computer system having a server network connection to the network and communicating with the client computer system, the server computer system further performs actions, comprising;

receiving the database-structured query from the client computer system;

determining the web domain address on the network from which to extract at least a portion of the data relevant to the query, wherein the determined web domain address is provided by the database-structured query;

providing authentication data to the web domain address;

extracting directly at least the portion of the data from the web domain address by retrieving a non-database structured arrangement of data from the determined web domain address and performing the database-structured query upon the retrieved non-database structured arrangement of data, wherein the extracting the portion of data from the web domain further comprises matching a plurality of patterns contained within the regular expression to retrieved data to determine the data to extract;

repeating, based on the at least one fundamental clause, actions of the determining a web domain address and the extracting the portion of data from the web domain address in an iterative manner;

reshaping the extracted data to a predetermined format; and

providing the extracted data from the web domain address, wherein the extracted data is provided in a tab delimited data file, and wherein the tab delimited data file is provided directly to the user.

16. The system of claim 15, wherein the regular expression within the database-structured query, further comprises, at least one symbol used to denote a part of the regular expression that a user desires to extract.

17. The system of claim 15, further comprising at least one data extraction engine to extract the data.

18. The system of claim 17, wherein the data extraction engine is a web crawler.

19. The system of claim 15, wherein the web domain address further comprises a link address, wherein at least another portion of the data is located with the link address.

20. The system of claim 15, wherein the server computer system is further configured to perform the actions including:

storing the database-structured query; and

providing a stored database-structured query to the client computer system upon user input request.

21. A method of extracting data from a network by a server, comprising:

(a) creating a database-structured query with at least one fundamental clause including a web domain address at the server based, in part, on a user input, wherein the database-structured query includes a regular expression used to determine the data to extract, creating a template of the regular expression used to extract the data, providing authentication data to the web domain address;

(b) determining a website to search based in part on the database-structured query and wherein the database-structured query further includes a conditional expression describing a number where to start searching and another number where to stop searching the data at the website, wherein the number where to start searching, the another number where to stop searching and an amount to increment are defined by a user;

(c) extracting the data at the website directly by retrieving a non-database structured arrangement of data from the web domain address and performing the database-structured query upon the retrieved non-database structured arrangement of data, wherein the extracting data from the web domain address further comprises matching a plurality of patterns contained within the regular expression to retrieved data to determine the data to extract, wherein the website is processed as a searchable database;

(d) repeating steps (b) and (c) in an iterative manner based on the at least one fundamental clause; and

(e) reshaping the extracted data to a predetermined format;

(f) providing the extracted data from the website, wherein the extracted data is provided in a data log, and wherein the data log is provided directly to the user.

22. The method of claim 21, wherein determining the website to search further comprises parsing the database-structured query to determine at least one link listed in the query, wherein the link is used to search the website indicated by the link in the query.

23. The method of claim 21, further comprising:

(a) determining at least one other website to search based in part on the database-structured query and a provided web domain address; and

(b) extracting at least another portion of the data at the at least one other website based on the database-structured query and the provided web domain address, wherein the at least one other website includes a non-database structured arrangement of data that is processed as a searchable database.

24. The method of claim 21, wherein determining the website to search further comprises determining what data to extract based in part on the database-structured query and a provided web domain address.

25. The method of claim 21, wherein extracting at least a portion of the data further comprises extracting data based in part on at least one of an Hypertext Markup Language (HTML) table, a binary file, and a matching pattern.

26. The method of claim 21, further comprising, reshaping the extracted data for at least one of a database, a spreadsheet Extensible Markup Language (XML) display, and a statistical tool.

27. The method of claim 21, wherein the website is a starting website based in part on the database-structured query.

28. A method of extracting data within at least one webpage, comprising:

(a) generating a database-structured query with at least one fundamental clause including a web domain address based, in part, on a user'"'"'s input, wherein the database-structured query further includes a regular expression used to determine the data to extract and a conditional expression describing a number where to start searching and another number where to stop searching the data at the at least one webpage, wherein the number where to start searching, the another number where to stop searching and an amount to increment are defined by a user;

(b) creating a template of the regular expression used to extract the data and providing authentication data to the web domain address;

(c) determining the at least one webpage with the data, wherein the determination of the at least one webpage is provided by the database-structured query;

d) extracting the data at the at least one webpage directly by retrieving a non-database structured arrangement of data from the web domain address and performing the database-structured query upon the retrieved non-database structured arrangement of data, wherein the extracting data from the web domain further comprises matching a plurality of patterns contained within the regular expression to retrieved data to determine the data to extract, wherein the extracted data that satisfies a query condition includes at least one binary file;

(e) repeating steps (c) and (d) in an iterative manner based on the at least one fundamental clause;

(f) reshaping the extracted data to a predetermined format; and

(g) providing the extracted data from the at least one webpage, wherein the extracted data is provided in a data log, and wherein the data log is provided directly to the user.

29. The method of claim 28, wherein the search of data is performed on at least a second webpage.

30. The method of claim 28, wherein parsing the data of the at least one webpage further comprises following links included on the webpage and further parsing the data of webpages determined by the links included on the webpage.

31. The method of claim 28, wherein the structured query is generated to parse a limited portion of the data of the at least one webpage with limits predetermined by the user.

32. The method of claim 28, wherein the structured query is generated to search for at least one of a text string, a table, and a predefined list of words.