Data extraction from world wide web pages

• US 5,913,214 A
• Filed: 08/08/1996
• Issued: 06/15/1999
• Est. Priority Date: 05/30/1996
• Status: Expired due to Term

• Current Assignee: Massachusetts Institute of Technology, Xerox Corporation (Xerox Holdings Corp.)
• Original Assignee: Massachusetts Institute of Technology
• Inventors: Madnick, Stuart E., Siegel, Michael D.
• Primary Examiner(s): Lintz, Paul R.
• Application Number:
  US08/698,166
• Time in Patent Office: 1,041 Days
• Field of Search: 707/2, 707/4, 707/12, 707/201, 707/10
• US Class Current: 1/1
• CPC Class Codes: G06F 16/2452 Query translation
• CPC Class Codes: G06F 16/2471 Distributed queries
• CPC Class Codes: G06F 16/25 Integrating or interfacing systems involving database management systems
• CPC Class Codes: Y10S 707/966 Distributed
• CPC Class Codes: Y10S 707/99932 Access augmentation or optimizing
• CPC Class Codes: Y10S 707/99933 Query processing, i.e. searching
• CPC Class Codes: Y10S 707/99934 Query formulation, input preparation, or translation
• CPC Class Codes: Y10S 707/99952 Coherency, e.g. same view to multiple users

Abstract

A system for querying disparate, heterogeneous data sources over a network, where at least some of the data sources are World Wide Web pages or other semi-structured data sources, includes a query converter, a command transmitter, and a data retriever. The query converter produces, from at least a portion of a query, a set of commands which can be used to interact with a semi-structured data source. The query converter may accept a request in the same form as normally used to access a relational data base, therefore increasing the number of data bases available to a user in a transparent manner. The command transmitter issues the produced commands to the semi-structured data source. The data retriever then retrieves the desired data from the data source. In this manner, structured queries may be used to access both traditional, relational data bases as well as non-traditional, semi-structured data bases such as web sites and flat files. The system may also include a request translator and a data translator for providing data context interchange. The request translator translates a request for data having a first data context into a query having a second data context which the query converter described above. The data translator translates data retrieved from the data context of the data source into the data context associated with the request. A related method for querying disparate data sources over a network is also described.

Claims

1. A system for querying heterogeneous data sources distributed over a network, said system comprising:

a request translator for translating a data request having an associated data context into a query having a second data context associated with at least one of the heterogeneous data sources;

a query converter for converting a portion of the query into at least one command which can be used to interact with a World Wide Web page by accessing a specification file associated with the data source, said specification file providing the commands necessary to access the World Wide Web page containing the requested data;

a command transmitter for issuing the at least one command over the network to a semi-structured data source;

a data retriever for extracting data from at least one of the heterogeneous data sources; and

a data translator which translates retrieved data from the data contexts associated with the data sources into the data context associated with the request.

2. The system of claim 1 wherein said request translator receives the request.

3. The system of claim 1 wherein said request translator generates the request.

4. The system of claim 1 wherein said request translator determines a heterogeneous data source to query based on the request.

5. The system of claim 4 wherein said request translator determines a heterogeneous data source to query based on an ontology.

6. The system of claim 4 wherein said request translator detects a difference between the context of data requested by the request and the context of data supplied by the data source and converts the data context of the request into the data context of the data source.

7. The system of claim 6 wherein said request translator optimizes the query based on the data context of the data source.

8. The system of claim 1 wherein said command transmitter optimizes the query by examining a specification file and determining if the commands listed by the specification file can be issued in order to access the World Wide Web page containing the requested data.

9. The system of claim 8 wherein said command transmitter separates the query into a plurality of sub-queries and queries at least one of the heterogeneous data sources using one of the sub-queries.

10. The system of claim 9 wherein the query transmitter queries a World Wide Web page with at least one of the sub-queries.

11. A method for querying heterogeneous data sources distributed over a network, said method comprising the steps of:

(a) translating a data request having an associated data context into a query having a second data context associated with at least one of the heterogeneous data sources to be queried;

(b) converting a portion of the query into at least one command which can be used to interact with a semi-structured data source;

(c) issuing the at least one command to at least one of the World Wide Web page by accessing a specification file associated with the data source, said specification file providing the commands necessary to access the World Wide Web page containing the requested data;

(d) retrieving data from at least one of the heterogeneous data sources; and

(e) translating retrieved data from the data contexts associated with the heterogeneous data sources into the data context associated with the request.

12. The method of claim 11 wherein step (b) further comprises converting a portion of the query into at least one command which can be used to interact with a flat file containing data.

13. The method of claim 11 further comprising the step of receiving a data request before step (a).

14. The method of claim 11 further comprising the step of generating a data request before step (a).

15. The method of claim 11 further comprising before step (a) the step of determining at least one heterogeneous data source to query based on the request.

16. The method of claim 15 further comprising before step (b) the step of determining at least one heterogeneous data source to query based on an ontology.

17. The method of claim 15 further comprising the steps of:

detecting a difference between the context of data requested by the request and the context of data supplied by the data source to be queried; and

converting the data context of the request into the data context of the data source.

18. The method of claim 11 further comprising before step (b) the step of optimizing the query.

19. The method of claim 11 further comprising the step of querying at least one of the data sources using the translated request.

20. The method of claim 19 wherein said optimization step further comprises:

separating the query into a plurality of sub-queries; and

querying at least one of the World Wide Web pages using at least one of sub-queries.

21. The method of claim 20 wherein said querying step further comprises querying a different data source with each one of the sub-queries.

22. A method for querying semi-structured data sources in response to a structured data request, the method comprising the steps of:

(a) converting a data request into one or more commands which can be used to interact with a World Wide Web page by accessing a specification file associated with the data source, said specification file providing the commands necessary to access the World Wide Web page containing the requested data;

(b) issuing at least one of the one or more commands to said semi-structured data source; and

(c) retrieving data from said semi-structured data source.

23. The method of claim 22 wherein step (a) further comprises converting a data request into one or more commands which can be used to interact with a flat file containing data.

24. The method of claim 22 wherein step (a) further comprises:

(a-a) determining if requested data is provided by one or more World Wide Web pages;

(a-b) determining, for each requested datum that is provided by a World Wide Web page, one or more commands which, when issued to the World Wide Web page, cause it to provide the requested datum.

25. The method of claim 24 wherein step (a-a) further comprises determining if requested data is provided by one or more World Wide Web pages by accessing a file stored in a memory element of a computer, said file including a list of all data the one or more World Wide Web page can provide .

26. The method of claim 24 wherein step (a-a) further comprises determining if requested data is provided by one or more World Wide Web pages by accessing a file stored in a memory element of a computer, said file containing a list of all data the World Wide Web page can provide and a data context associated with each datum provided by the World Wide Web page.

27. The method of claim 24 wherein step (a-b) further comprises determining, for each requested datum that is provided by a World Wide Web page, one or more commands which cause the World Wide Web page to provide the requested datum, the determination made by accessing a file located in a memory element of a computer which contains at least one instruction to be issued to the World Wide Web page.

28. A system for retrieving data from a semi-structured data source in response to a request, the system comprising:

a request converter for converting a request into one or more commands which can be used to interact with a World Wide Web page by accessing a specification file associated with the data source, said specification file providing the commands necessary to access the World Wide Web page containing the requested data;

a command transmitter for issuing at least one of the one or more commands to said semi-structured data source; and

a data retriever for extracting data from said semi-structured data source.

29. The system of claim 28 wherein said request converter accesses a file contained in a memory element of a computer in order to determine which data can be retrieved from a World Wide Web page and accesses a second file contained in a memory element of a computer which specifies commands to be used to access one of the World Wide Web pages.

30. The system of claim 29 wherein the query converter accesses only one file.