Query and retrieving semi-structured data from heterogeneous sources by translating structured queries

• US 6,282,537 B1
• Filed: 04/06/1999
• Issued: 08/28/2001
• Est. Priority Date: 05/30/1996
• Status: Expired due to Term

• Current Assignee: Michael D. Siegel, Stuart E. Madnick
• Original Assignee: Massachusetts Institute of Technology
• Inventors: Madnick, Stuart E., Siegel, Michael D.
• Primary Examiner(s): Lintz, Paul R.
• Application Number:
  US09/286,852
• Time in Patent Office: 875 Days
• Field of Search: 707/4, 707/2, 707/3
• US Class Current: 707/716
• CPC Class Codes: G06F 16/2452 Query translation
• CPC Class Codes: G06F 16/2471 Distributed queries
• CPC Class Codes: G06F 16/25 Integrating or interfacing systems involving database management systems
• CPC Class Codes: Y10S 707/966 Distributed
• CPC Class Codes: Y10S 707/99932 Access augmentation or optimizing
• CPC Class Codes: Y10S 707/99933 Query processing, i.e. searching
• CPC Class Codes: Y10S 707/99934 Query formulation, input preparation, or translation
• CPC Class Codes: Y10S 707/99952 Coherency, e.g. same view to multiple users

Abstract

A system for querying disparate, heterogeneous data sources over a network, where at least some of the data sources are World Wide Web pages or other semi-structured data sources, includes a query converter, a command transmitter, and a data retriever. The query converter produces, from at least a portion of a query, a set of commands which can be used to interact with a semi-structured data source. The query converter may accept a request in the same form as normally used to access a relational data base, therefore increasing the number of data bases available to a user in a transparent manner. The command transmitter issues the produced commands to the semi-structured data source. The data retriever then retrieves the desired data from the data source. In this manner, structured queries may be used to access both traditional, relational data bases as well as non-traditional semi-structured data bases such as web sites and flat files. The system may also include a request translator and a data translator for providing data context interchange. The request translator translates a request for data having a first data context into a query having a second data context which the query converter described above. The data translator translates data retrieved from the data context of the data source into the data context associated with the request. A related method for querying disparate data sources over a network is also described.

Claims

1. A system for querying heterogeneous data sources distributed over a network, said system comprising:

a plurality of heterogeneous data sources, each of which includes local information relating to a first semantic data context associated with the heterogeneous data source;

a request translator associated with each requester, said request translator receiving a request having a second semantic data context and comparing said first semantic data context and said second semantic data context to determine if a conflict between those contexts exists, said request translator translating the request, responsive to the conflict determination, into a query having at least one semantic data context associated with at least one of the heterogeneous data sources;

a query converter for converting a portion of the query into at least one command which can be used to interact with a semi-structured data source;

a command transmitter for issuing the at least one command over the network to a semi-structured data source;

a data retriever for extracting data from at least one of the heterogeneous data sources; and

a data translator which translates retrieved data from the data contexts associated with the data sources into the data context associated with the request.

2. The system of claim 1 wherein the semi-structured data source is a World Wide Web page.

3. The system of claim 1 wherein the semi-structured data source is a flat file containing data.

4. The system of claim 1 wherein said request translator receives the request.

5. The system of claim 1 wherein said request translator generates the request.

6. The system of claim 1 wherein said request translator determines a heterogeneous data source to query based on the request.

7. The system of claim 6 wherein said request translator determines a heterogeneous data source to query based on an ontology.

8. The system of claim 6 wherein said request translator detects a difference between the context of data requested by the request and the context of data supplied by the data source and converts the data context of the request into the data context of the data source.

9. The system of claim 8 wherein said request translator optimizes the query based on the data context of the data source.

10. The system of claim 1 wherein said query converter converts a portion of the query into at least one command which can be used to interact with a World Wide Web page by accessing a specification file associated with the data source, said specification file providing the commands necessary to access the World Wide Web page containing the requested data.

11. The system of claim 1 wherein said command transmitter optimizes the query by examining a specification file and determining if the commands listed by the specification file can be issued in order to access the World Wide Web page containing the requested data.

12. The system of claim 11 wherein said command transmitter separates the query into a plurality of sub-queries and queries at least one of the heterogeneous data sources using one of the sub-queries.

13. The system of claim 12 wherein the query transmitter queries a World Wide Web page with at least one of the sub-queries.

14. A method for querying semi-structured data sources distributed over a network, said method comprising the steps of:

(a) accessing a plurality of semi-structured data sources to be queried, each of which includes local information relating to a first semantic data context associated with the heterogeneous data source;

(b) receiving a request having a second semantic data context and comparing said first semantic data context and said second semantic data context to determine if a conflict between those contexts exists;

(c) translating the request responsive to the conflict determination into a query having a second data context associated with at least one of the semi-structured data sources to be queried;

(d) converting a portion of the query into at least one command which can be used to interact with a semi-structured data source;

(e) issuing the at least one command to at least one of the semi-structured data sources;

(f) retrieving data from at least one of the semi-structured data sources; and

(g) translating retrieved data from the data contexts associated with the semi-structured data sources into the data context associated with the request.

15. The method of claim 14 wherein step (d) further comprises converting a portion of the query into at least one command which can be used to interact with a World Wide Web page.

16. The method of claim 14 wherein step (d) further comprises converting a portion of the query into at least one command which can be used to interact with a flat file containing data.

17. The method of claim 14 wherein step (b) comprises generating a request having a second semantic data context and comparing said first semantic data context and said second semantic data context to determine if a conflict between those contexts exits.

18. The method of claim 14 further comprising the step of determining at least one semi-structured data source to query based on the request.

19. The method of claim 18 wherein said determining step comprises the step of determining at least one semi-structured data source to query based on an ontology.

20. The method of claim 18 further comprising the steps of:

detecting a difference between the context of data requested by the request and the context of data supplied by the data source to be queried; and

converting the data context-of the request into the data context of the data source.

21. The method of claim 14 further comprising the step of optimizing the query.

22. The method of claim 14 further comprising the step of querying at least one of the data sources using the translated request.

23. The method of claim 21 wherein said optimization step further comprises:

separating the query into a plurality of sub-queries; and

querying at least one of the semi-structured data sources using at least one of sub-queries.

24. The method of claim 23 wherein said querying step further comprises querying a different data source with each one of the sub-queries.

25. A method for querying semi-structured data sources in response to a structured data request, the method comprising the steps of:

(a) converting a data request into one or more commands which can be used to interact with a semi-structured data source;

(b) issuing at least one of the one or more commands to said semi-structured data source; and

(c) retrieving data from said semi-structured data source.

26. The method of claim 25 wherein step (a) further comprises converting a data request into one or more commands which can be used to interact with a World Wide Web page.

27. The method of claim 25 wherein step (a) further comprises converting a data request into one or more commands which can be used to interact with a flat file containing data.

28. The method of claim 25 wherein step (a) further comprises:

(a-a) determining if requested data is provided by one or more World Wide Web pages;

(a-b) determining, for each requested datum that is provided by a World Wide Web page, one or more commands which, when issued to the World Wide Web page, cause it to provide the requested datum.

29. The method of claim 28 wherein step (a-a) further comprises determining if requested data is provided by one or more World Wide Web pages by accessing a file stored in a memory element of a computer, said file including a list of all data the one or more World Wide Web page can provide.

30. The method of claim 28 wherein step (a-a) further comprises determining if requested data is provided by one or more World wide Web pages by accessing a file stored in a memory element of a computer, said file containing a list of all data the World Wide Web page can provide and a data context associated with each datum provided by the World Wide Web page.

31. The method of claim 28 wherein step (a-b) further comprises determining, for each requested datum that is provided by a World Wide Web page, one or more commands which cause the World Wide Web page to provide the requested datum, the determination made by accessing a file located in a memory element of a computer which contains at least one instruction to be issued to the World Wide Web page.

32. A system for retrieving data from a semi-structured data source in response to a request, the system comprising:

a plurality of heterogeneous data sources, each of which includes local information relating to a first semantic data context associated with the semi-structured data source;

a request converter for converting a request having an associated second semantic data context into one or more commands which can be used to interact with a semi-structured data source by comparing said first semantic data context and said second semantic data context to determine if a conflict between those contexts exists, said request being converted responsive to the conflict determination;

a command transmitter for issuing at least one of the one or more commands to said semi-structured data source; and

a data retriever for extracting data from said semi-structured data source.

33. The system of claim 32 wherein the semi-structured data source is a World Wide Web page.

34. The system of claim 32 wherein the semi-structured data is a flat file containing data.

35. The system of claim 32 wherein said request converter accesses a file contained in a memory element of a computer in order to determine which data can be retrieved from a World Wide Web page and accesses a second file contained in a memory element of a computer which specifies commands to be used to access one of the World Wide Web pages.

36. The system of claim 35 wherein the query converter accesses only one file.