Querying and integrating structured and unstructured data

• US 9,037,615 B2
• Filed: 06/11/2012
• Issued: 05/19/2015
• Est. Priority Date: 05/14/2010
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Bornea, Mihaela Ancuta, Duan, Songyun, Fokoue-Nkoutche, Achille, Gliozzo, Alfio M., Kalyanpur, Aditya, Kementsietsidis, Anastasios, Srinivas, Kavitha, Ward, Michael J., Fan, James J.
• Primary Examiner(s): LIAO, JASON G
• Application Number:
  US13/493,174
• Publication Number:
  US 20130332478A1
• Time in Patent Office: 1,072 Days
• Field of Search: 707/705
• US Class Current: 707/803
• CPC Class Codes: G06F 16/211 Schema design and management
• CPC Class Codes: G06F 16/901 Indexing; Data structures therefor; Storage structures for retrieval from the web G06F16/951

Abstract

A computer-implemented method, system, and article of manufacture for querying and integrating structured and unstructured data. The method includes: receiving entity information that is extracted from a first set of unstructured data using an open domain information extraction system, wherein the entity in-formation comprises relationship information between a first entity and a second entity of the first set of unstructured data; recognizing a pattern based on the relationship information and creating a schema for the first set of unstructured data based on the pattern; and associating an element of the created schema with (i) an entity of a second set of unstructured data or (ii) a schema element of an existing set of structured data if there is sufficient overall similarity between the created schema element and either the second unstructured data entity or the schema element of the existing structured data.

Claims

1. A computer-implemented method of querying and integrating structured and unstructured data, the method comprising:

receiving entity information that is extracted from a first set of unstructured data using an open domain information extraction system, wherein the entity information comprises relationship information between first entities and second entities of the first set of unstructured data, and the relationship information stores semantics of verbs connecting the first entities and the second entities;

recognizing a pattern based on the relationship information by identifying repeated occasions of a first entity, a second entity and a verb connecting the first entity and the second entity in the first set of unstructured data;

creating a schema for the first set of unstructured data based on the pattern, wherein the schema comprises the first entity, the second entity and the verb as elements; and

associating an element of the created schema with (i) an entity of a second set of unstructured data or (ii) a schema element of an existing set of structured data if there is sufficient overall similarity between the created schema element and either the second unstructured data entity or the schema element of the existing structured data, thereby creating a link between the created schema element and either the second unstructured data entity or the schema element of the existing set of structured data.

2. The method according to claim 1, wherein the associating step comprises:

receiving a set of mapping rules; and

mapping the created schema element to (i) the second unstructured data entity or (ii) the schema element of the existing set of structured data based on the set of mapping rules to create the link.

3. The method according to claim 2, wherein the mapping step comprises:

measuring lexical similarity, documentation similarity, semantic similarity, and structural similarity between the created schema element and (i) the second unstructured data entity or (ii) the schema element of the existing set of structured data to determine if there is said sufficient overall similarity based on a given threshold value;

combining the created schema element and either the second unstructured data entity or the schema element of the existing set of structured data into a group of similar elements or element-entity combination, thereby creating the link, if there is said sufficient overall similarity; and

clustering the created schema element and either the second unstructured data entity or the schema element of the existing set of structured data into a group to be reviewed, if there is insufficient overall similarity,

wherein the measuring, combining, and clustering steps are based on the mapping rules.

4. The method according to claim 2, wherein the associating step further comprises:

receiving feedback on results of the mapping; and

improving the set of mapping rules based on the feedback to produce an improved set of mapping rules.

5. The method according to claim 1, further comprising formulating a query to be sent to a database that is selected from the group consisting of:

the first set of unstructured data,

the second set of unstructured data,

a third set of unstructured data,

the existing set of structured data, and

another existing set of structured data.

6. The method according to claim 1, further comprising searching an instance level link at query evaluation time.

7. A computer-implemented system for querying and integrating structured and unstructured data, system comprising:

A receiving device configured to receive entity information that is extracted from a first set of unstructured data using an open domain information extraction system, wherein the entity information comprises relationship information between a first entities and a second entities of the first set of unstructured data, and the relationship information stores semantics of verbs connecting the first entities and the second entities;

A pattern recognition device configured to recognize a pattern based on the relationship information by identifying repeated occasions of a first entity, a second entity and a verb connecting the first entity and the second in the first set of unstructured data;

creating a schema for the first set of unstructured data based on the pattern, wherein the schema comprises the first entity, the second entity and the verb as elements; and

an element association device configured to associate an element of the created schema with (i) an entity of a second set of unstructured data or (ii) a schema element of an existing set of structured data if there is sufficient overall similarity between the created schema element and either the second unstructured data entity or the schema element of the existing structured data, thereby creating a link between the created schema element and either the second unstructured data entity or the schema element of the existing set of structured data.

8. The computer-implemented system according to claim 7, wherein:

the receiving device is further configured to receive a set of mapping rules; and

the element association device comprises a mapping device configured to map the created schema element to (i) the second unstructured data entity or (ii) the schema element of the existing set of structured data based on the set of mapping rules to create the link.

9. The computer-implemented system according to claim 8, wherein:

the receiving device is further configured to receive a threshold value for determining overall similarity between schema elements; and

the mapping device, based on the mapping rules, is further configured to;

measure lexical similarity, documentation similarity, semantic similarity, and structural similarity between the created schema element and (i) the second unstructured data entity or (ii) the schema element of the existing set of structured data to determine if there is said sufficient overall similarity based on the threshold value;

combine the created schema element and either the second unstructured data entity or the schema element of the existing set of structured data into a group of similar elements or element-entity combination, thereby creating the link, if there is said sufficient overall similarity; and

cluster the created schema and either the second unstructured data entity or the schema element of the existing set of structured data into a group to be reviewed, if there is insufficient overall similarity.

10. The computer-implemented system according to claim 7, further comprising a querying device configured to formulate a query to be sent to a database that is selected from the group consisting of:

the first set of unstructured data,

the second set of unstructured data,

a third set of unstructured data,

the existing set of structured data, and

another existing set of structured data.

11. The computer-implemented system according to claim 10, wherein the querying device is further configured to search instance level links at query evaluation time.

12. The computer-implemented system according to claim 9, wherein the mapping device comprises a logic device configured to determine if there is said sufficient overall similarity or said insufficient overall similarity between the created schema element and either the second unstructured data entity or the schema element of the existing set of structured data.

13. The computer-implemented system according to claim 8, wherein

the receiving device is further configured to receive feedback on the results of the mapping;

and

the mapping device is further configured to incorporate the feedback on the mapping into the set of mapping rules and to produce an improved set of mapping rules.

14. A computer program product for querying and integrating structured and unstructured data, the computer program product comprising:

receiving entity information that is extracted from a first set of unstructured data using an open domain information extraction system, wherein the entity information comprises relationship information between a first entities and a second entities of the first set of unstructured data, and the relationship information stores semantics of verbs connecting the first entities and the second entities;

recognizing a pattern based on the relationship information by identifying repeated occasions of a first entity, a second entity and a verb connecting the first entity and the second entity in the first set of unstructured data;

creating a schema for the first set of unstructured data based on the pattern, wherein the schema comprises the first entity, the second entity and the verb as elements; and

associating an element of the created schema with (i) an entity of a second set of unstructured data or (ii) a schema element of an existing set of structured data if there is sufficient overall similarity between the created schema element and either the second unstructured data entity or the schema element of the existing structured data, thereby creating a link between the created schema element and either the second unstructured data entity or the schema element of the existing set of structured data.

15. The computer program product according to claim 14, wherein the associating step comprises:

receiving a set of mapping rules; and

mapping the created schema element to (i) the second unstructured data entity or (ii) the schema element of the existing set of structured data based on the set of mapping rules to create the link.

16. The computer program product according to claim 15, wherein the mapping step comprises:

measuring lexical similarity, documentation similarity, semantic similarity, and structural similarity between the created schema element and (i) the second unstructured data entity or (ii) the schema element of the existing set of structured data to determine if there is said sufficient overall similarity based on a given threshold value;

combining the created schema element and either the second unstructured data entity or the schema element of the existing set of structured data into a group of similar elements or element-entity combination, if there is said sufficient overall similarity; and

clustering the created schema element and either the second unstructured data entity or the schema element of the existing set of structured data into a group to be reviewed, if there is insufficient overall similarity.

17. The computer program product according to claim 15, wherein the associating step further comprises:

receiving feedback on results of the mapping; and

improving the set of mapping rules based on the feedback to produce an improved set of mapping rules.

18. The computer program product according to claim 14, wherein the method further comprises formulating a query to be sent to a database that is selected from the group consisting of:

the first set of unstructured data,

the second set of unstructured data,

a third set of unstructured data,

the existing set of structured data, and

another existing set of structured data.

19. The computer program according to claim 14, wherein the method further comprises searching instance level links at query evaluation time.