Identifying entity mappings across data assets

• US 10,120,930 B2
• Filed: 09/16/2016
• Issued: 11/06/2018
• Est. Priority Date: 09/14/2015
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Deshpande, Prasad M., Dey, Atreyee, Gupta, Rajeev, Gupta, Sanjeev K., Joshi, Salil, Padmanabhan, Sriram K.
• Primary Examiner(s): Stevens, Robert
• Application Number:
  US15/268,400
• Publication Number:
  US 20170075898A1
• Time in Patent Office: 781 Days
• Field of Search: 707723
• CPC Class Codes: G06F 16/215 Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
• CPC Class Codes: G06F 16/288 Entity relationship models
• CPC Class Codes: G06F 16/319 Inverted lists
• CPC Class Codes: G06F 16/80 of semi-structured data, e.g. markup language structured data such as SGML, XML or HTML content-based retrieval of web data G06F16/95
• CPC Class Codes: G06F 16/86 Mapping to a database
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation

Abstract

Entity mappings that produce matching entities for a first data asset having attributes and a second data asset having attributes are generated by: generating entity mappings that produce matching entities for a first data asset having attributes with attribute values and a second data asset having attributes with attribute values by: matching the attribute values of the attributes of the first data asset with the attribute values of the attributes of the second data asset, using the matching attribute values to generate matching attribute pairs, and using the matching attribute pairs to identify entity mappings; computing an entity mapping score for each of the entity mappings based on a combination of factors; ranking the entity mappings based on each entity mapping score; and using some of the ranked entity mappings to determine whether a same real-world entity is described by the first data asset and the second data asset.

Claims

1. A method, comprising:

generating entity mappings that produce matching entities for a first data asset having attributes with attribute values and a second data asset having attributes with attribute values by;

matching the attribute values of the attributes of the first data asset with the attribute values of the attributes of the second data asset;

using the matching attribute values to generate matching attribute pairs; and

using the matching attribute pairs to identify entity mappings;

computing an entity mapping score for each of the entity mappings based on a combination of factors;

ranking the entity mappings based on each entity mapping score; and

using the ranked entity mappings to determine which of the entity mappings are to be used to determine whether a same real-world entity is described by the first data asset and the second data asset.

2. The method of claim 1, further comprising:

generating a first inverted index of entity identifier pairs for the first data asset;

generating a second inverted index of entity identifier pairs for the second data asset; and

using the first inverted index and the second inverted index to generate the matching attribute pairs based on matching attribute values that form the entity mappings.

3. The method of claim 1, wherein values match fuzzily for the matching entities.

4. The method of claim 1, wherein computing the entity mapping score for each of the entity mappings comprises:

generating an entity mapping score for factors selected from;

a number of attributes involved in an entity mapping, a cardinality of that individual entity mapping, support of that entity mapping, a probability of one to one matching for that entity mapping, a join utility measure for that entity mapping, and a probability of previous user selections for that entity mapping; and

adding the entity mapping score for each of the factors to generate the entity mapping score for that entity mapping.

5. The method of claim 1, wherein one of the first data asset and the second data asset is semi-structured data having hierarchical data that is flattened.

6. The method of claim 1, wherein one of the first data asset and the second data asset is an unstructured data asset formed by a collection of documents and is modelled based one of a bag of words and annotated words.

7. The method of claim 1, further comprising:

integrating the first data asset and the second data asset using ranked entity mappings by performing one of a join operation, a merge operation, and a union operation.

8. The method of claim 1, wherein Software as a Service (SaaS) is configured to perform method operations.