Method, system, and product for data integration through a dynamic common model

• US 7,257,594 B2
• Filed: 04/06/2004
• Issued: 08/14/2007
• Est. Priority Date: 05/07/2001
• Status: Expired due to Term

• Current Assignee: Petris Technology Corporation
• Original Assignee: Petris Technology Corporation
• Inventors: Jacobs, John, Tamboli, Aderbad
• Primary Examiner(s): WOO, ISAAC M
• Application Number:
  US10/818,618
• Publication Number:
  US 20040230605A1
• Time in Patent Office: 1,225 Days
• Field of Search: 707 1- 10, 707100-1041, 707200-205
• US Class Current: 1/1
• CPC Class Codes: G06F 16/258 Data format conversion from or to a database
• CPC Class Codes: Y10S 707/99932 Access augmentation or optimizing
• CPC Class Codes: Y10S 707/99933 Query processing, i.e. searching
• CPC Class Codes: Y10S 707/99942 Manipulating data structure, e.g. compression, compaction, compilation
• CPC Class Codes: Y10S 707/99943 Generating database or data structure, e.g. via user interface
• CPC Class Codes: Y10S 707/99944 Object-oriented database structure
• CPC Class Codes: Y10S 707/99952 Coherency, e.g. same view to multiple users
• CPC Class Codes: Y10S 707/99953 Recoverability

Abstract

Data integration including extracting a first native record having a first native format from a first native repository through a first adapter, the first adapter loosely coupled for data integration to a data integration application, the first native format having a datatype; transforming the first native record having first native format to a first native record having dynamic common format, the dynamic common format being a subset of a dynamic common model, the dynamic common model comprising mappings to and from the dynamic common format for all native records in all datatypes; transforming the format of the first native record having dynamic common format to a first native record having second native format; and inserting through a second adapter, also loosely coupled to the application, the first native record the second native format into a second native repository.

Claims

1. A method of data integration with respect to data stores in native repositories, the method implemented in conjunction with a data integration application coupled for data communications through a multiplicity of adapters to a multiplicity of native repositories, the native repositories comprising native records having formats having datatypes supported by the native repositories, the method comprising the steps of:

extracting through a first adapter from a first native repository a first native record having a first native format, the first native format belonging to a category of formats identified as a first datatype;

transforming, through the first adapter, the first native record having a first native format to a first native record having dynamic common format;

transforming, through a second adapter, the first native record having dynamic common format to a first native record having second native format, the second native format belonging to a category of formats identified as the first datatype; and

inserting, through the second adapter, the first native record having a second native format into a second native repository;

wherein the first adapter and the second adapter each are loosely coupled for data integration to the data integration application;

wherein dynamic common format comprises a subset of a dynamic common model, the dynamic common model comprising mappings specifying transformations to and from dynamic common formats for all native records having all datatypes supported in all native repositories coupled through adapters to the data integration application;

finding in a catalog, in dependence upon search parameters, catalog records having identifying attributes that match the search parameters,

wherein the identifying attributes for each catalog record include a catalog key for each catalog record,

wherein the catalog comprises identifying attributes and proxy data for all native records in a multiplicity of native repositories,

wherein the multiplicity of native repositories comprises the first native repository, and

wherein at least one found catalog record contains identifying attributes that identify the first native record;

marking for extraction the identifying attributes of the at least one found catalog record containing identifying attributes that identify the first native record;

posting from the marked identifying attributes a catalog key to a transfer cart in the data integration application; and

extracting, in dependence upon the posted catalog key, from the catalog through a catalog adapter proxy data for the first native record, wherein the proxy data comprises data representing the location of the first native record in the first native repository;

wherein extracting a first native record from a first native repository further comprises reading, in dependence upon the proxy data, though the first adapter, from the first native repository, the first native record having a first native format.

2. The method of claim 1 wherein the data integration application comprises a catalog, the method further comprising finding in a catalog at least one catalog record containing identifying attributes that identify the first native record.

3. The method of claim 2 wherein the catalog record comprises a catalog key and the data integration application further comprises a transfer cart, the method further comprising posting the catalog key to the transfer cart.

4. The method of claim 3 further comprising extracting, in dependence upon the posted catalog key, from the catalog through a catalog adapter proxy data for the first native record, wherein the proxy data comprises data representing the location of the first native record in the first native repository.

5. The method of claim 4 wherein extracting a first native record from a first native repository further comprises reading, in dependence upon the proxy data, through the first adapter, from the first native repository, the first native record having a first native format.

6. The method of claim 3 wherein

the first mapping comprises a first XML stylesheet,

the translator comprises an XSL translator,

the first native mapping format is implemented in XML,

the dynamic common format is implemented in XML,

the first native record having a first native mapping format is a first XML document,and

the first native record having dynamic common format is a second XML document.

7. The method of claim 1, wherein transforming the format of the first native record having the first native format comprises the further steps of:

converting, through the first adapter, the first native record having the first native format to a first native record having a first native mapping format;

retrieving from a mapping store a first mapping, wherein the first mapping specifies a data transformation from the first native mapping format to dynamic common format; and

translating, through a translator, in dependence upon the first mapping, the first native record having a first native mapping format to first native record having a dynamic common format.

8. The method of claim 1, wherein transforming the format of the first native record having dynamic common format comprises the further steps of:

receiving, through the second adapter, the first native record having dynamic common format;

retrieving from a mappings store a second mapping, wherein the second mapping specifies a data transformation from dynamic common format to a second native mapping format;

translating, through a translator, in dependence upon the second mapping, the first native record having dynamic common format, into a first native record having the second native mapping format; and

converting, through the second adapter, the format of the first native record having the second native mapping format into a first native record having the second native format.

9. The method of claim 8 wherein

the second mapping comprises an XML stylesheet,

the translator is an XSL translator,

the dynamic common format is implemented in XML,

the second native mapping format is implemented in XML,

the first native record having dynamic common format is a first XML document, and

the first native record having a second native mapping format comprises a second XML document.

10. The method of claim 1 wherein inserting through the second adapter comprises the further steps of:

writing, through the second adapter, the first native record having a second native format into the second native repository, whereby is created a new second native record;

creating new proxy data and identifying attributes for the new second native record; and

inserting the new proxy data and identifying attributes through a catalog adapter into a catalog;

wherein the catalog comprises identifying attributes and proxy data for all native records in a multiplicity of native repositories, wherein the multiplicity of native repositories comprises the second native repository.

11. The method of claim 1, further comprising:

spidering through a spider proxy data and identifying attributes from a single native repository to a catalog, wherein the single native repository is coupled for data communications to an adapter, wherein the adapter is coupled for data communications to a data integration application, wherein the data integration application comprises the spider;

wherein the catalog comprises a database of identifying attributes and proxy data for all native records in a multiplicity of native repositories, wherein the multiplicity of native repositories comprises the single native repository.

12. The method of claim 11, wherein spidering further comprises the steps of:

providing to the spider an identification code for the single native repository;

retrieving from the single native repository all native records in the repository;

creating, in dependence upon the retrieved native records, proxy data and identifying attributes for all the retrieved native records;

writing to the catalog, through the catalog adapter, the created proxy data and identifying attributes.

13. The method of claim 11, wherein spidering further comprises the steps of:

providing to the spider an identification code for the single native repository;

reading, in dependence upon the identification code for the single native repository, from a native repository registration list a last spider time for the native repository to be spidered;

reirieving from the single native repository native records having time stamps later than the last spider time;

creating, in dependence upon the retrieved native records, proxy data and identifying attributes for all the retrieved native records;

writing to the catalog, through the catalog adapter, the created proxy data and identifying attributes; and

updating the last spider time in the native repository registration list.

14. A system for data integration with respect to data stores in native repositories, the system implemented in conjunction with a data integration application coupled for data communications through a multiplicity of adapters to a multiplicity of native repositories, the native repositories comprising native records having formats having datatypes supported by the native repositories, the system comprising:

means for extracting through a first adapter from a first native repository a first native record having a first native format, the first native format belonging to a category of formats identified as a first datatype;

means for transforming, through the first adapter, the first native record having a first native format to a first native record having dynamic common format;

means for transforming, through a second adapter, the first native record having dynamic common format to a first native record having second native format, the second native format belonging to a category of formats identified as the first datatype; and

means for inserting, through the second adapter, the first native record having a second native format into a second native repository;

wherein the first adapter and the second adapter each are loosely coupled for data integration to the data integration application;

wherein dynamic common format comprises a subset of a dynamic common model, the dynamic common model comprising mappings specifying transformations to and from dynamic common formats for all native records having all datatypes supported in all native repositories coupled through adapters to the data integration application;

means for finding in a catalog, in dependence upon search parameters, catalog records having identifying attributes that match the search parameters,

wherein the identifying attributes for each catalog record include a catalog key for each catalog record,

wherein the catalog comprises identifying attributes and proxy data for all native records in a multiplicity of native repositories,

wherein the multiplicity of native repositories comprises the first native repository; and

wherein at least one found catalog record contains identifying attributes that identify the first native record;

means for marking for extraction the identifying attributes of the at least one found catalog record containing identifying attributes that identify the first native record;

means for posting from the marked identifying attributes a catalog key to a transfer cart in the data integration application; and

means for extracting, in dependence upon the posted catalog key, from the catalog through a catalog adapter proxy data for the first native record, wherein the proxy data comprises data representing the location of the first native record in the first native repository;

wherein means for extracting a first native record from a first native repository further comprises means for reading, in dependence upon the proxy data, through the first adapter, from the first native repository, the first native record having a first native format.

15. The system of claim 14 wherein the data integration application comprises a catalog, the system further comprising means for finding in a catalog at least one catalog record containing identifying attributes that identify the first native record.

16. The system of claim 15 wherein the catalog record comprises a catalog key and the data integration application further comprises a transfer cart, the system further comprising means for posting the catalog key to the transfer cart.

17. The system of claim 16 further comprising means for extracting, in dependence upon the posted catalog key, from the catalog through a catalog adapter proxy data for the first native record, wherein the proxy data comprises data representing the location of the first native record in the first native repository.

18. The system of claim 17 wherein means for extracting a first native record from a first native repository further comprises means for reading, in dependence upon the proxy data, through the first adapter, from the first native repository, the first native record having a first native format.

19. The system of claim 14, wherein means for transforming the format of the first native record having the first native format further comprises:

means for converting, through the first adapter, the first native record having the first native format to a first native record having a first native mapping format;

means for retrieving from a mapping store a first mapping, wherein the first mapping specifies a data transformation from the first native mapping format to dynamic common format; and

means for translating, through a translator, in dependence upon the first mapping, the first native record having a first native mapping format to first native record having a dynamic common format.

20. The system of claim 19 wherein

the first mapping comprises a first XML stylesheet,

the translator comprises an XSL translator,

the first native mapping format is implemented in XML,

the dynamic common format is implemented in XML,

the first native record having a first native mapping format is a first XML document, and

the first native record having dynamic common format is a second XML document.

21. The system of claim 14, wherein means for transforming the format of the first native record having dynamic common format further comprises:

means for receiving, through the second adapter, the first native record having dynamic common format;

means for retrieving from a mappings store a second mapping, wherein the second mapping specifies a data transformation from dynamic common format to a second native mapping format;

means for translating, through a translator, in dependence upon the second mapping, the first native record having dynamic common format, into a first native record having the second native mapping format; and

means for converting, through the second adapter, the format of the first native record having the second native mapping format into a first native record having the second native format.

22. The system of claim 21 wherein

the second mapping comprises an XML stylesheet,

the translator is an XSL translator,

the dynamic common format is implemented in XML,

the second native mapping format is implemented in XML,

the first native record having dynamic common format is a first XML document, and

he first native record having a second native mapping format comprises a second XML document.

23. The system of claim 14 wherein means for inserting through the second adapter further comprises:

means for writing, through the second adapter, the first native record having a second native format into the second native repository, wherein the first native record having second native format comprises a new second native record;

means for creating new proxy data and identifying attributes for the new second native record; and

means for inserting the new proxy data and identifying attributes through a catalog adapter into a catalog;

wherein the catalog comprises identifying attributes and proxy data for all native records in a multiplicity of native repositories, wherein the multiplicity of native repositories comprises the second native repository.

24. The system of claim 14, further comprising:

means for spidering through a spider proxy data and identifying attributes from a single native repository to a catalog, wherein the single native repository is coupled for data communications to an adapter, wherein the adapter is coupled for data communications to a data integration application, wherein the data integration application comprises the spider;

wherein the catalog comprises a database of identifying attributes and proxy data for all native records in a multiplicity of native repositories, wherein the multiplicity of native repositories comprises the single native repository.

25. The system of claim 24, wherein means for spidering further comprises:

means for providing to the spider an identification code for the single native repository;

means for retrieving from the single native repository all native records in the repository;

means for creating, in dependence upon the retrieved native records, proxy data and identifying attributes for all the retrieved native records;

means for writing to the catalog, through the catalog adapter, the created proxy data and identifying attributes.

26. The system of claim 24, wherein means for spidering further comprises:

means for providing to the spider an identification code for the single native repository;

means for reading, in dependence upon the identification code for the single native repository, from a native repository registration list a last spider time for the native repository to be spidered;

means for retrieving from the single native repository native records having time stamps later than the last spider time;

means for creating, in dependence upon the retrieved native records, proxy data and identifying attributes for all the retrieved native records;

means for writing to the catalog, through the catalog adapter, the created proxy data and identifying attributes; and

means for updating the last spider time in the native repository registration list.

27. A computer program product for data integration of a multiplicity of native repositories coupled for data communications through a multiplicity of adapters to at least one data integration application, the computer program product including the data integration application, the native repositories comprising native records having formats having datatypes supported by the native repositories, the computer program product comprising:

a recording medium;

means, recorded on the recording medium, for extracting through a first adapter from a first native repository a first native record having a first native format, the first native format belonging to a category of formats identified as a first datatype;

means, recorded on the recording medium, for transforming, through the first adapter, the first native record having a first native format to a first native record having dynamic common format;

means, recorded on the recording medium, for transforming, through a second adapter, the first native record having dynamic common format to a first native record having second native format, the second native format belonging to a category of formats identified as the first datatype; and

means, recorded on the recording medium, for inserting, through the second adapter, the first native record having a second native format into a second native repository;

wherein the first adapter and the second adapter each are loosely coupled for data integration to the data integration application;

wherein dynamic common format comprises a subset of a dynamic common model, the dynamic common model comprising mappings specifying transformations to and from dynamic common formats for all native records having all datatypes supported in all native repositories coupled through adapters to the data integration application;

means, recorded on the recording medium, for finding in a catalog, in dependence upon search parameters, catalog records having identifying attributes that match the search parameters,

wherein the identifying attributes for each catalog record include a catalog key for each catalog record,

wherein the catalog comprises identifying attributes and proxy data for all native records in a multiplicity of native repositories,

wherein the multiplicity of native repositories comprises the first native repository, and

wherein at least one found catalog record contains identifying attributes that identify the first native record;

means, recorded on the recording medium, for marking for extraction the identifying attributes of the at least one found catalog record containing identifying attributes that identify the first native record;

means, recorded on the recording medium, for posting from the marked identifying attributes a catalog key to a transfer cart in the data integration application; and

means, recorded on the recording medium, for extracting, in dependence upon the posted catalog key, from the catalog through a catalog adapter proxy data for the first native record, wherein the proxy data comprises data representing the location of the first native record in the first native repository;

wherein means for extracting a first native record from a first native repository further comprises means, recorded on the recording medium, for reading, in dependence upon the proxy data, through the first adapter, from the first native repository, the first native record having a first native format.

28. The computer program product of claim 27 wherein the data integration application comprises a catalog, the computer program product further comprising means, recorded on the recording medium, for finding in a catalog at least one catalog record containing identifying attributes that identify the first native record.

29. The computer program product of claim 28 wherein the catalog record comprises a catalog key and the data integration application further comprises a transfer cart, the computer program product further comprising means, recorded on the recording medium, for posting the catalog key to the transfer cart.

30. The computer program product of claim 29 further comprising means, recorded on the recording medium, for extracting, in dependence upon the posted catalog key, from the catalog through a catalog adapter proxy data for the first native record, wherein the proxy data comprises data representing the location of the first native record in the first native repository.

31. The computer program product of claim 30 wherein means for extracting a first native record from a first native repository further comprises means, recorded on the recording medium, for reading, in dependence upon the proxy data, through the first adapter, from the first native repository, the first native record having a first native format.

32. The computer program product of claim 27, wherein means for transforming the format of the first native record having the first native format further comprises:

means, recorded on the recording medium, for converting, through the first adapter, the first native record having the first native format to a first native record having a first native mapping format;

means, recorded on the recording medium, for retrieving from a mapping store a first mapping, wherein the first mapping specifies a data transformation from the first native mapping format to dynamic common format; and

means, recorded on the recording medium, for translating, through a translator, in dependence upon the first mapping, the first native record having a first native mapping format to first native record having a dynamic common format.

33. The computer program product of claim 32 wherein

the first mapping comprises a first XML stylesheet,

the translator comprises an XSL translator,

the first native mapping format is implemented in XML,

the dynamic common format is implemented in XML,

the first native record having a first native mapping format is a first XML document, and

the first native record having dynamic common format is a second XML document.

34. The computer program product of claim 27, wherein means for transforming the format of the first native record having dynamic common format further comprises:

means, recorded on the recording medium, for receiving, through the second adapter, the first native record having dynamic common format;

means, recorded on the recording medium, for retrieving from a mappings store a second mapping, wherein the second mapping specifies a data transformation from dynamic common format to a second native mapping format;

means, recorded on the recording medium, for translating, through a translator, in dependence upon the second mapping, the first native record having dynamic common format, into a first native record having the second native mapping format; and

means, recorded on the recording medium, for converting, through the second adapter, the format of the first native record having the second native mapping format into a first native record having the second native format.

35. The computer program product of claim 34 wherein

the second mapping comprises an XML stylesheet,

the translator is an XSL translator,

the dynamic common format is implemented in XML,

the second native mapping format is implemented in XML,

the first native record having dynamic common format is a first XML document, and

the first native record having a second native mapping format comprises a second XML document.

36. The computer program product of claim 27 wherein means for inserting through the second adapter further comprises:

means, recorded on the recording medium, for writing, through the second adapter, the first native record having a second native format into the second native repository, wherein the first native record having second native format comprises a new second native record;

means, recorded on the recording medium, for creating new proxy data and identifying attributes for the new second native record; and

means, recorded on the recording medium, for inserting the new proxy data and identifying attributes through a catalog adapter into a catalog;

wherein the catalog comprises identifying attributes and proxy data for all native records in a multiplicity of native repositories, wherein the multiplicity of native repositories comprises the second native repository.

37. The computer program product of claim 27, further comprising:

means, recorded on the recording medium, for spidering through a spider proxy data and identifying attributes from a single native repository to a catalog, wherein the single native repository is coupled for data communications to an adapter, wherein the adapter is coupled for data communications to a data integration application, wherein the data integration application comprises the spider;

wherein the catalog comprises a database of identifying attributes and proxy data for all native records in a multiplicity of native repositories, wherein the multiplicity of native repositories comprises the single native repository.

38. The computer program product of claim 37, wherein means for spidering further comprises:

means, recorded on the recording medium, for providing to the spider an identification code for the single native repository;

means, recorded on the recording medium, for retrieving from the single native repository all native records in the repository;

means, recorded on the recording medium, for creating, in dependence upon the retrieved native records, proxy data and identifying attributes for all the retrieved native records;

means, recorded on the recording medium, for writing to the catalog, through the catalog adapter, the created proxy data and identifying attributes.

39. The computer program product of claim 37, wherein means for spidering further comprises:

means, recorded on the recording medium, for providing to the spider an identification code for the single native repository;

means, recorded on the recording medium, for reading, in dependence upon the identification code for the single native repository, from a native repository registration list a last spider time for the native repository to be spidered;

means, recorded on the recording medium, for retrieving from the single native repository native records having time stamps later than the last spider time;

means, recorded on the recording medium, for creating, in dependence upon the retrieved native records, proxy data and identifying attributes for all the retrieved native records;

means, recorded on the recording medium, for writing to the catalog, through the catalog adapter, the created proxy data and identifying attributes; and

means, recorded on the recording medium, for updating the last spider time in the native repository registration list.