Method and architecture for data transformation, normalization, profiling, cleansing and validation

• US 20040083199A1
• Filed: 08/05/2003
• Published: 04/29/2004
• Est. Priority Date: 08/07/2002
• Status: Abandoned Application

• Current Assignee: David O. Mcgoveran, Diwakar R. Govindugari
• Original Assignee: David O. Mcgoveran, Diwakar R. Govindugari
• Inventors: Govindugari, Diwakar R., McGoveran, David O.
• Application Number:
  US10/635,891
• Publication Number:
  US 20040083199A1
• Time in Patent Office: Days
• US Class Current: 707/1
• CPC Class Codes: G06F 16/215 Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors

Abstract

This is a computer software architecture and method for managing data transformation, normalization, profiling, cleansing, and validation. In the preferred embodiment, the architecture and method includes seven integrated functional elements: Dispatcher to route data and metadata among system elements; Semantic Modeler to build semantic models; Model Mapper to associate related concepts between semantic models; Transformation Manager to capture transformation rules and apply them to data driven by maps between semantic models; Validation Manager to capture data constraints and apply them to data; Interactive Guides to assist the processes of semantic modeling and semantic model mapping; and Adapters to convert data to and from specialized formats and protocols.

Claims

1. A computer implemented method for integrating data, said method comprising:

creating at least a first and a second semantic model wherein said first semantic model is restricted to a first category of knowledge and said second semantic model is restricted to a second category of knowledge;

storing said semantic models;

mapping the stored first semantic model to the stored second semantic model, thereby creating a model mapping;

storing said model mapping;

accepting as input a first data associated with said first semantic model;

transforming said first data, according to said model mapping;

validating said first data according to a set of validation rules; and

,

forwarding said transformed and validated first data to at least a first software system.

2. A method as in claim 1, wherein said step of mapping is further augmented with at least a third semantic model and said third semantic model is restricted to a third category of knowledge.

3. A method as in claim 1, wherein said first and second categories of knowledge pertain to a common application domain.

4. A method as in claim 3, wherein the common application domain is further modeled by at least one topic semantic model.

5. A method as in claim 4, wherein at least a first topic is associated with the common application domain and the said association is maintained in a template.

6. A method as in claim 5, wherein the template incorporates a second topic, relationships among the first and second topics, and at least one pre-defined rule.

7. A method as in claim 2, wherein said third semantic model is a referent semantic model.

8. A method as in claim 1, wherein at least one of the semantic models describes the semantics of a message.

9. A method as in claim 1, wherein at least one of the semantic models describes the semantics of a Web Service.

10. A method as in claim 1, wherein at least one of the semantic models describes the semantics of a business document.

11. A method as in claim 1, wherein at least one of the semantic models describes the semantics of an XML document.

12. A method as in claim 1, wherein at least one of the semantic models describes the semantics of a database.

13. A method as in claim 1, wherein the step of creating the semantic models may be augmented at the discretion of a human user by importing a set of semantic information.

14. A method as in claim 13, wherein the set of semantic information is imported by means of a first adapter.

15. A method as in claim 1, wherein the step of creating the semantic models includes user modification of at least one of the said semantic models.

16. A method as in claim 1, wherein the step of creating the semantic models includes augmenting the semantic models indirectly with at least one validation rule.

17. A method as in claim 1, wherein the step of creating the semantic models includes augmenting the semantic models indirectly with at least one transformation rule.

18. A method as in claim 1, wherein at least one of the semantic models is implemented as an ontology.

19. A method as in claim 1, wherein at least one of the semantic models is represented by a standard knowledge description and querying language.

20. A method as in claim 13, wherein the semantic information is processed according to at least a first rule in order to accomplish at least one of the operations of data profiling, semantic mapping, semantic resolution, data cleansing, normalization, transformation, and validation.

21. A method as in claim 1, wherein said step of mapping the stored first semantic model to the stored second semantic model further comprises:

selecting and accessing said first semantic model based on association with a source;

selecting and accessing said second semantic model based on association with a destination;

presenting the semantic models to a user;

eliciting selection of a first semantic element belonging to the first semantic model;

eliciting selection of a second semantic element belonging to the second semantic model;

establishing an association between the first semantic element and the second semantic element;

providing the option of using system help as needed;

defining each relevant transformation rule;

defining each relevant validation rule;

providing the option of storing the resulting model mapping;

permitting editing of the association; and

,

storing the model mapping.

22. A method as in claim 21, where in the step of providing the option of using system help is accomplished using an Interactive Guide.

23. A method as in claim 22, wherein the method implemented by said Interactive Guide comprises the steps of:

creating at least one candidate mapping between elements of said first semantic model and said second semantic model;

assigning a weight to each said candidate mapping, said weight derived from one or more portions that may be individually computed;

evaluating each candidate mapping and eliminating any candidate mapping that is invalid;

presenting a set of one or more candidate mappings to a human user;

eliciting from the user selection of at least one weighted candidate mapping in the set; and

,

modifying the model mapping according to the user selection.

24. A method as in claim 23, wherein the weight assigned to the candidate mapping is determined according to one or more heuristic rules, each of which determines a portion of said weight.

25. A method as in claim 24, wherein at least one heuristic rule is defined the user.

26. A method as in claim 24, wherein at least one heuristic rule is modified by a human user.

27. A method as in claim 24, wherein a first heuristic rule is pre-defined and a criterion of applicability of the heuristic rule is determined by a human user.

28. A method as in claim 23 wherein the system identifies those portions of the weight that cannot change on recalculation and does not recalculate them once they have been calculated.

29. A method as in claim 23, wherein the inclusion of each candidate mapping in the set is decided based on the weight of that candidate mapping.

30. A method as in claim 29, wherein the inclusion of each candidate mapping in the set is decided based on the weight of that candidate mapping exceeding a threshold.

31. A method as in claim 30, wherein the threshold may be modified by the user.

32. A method as in claim 23, wherein the number of candidate mappings included in the set is limited to a maximum number.

33. A method as in claim 32, wherein the maximum number may be modified by the user.

34. A method as in claim 23, wherein the user obtains an explanation of the weight of a selected candidate mapping was computed.

35. A method as in claim 23, wherein the user may modify any portion of the weight.

36. A method as in claim 23, wherein the user may modify the method by which the weight is derived.

37. A method as in claim 1, wherein the means of accepting data is via an Adapter.

38. A method as in claim 37, wherein the Adapter is a SOAP Message Handler.

39. A method as in claim 1, wherein the means of forwarding data is via an Adapter.

40. A method as in claim 39, wherein the Adapter is a SOAP Message Handler.

41. A general-purpose computer incorporating specific hardware and software for transforming, profiling, cleansing, normalizing, and validating data, wherein said specific hardware and software comprise:

means for defining at least a first semantic model and a second semantic model;

means for defining a model mapping among semantic models;

means for storing said semantic models and said model mapping;

means for defining validation rules and transformation rules;

means for accepting data from at least one source;

means for transforming said data according to the model mapping;

means for validating said data; and

,

means for forwarding said data to at least one destination.