Use of federation services and transformation services to perform extract, transform, and load (ETL) of unstructured information and associated metadata

• US 7,774,301 B2
• Filed: 12/21/2006
• Issued: 08/10/2010
• Est. Priority Date: 12/21/2006
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Johnson, Sean Allen, Reese, Angela Fagundes, Wolfe, Anderson Edward, Parikh, Amisha, Sathyam, Ravishankar, Weise, Jedd Samuel, Vars, Clifford J. IV
• Primary Examiner(s): Cottingham; John R.
• Assistant Examiner(s): Allen; Nicholas E
• Application Number:
  US11/614,923
• Publication Number:
  US 20080154927A1
• Time in Patent Office: 1,328 Days
• Field of Search: None
• US Class Current: 707/602
• CPC Class Codes: G06F 16/254 Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses

Abstract

Provided are techniques for transforming unstructured information into content in a uniform context. The unstructured information and metadata associated with the unstructured information are extracted from one or more source content repositories. One or more custom transformations are performed on at least one of the unstructured information and the metadata. At least one of the transformed, unstructured information and the metadata are loaded into one or more target content repositories.

Claims

1. A computer-implemented method for transforming unstructured information into content in a uniform context, comprising:

with a federation service of a computer including a processor that presents a single view of source content repositories to a user;

receiving a query specifying source content groups stored in a set of the source content repositories;

running the query to retrieve metadata schemas of the source content groups, wherein each source content group has a metadata schema that describes a structure of metadata associated with the unstructured information in the source content group;

extracting the unstructured information and metadata associated with the unstructured information from the set of the source content repositories;

in response to user input, receiving selection of target content groups in another set of target content repositories;

in response to receiving the selection of the target content groups, identifying metadata schemas of the target content groups, wherein each metadata schema describes a structure of metadata associated with the unstructured information in the target content group;

creating a schema definition file including the retrieved metadata schemas of the source content groups and the identified metadata schemas of the target content groups;

forwarding the unstructured information, metadata, and schema definition file to a transformation service of the computer;

receiving, from the transformation service, transformed unstructured information and transformed metadata; and

loading the transformed, unstructured information and the transformed metadata into the set of the target content repositories.

2. The computer-implemented method of claim 1, wherein the unstructured information is in a plurality of different formats.

3. The computer-implemented method of claim 1, further comprising:

using a transformation service to perform custom mappings to map the retrieved metadata schemas of the source content groups to metadata schemas of the target content groups and to perform custom transformations on at least one of the unstructured information and the metadata.

4. The computer-implemented method of claim 1, further comprising:

using the custom mappings to map elements of the metadata schemas for the source content groups and the target content groups.

5. The computer-implemented method of claim 1, further comprising:

extracting the unstructured information from the source content groups in the set of the source content repositories; and

loading the unstructured information into the target content groups in the set of the target content repositories.

6. The computer-implemented method of claim 1, further comprising:

in response to determining that the target content groups do not exist, creating the target content groups in the set of the target content repositories using the metadata schemas for the source content groups in the set of the source content repositories.

7. The computer-implemented method of claim 1, further comprising:

in response to determining that the metadata schemas of the target content groups into which content is to be loaded do not exist, creating the metadata schemas of the target content groups using the metadata schemas of the source content groups in the set of the source content repositories.

11. The computer program product of claim 1, wherein the computer readable program when executed on a computer causes the computer to:

use the custom mappings to map elements of the metadata schemas for the source content groups and the target content groups.

8. A computer program product comprising a computer readable storage medium including a computer readable program, wherein the computer readable program when executed by a processor on a computer causes the computer to:

with a federation service of the computer that presents a single view of source content repositories to a user;

receive a query specifying source content groups stored in a set of the source content repositories;

run the query to retrieve metadata schemas of the source content groups, wherein each source content group has a metadata schema that describes a structure of metadata associated with the unstructured information in the source content group;

extract the unstructured information and metadata associated with the unstructured information from the set of the source content repositories;

in response to user input, receive selection of target content groups in another set of target content repositories;

in response to receiving the selection of the target content groups, identify metadata schemas of the target content groups, wherein each metadata schema describes a structure of metadata associated with the unstructured information in the target content group;

create a schema definition file including the retrieved metadata schemas of the source content groups and the identified metadata schemas of the target content groups;

forward the unstructured information, metadata, and schema definition file to a transformation service of the computer;

receive, from the transformation service, transformed unstructured information and transformed metadata; and

load the transformed, unstructured information and the metadata into the set of the target content repositories.

9. The computer program product of claim 8, wherein the unstructured information is in a plurality of different formats.

10. The computer program product of claim 8, wherein the computer readable program when executed on a computer causes the computer to:

use a transformation service to perform custom mappings to map the retrieved metadata schemas of the source content groups to metadata schemas of the target content groups and to perform custom transformations on at least one of the unstructured information and the metadata.

12. The computer program product of claim 8, wherein the computer readable program when executed on a computer causes the computer to:

extract the unstructured information from the source content groups in the set of the source content repositories; and

load the unstructured information into the target content groups in the set of the target content repositories.

13. The computer program product of claim 8, wherein the computer readable program when executed on a computer causes the computer to:

in response to determining that the target content groups do not exist, create the target content groups in the set of the target content repositories using the metadata schemas for the source content groups in the set of the source content repositories.

14. The computer program product of claim 8, wherein the computer readable program when executed on a computer causes the computer to:

in response to determining that the metadata schemas of the target content groups into which content is to be loaded do not exist, create the metadata schemas of the target content groups using the metadata schemas of the source content groups in the set of the source content repositories.

15. A system for transforming unstructured information into content in a uniform context, comprising:

hardware logic implemented in a computer to perform operations, the operations comprising;

with a federation service of the computer that presents a single view of source content repositories to a user;

receiving a query specifying source content groups stored in a set of the source content repositories;

running the query to retrieve metadata schemas of the source content groups, wherein each source content group has a metadata schema that describes a structure of metadata associated with the unstructured information in the source content group;

extracting the unstructured information and metadata associated with the unstructured information from the set of the source content repositories;

in response to user input, receiving selection of target content groups in another set of target content repositories;

in response to receiving the selection of the target content groups, identifying metadata schemas of the target content groups, wherein each metadata schema describes a structure of metadata associated with the unstructured information in the target content group;

creating a schema definition file including the retrieved metadata schemas of the source content groups and the identified metadata schemas of the target content groups;

forwarding the unstructured information, metadata, and schema definition file to a transformation service of the computer;

receiving, from the transformation service, transformed unstructured information and transformed metadata; and

loading the transformed, unstructured information and the metadata into the set of the target content repositories.

16. The system of claim 15, wherein the unstructured information is in a plurality of different formats.

17. The system of claim 15, wherein the operations further comprise:

using a transformation service to perform custom mappings to map the retrieved metadata schemas of the source content groups to metadata schemas of the target content groups and to perform custom transformations on at least one of the unstructured information and the metadata.

18. The system of claim 15, wherein the operations further comprise:

using the custom mappings to map elements of the metadata schemas for the source content groups and the target content groups.

19. The system of claim 15, wherein the operations further comprise:

extracting the unstructured information from the source content groups in the set of the source content repositories; and

loading the unstructured information into the target content groups in the set of the target content repositories.

20. The system of claim 15, wherein the operations further comprise:

in response to determining that the target content groups do not exist, creating the target content groups in the set of the target content repositories using the metadata schemas for the source content groups in the set of the source content repositories.

21. The system of claim 15, wherein the operations further comprise:

in response to determining that the metadata schemas of the target content groups into which content is to be loaded do not exist, creating the metadata schemas of the target content groups using the metadata schemas of the source content groups in the set of the source content repositories.

22. A computer-implemented method for transforming unstructured information and associated metadata into content in a uniform context, comprising:

using a federation service of a computer including a processor that presents a single view of source content repositories to a user;

receiving a query specifying source content groups in a set of the source content repositories;

running the query to retrieve metadata schemas of the source content groups specified in the query, wherein each source content group has a metadata schema that describes a structure of metadata associated with the unstructured information in the source content group;

extracting the unstructured information and metadata associated with the unstructured information from the set of the source content repositories;

in response to user input, receiving selection of target content groups in another set of target content repositories;

in response to receiving the selection of the target content groups, identifying metadata schemas of the target content groups, wherein each metadata schema describes a structure of metadata associated with the unstructured information in a target content group;

creating a schema definition file including the extracted metadata schemas of the source content groups and the identified metadata schemas of the target content groups;

forwarding the unstructured information, metadata, and schema definition file to a transformation service of the computer;

using the transformation service,

performing one or more custom mappings on at least one of the unstructured information and the associated metadata by mapping elements of the extracted metadata schemas of the source content groups to the identified metadata schemas of the target content groups;

transforming at least one of the unstructured information and the associated metadata with custom transformations; and

forwarding the mapped and transformed unstructured information and the associated metadata to the federation service.

23. The computer-implemented method of claim 22, further comprising:

using the federation service;

receiving, from the transformation service, mapped and transformed unstructured information and the associated metadata; and

loading the mapped and transformed unstructured information and the associated metadata into the target content groups.

24. A computer program product comprising a computer readable storage medium storing a computer readable program, wherein the computer readable program when executed by a processor on a computer causes the computer to:

using a federation service of the computer that presents a single view of source content repositories to a user;

receive a query specifying source content groups in a set of the source content repositories;

run the query to retrieve metadata schemas of the source content groups specified in the query, wherein each source content group has a metadata schema that describes a structure of metadata associated with the unstructured information in the source content group;

extract the unstructured information and metadata associated with the unstructured information from the set of the source content repositories;

in response to user input, receive selection of target content groups in another set of target content repositories;

in response to receiving the selection of the target content groups, identify metadata schemas of the target content groups, wherein each metadata schema describes a structure of metadata associated with the unstructured information in a target content group;

creating a schema definition file including the extracted metadata schemas of the source content groups and the identified metadata schemas of the target content groups;

forwarding the unstructured information, metadata, and schema definition file to a transformation service of the computer; and

using the transformation service, perform one or more custom mappings on at least one of the unstructured information and the associated metadata by mapping elements of the extracted metadata schemas of the source content groups to the identified metadata schemas of the target content groups;

transform at least one of the unstructured information and the associated metadata with custom transformations; and

forward the mapped and transformed unstructured information and the associated metadata to the federation service.

25. The computer program product of claim 24, wherein the computer readable program when executed on a computer causes the computer to:

using the federation service;

receive, from the transformation service, mapped and transformed unstructured information and the associated metadata; and

load the mapped and transformed unstructured information and the associated metadata into the target content groups.

26. A system for transforming unstructured information and associated metadata into content in a uniform context, comprising:

hardware logic implemented in a computer to perform operations, the operations comprising;

using a federation service of the computer that presents a single view of source content repositories to a user;

receiving a query specifying source content groups in a set of the source content repositories;

running the query to retrieve metadata schemas of the source content groups specified in the query, wherein each source content group has a metadata schema that describes structure of metadata associated with the unstructured information in the source content group;

extracting the unstructured information and metadata associated with the unstructured information from the set of the source content repositories;

in response to user input, receiving selection of target content groups in another set of target content repositories;

in response to receiving the selection of the target content groups, identifying metadata schemas of the target content groups, wherein each metadata schema describes a structure of metadata associated with the unstructured information in a target content group;

creating a schema definition file including the extracted metadata schemas of the source content groups and the identified metadata schemas of the target content groups;

forwarding the unstructured information, metadata, and schema definition file to a transformation service of the computer; and

using the transformation service,

performing one or more custom mappings on at least one of the unstructured information and the associated metadata by mapping elements of the extracted metadata schemas of the source content groups to the identified metadata schemas of the target content groups;

transforming at least one of the unstructured information and the associated metadata with custom transformations; and

forwarding the mapped and transformed unstructured information and the associated metadata to the federation service.

27. The system of claim 26, wherein the operations further comprise:

using the federation service;

receiving, from the transformation service, mapped and transformed unstructured information and the associated metadata; and

loading the mapped and transformed unstructured information and the associated metadata into the target content groups.