System and method for processing electronic data from multiple data sources

• US 20040177062A1
• Filed: 03/03/2003
• Published: 09/09/2004
• Est. Priority Date: 03/03/2003
• Status: Active Grant

• Current Assignee: Callidus Software, Inc. (SAP SE)
• Original Assignee: Raytheon Company (Rtx Corporation)
• Inventors: Ernst, Richard J., Carbone, John N., Lewis, Jeffrey D., Urquhart, Andrew L., Scoggins, Rick D., Poindexter, Carl A., Campbell, Susannah R.
• Granted Patent: US 7,328,219 B2
• Time in Patent Office: Days
• US Class Current: 707/3
• CPC Class Codes: G06F 16/258 Data format conversion from or to a database
• CPC Class Codes: Y10S 707/99936 Pattern matching access
• CPC Class Codes: Y10S 707/99942 Manipulating data structure, e.g. compression, compaction, compilation
• CPC Class Codes: Y10S 707/99943 Generating database or data structure, e.g. via user interface

Abstract

A method for processing data streams from multiple data sources comprises receiving a data stream from a data source. A data template is dynamically selected, the data template comprising at least one data template region. A data region of the data stream is selected based on one of the data template regions. At least a portion of the data region is mapped to an intermediate data store based on the data template region. The mapped portion of the data region is extracted and stored in the intermediate data store.

Claims

1. A method for processing data streams from multiple data sources comprising:

receiving a data stream from a data source;

dynamically selecting a data template, the data template comprising at least one data template region;

selecting a data region of the data stream based on one of the data template regions;

mapping at least a portion of the data region to an intermediate data store based on the data template region;

extracting the mapped portion of the data region; and

storing the extracted portion in the intermediate data store.

2. The method of claim 1 further comprising:

converting the extracted portion of the data region to an intermediate format; and

wherein storing the extracted portion in the intermediate data store comprises storing the converted extracted portion in the intermediate data store.

3. The method of claim 1, wherein dynamically selecting a data template comprises dynamically selecting the data template based on the data source.

4. The method of claim 1 further comprising:

receiving the data template from the data source; and

wherein dynamically selecting a data template comprises selecting the received data template.

5. The method of claim 1, wherein the data stream comprises a first data stream, the data source comprises a first data source, and the data template comprises a first data template, the method further comprising:

receiving a second data stream from a second data source, wherein the second data source differs from the first data source;

dynamically selecting a second data template;

selecting a data region of the second data stream based on one of the second data template regions;

mapping at least a portion of the data region of the second data stream to the intermediate data store based on the second data template region;

extracting the mapped portion of the data region of the second data stream; and

storing the extracted portion of the second data stream in the intermediate data store.

6. The method of claim 1, wherein the data region comprises a first data region and the data template region comprises a first data template region, the method further comprising:

determining an end of the first data region based on the first data template region;

selecting a second data region of the data stream based on a second data template region;

mapping at least a portion of the second data region to the intermediate data store based on the data template region;

extracting the mapped portion of the second data region; and

storing the second extracted portion of the data stream in the intermediate data store.

7. The method of claim 1, wherein selecting a data region of the data stream based on one of the data template regions comprises:

selecting a regular expression from one of the data template regions;

scanning the data stream for the regular expression;

selecting a second regular expression from the data template region;

scanning the data stream for the second regular expression; and

selecting data in the data stream delimited by the first and second regular expressions.

8. The method of claim 1, wherein mapping at least a portion of the data region to an intermediate data store comprises:

selecting at least one sub-expression from the data template region, each sub-expression having a unique identifier; and

scanning the data region of the data stream for each sub-expression.

9. The method of claim 8, wherein the data region comprises a first data region and, in response to not locating one sub-expression in the data region of the data stream, the method further comprising:

selecting an alternate data region of the data stream based on the data template;

scanning the alternate data region of the data stream for the sub-expression;

mapping at least a portion of the alternate data region of the data stream to the intermediate data store based on the data template region; and

extracting the mapped portion of the alternate data region of the data stream.

10. The method of claim 8, wherein in response to not locating one sub-expression in the data region of the data stream, the method further comprising:

selecting an alternate expression from the data template region, the alternate sub-expression having a unique identifier; and

scanning the data region of the data stream for the alternate expression.

11. The method of claim 8, wherein extracting the mapped portion of the data region comprises creating a datum in response to locating data in the data region that matches one sub-expression, the datum comprising the respective identifier of the sub-expression and the located data.

12. The method of claim 1 further comprising:

scanning the received data stream for a template identifier; and

wherein dynamically selecting a data template comprises selecting the data template based on the template identifier.

13. The method of claim 1, wherein dynamically selecting a data template comprises:

selecting a first data template; and

in response to the first data template being incompatible with the received data stream, selecting a second data template.

14. Software for processing data from multiple data sources operable to:

receive a data stream from a data source;

dynamically select a data template, the data template comprising at least one data template region;

select a data region of the data stream based on one of the data template regions;

map at least a portion of the data region to an intermediate data store based on the data template region;

extract the mapped portion of the data region; and

store the extracted portion in the intermediate data store.

15. The software of claim 14 further operable to:

convert the extracted portion of the data region to an intermediate format; and

wherein the software operable to store the extracted portion in the intermediate data store comprises the software operable to store the converted extracted portion in the intermediate data store.

16. The software of claim 14, wherein the software operable to dynamically select a data template comprises the software operable to dynamically select the data template based on the data source.

17. The software of claim 14 further operable to:

receive the data template from the data source; and

wherein the software operable to dynamically select a data template comprises the software operable to select the received data template.

18. The software of claim 14, wherein the data stream comprises a first data stream, the data source comprises a first data source, and the data template comprises a first data template, the software further operable to:

receive a second data stream from a second data source, wherein the second data source differs from the first data source;

dynamically select a second data template;

select a data region of the second data stream based on one of the second data template regions;

map at least a portion of the data region of the second data stream to the intermediate data store based on the second data template region;

extract the mapped portion of the data region of the second data stream; and

store the extracted portion of the second data stream in the intermediate data store.

19. The software of claim 14, wherein the data region comprises a first data region and the software further operable to:

determine an end of the first data region based on the data template region;

select a second data region of the data stream based on a second data template region;

map at least a portion of the second data region to the intermediate data store based on the data template region;

extract the mapped portion of the second data region; and

store the second extracted portion of the data stream in the intermediate data store.

20. The software of claim 14, wherein the software operable to select a data region of the data stream based on one of the data template regions comprises the software operable to:

select a regular expression from one of the data template regions;

scan the data stream for the regular expression;

select a second regular expression from the data template region;

scan the data stream for the second regular expression; and

select data in the data stream delimited by the first and second regular expressions.

21. The software of claim 14, wherein the software operable to map at least a portion of the data region to an intermediate data store comprises the software operable to:

select at least one sub-expression from the data template region, each sub-expression having a unique identifier; and

scan the data region of the data stream for each sub-expression.

22. The software of claim 21, wherein, in response to the software not locating one sub-expression in the data region of the data stream, the software further operable to:

select an alternate sub-expression from the data template region, the alternate sub-expression having a unique identifier; and

scan the data region of the data stream for the alternate sub-expression.

23. The software of claim 21, wherein the data region comprises a first data region and, in response to the software not locating one sub-expression in the data region of the data stream, the software further operable to:

select an alternate data region of the data stream based on the data template;

scan the alternate data region of the data stream for the sub-expression;

map at least a portion of the alternate data region of the data stream to the intermediate data store based on the data template region; and

extract the mapped portion of the alternate data region of the data stream.

24. The software of claim 21, wherein the software operable to extract the mapped portion of the data region comprises the software operable to create a datum in response to locating data in the data region that matches one sub-expression, the datum comprising the respective identifier of the sub-expression and the located data.

25. The software of claim 14 further operable to:

scan the received data stream for a template identifier; and

wherein the software operable to dynamically select a data template comprises the software operable to select the data template based on the template identifier.

26. The software of claim 14, wherein the software operable to dynamically select a data template comprises the software operable to:

select a first data template; and

in response to the first data template being incompatible with the received data stream, select a second data template.