Methods and systems for high throughput information refinement

• US 8,090,873 B1
• Filed: 03/14/2005
• Issued: 01/03/2012
• Est. Priority Date: 03/14/2005
• Status: Active Grant

• Current Assignee: Oracle America, Inc. (Oracle Corporation)
• Original Assignee: Oracle America, Inc. (Oracle Corporation)
• Inventors: Wookey, Michael J.
• Primary Examiner(s): Divecha, Kamal B
• Application Number:
  US11/080,839
• Time in Patent Office: 2,486 Days
• Field of Search: 709/203, 709/246, 707600-602
• US Class Current: 709/246
• CPC Class Codes: H04L 51/214 using selective forwarding
• CPC Class Codes: H04L 67/2895 Intermediate processing functionally located close to the data provider application, e.g. reverse proxies
• CPC Class Codes: H04L 67/565 Conversion or adaptation of application format or content adding application control or application functional data H04L67/561
• CPC Class Codes: H04L 69/22 Parsing or analysis of headers

Abstract

Methods, systems, and articles of manufacture consistent with the present invention provide a data processing system comprising a business application that receives data messages from a plurality of client data sources. The business application comprises a message pre-processor and a parsing processor. The message pre-processor classifies and identifies the data messages and sends the messages in a structured format to a message queue corresponding to its data type. The parsing processor receives the data messages from the message queues and selects a parser by applying a set of parsing rules. The parsing rules apply information about the data message and provide a decision as to the best parsing engine to use out of a plurality of paring engines. The parsing engines are also able to perform information refinement in accordance with selected components defined in a target output data model.

Claims

1. A method for processing messages, the method comprising the steps of:

a computer system;

receiving, from a plurality of different data sources in a data network, a plurality of data messages each having a data type associated therewith and comprising a payload, wherein the plurality of data messages comprises different data formats;

for each of the data messages, determining a classification of the data message by parsing out information identifying the data type;

re-formatting each of the data messages, each of the re-formatted messages having a uniform data structure comprising an identifier, the classification, and the payload;

selecting a message service queue for each of the reformatted messages from a plurality of message service queues each dedicated to storing messages of a particular data type according to the classifications of the reformatted messages such that each of the selected service queues store a subset of the reformatted messages of a single data type;

with a parsing processor, monitoring the plurality of message service queues; and

with a parsing processor, selecting one of the message service queues based on the monitoring and then retrieving and parsing a next one of the reformatted messages from the selected one of the message service queues in accordance with a target output data model,

wherein the parsing processor comprises a plurality of parsers each operable to parse data messages having a different data type and wherein the parsing processor parses the next one of the reformatted messages using one of the plurality of parsers that is configured for parsing the data type associated with the selected service queue and is dynamically selected and allocated during the parsing step,

wherein the parsing processor, during the parsing step, selects one of the parsers to use for parsing the next one of the reformatted messages based on the data type and at least one parsing rule applied to one or more characteristics of the next one of the reformatted messages and wherein the plurality of parsers includes at least two parsers adapted for parsing the data type associated with the selected service queue, and

wherein the parsing includes extracting a subset of information in the payload of the next one of the reformatted messages defined in the target output data model, whereby throughput of the parsing processor is enhanced by extracting only select information from each of the data messages.

2. The method of claim 1 further comprising the steps of:

the computer system;

inserting a unique key in the data message;

searching for the unique key in a history of processed keys; and

if the unique key is not found in the history of processed keys, adding the unique key to the history of processed keys.

3. The method of claim 1 wherein the step of receiving the data message comprises the steps of:

the computer system;

receiving an email message from a message server; and

processing the email message by extracting the data type and the payload.

10. The method of claim 1, wherein the data message is an email message and the data type is one of the following data types:

a configuration file, a patch list, a storage identification file, a performance record, a maintenance record, a bug list, an installed driver list, an installed software list, and a license information record.

14. The method of claim 1, wherein the at least one parsing rule defines messages to process with a grid parser and wherein the plurality of parsers includes two or more grid parsers.

15. The method of claim 1, wherein the next one of the messages is a configuration file and the subset of information includes at least one of processor speed, hard disk storage capacity, random access memory capacity, and bus speed.

4. A non-transitory computer-readable storage device having a program code embodied therein to perform a method for processing messages, the method comprising the steps of:

a computer system;

receiving, from a plurality of different data sources in a data network, a plurality of data messages each having a data type associated therewith and comprising a payload, wherein the plurality of data messages comprises different data formats;

for each of the data messages, determining a classification of the data message by parsing out information identifying the data type;

re-formatting each of the data messages, each of the re-formatted messages having a uniform data structure comprising an identifier, the classification, and the payload;

selecting a message service queue for each of the re-formatted messages from a plurality of message service queues each dedicated to storing messages of a particular type according to the classifications of the reformatted messages such that each of the selected service queues store a subset of the reformatted messages of a single type;

with a parsing processor, monitoring the plurality of message service queues; and

with a parsing processor, selecting a next one of the message service queues to service based on the monitoring, receiving a next one of the reformatted messages from the selected one of the message service queues, and parsing the next one of the reformatted messages in accordance with a target output data model using one of a plurality of parsers selected by the parsing processor, after the receiving, using characteristics of the data message,

wherein the parsing processor comprises a plurality of parsers each operable to parse data messages having a different data type and wherein the parsing processor parses the next one of the reformatted messages using one of the plurality of parsers that is configured for parsing the data type associated with the selected service queue and is dynamically selected and allocated during the parsing step,

wherein the parsing processor selects one of the parsers to use for parsing the next one of the reformatted messages based on the data type and at least one parsing rule applied to one or more characteristics of the next one of the reformatted messages and wherein the plurality of parsers includes at least two parsers adapted for parsing the data type associated with the selected service queue, and

wherein the parsing processor extracts a subset of information in the payload of the next one of the reformatted messages defined in the target output data model, whereby throughput of the parsing processor is enhanced by extracting only select information from each of the data messages.

5. The non-transitory computer-readable storage device in claim 4 wherein the method further comprises the steps of:

the computer system;

inserting a unique key in the data message;

searching for the unique key in a history of processed keys; and

if the unique key is not found in the history of processed keys, adding the unique key to the history of processed keys.

6. The non-transitory computer-readable storage device in claim 5 wherein the step of receiving the data message in the method comprises the steps of:

receiving an email message from a message server; and

processing the email message by extracting the data type and the payload.

11. The non-transitory computer-readable storage device of claim 4, wherein each of the message service queues process reformatted messages based on a data type associated with the reformatted messages, whereby each of the message service queues stores a differing data type.

12. The non-transitory computer-readable storage device of claim 4, wherein each of the parsers is operable to parse a particular data type and wherein the characteristics of the data message used to select the one of the parsers include the identified data type of the received data message and at least one characteristic of the payload identified in a set of parsing rules allowing the parsing processor to heuristically learn about the data messages, whereby the selected one of the parsers is operable to parse the identified data type.

7. A computer system comprising:

a central processing unit and a memory;

a receiving unit for receiving, from a plurality of different data sources in a data network a plurality of data messages each having a data type associated therewith and comprising a payload, wherein the plurality of data messages comprises different data formats;

an identification unit for determining a classification of each of the data messages by parsing out information identifying the data type;

a formatting unit for re-formatting each of the data messages, wherein each of the re-formatted massages comprises an identifier, the classification, and the payload;

a selecting unit for selecting, based on the data type, a message service queue for each of the reformatted messages from a plurality of message service queues from the memory each adapted to process messages having differing data types such that each of the selected service queues store a subset of the reformatted messages of a single data type;

monitoring the plurality of message service queues; and

a parsing processor executed by the central processing unit that retrieves the reformatted message and parses the retrieved message using one of a plurality of parsers operable to parse data messages according to data types, wherein the parsing processor selects the one of the parsers based on the monitoring and based on the classification of the retrieved message and another characteristic of the retrieved message determined from the message payload,

wherein the parsing processor comprises a plurality of parsers each operable to parse data messages having a different data type and wherein the parsing processor parses the next one of the reformatted messages using one of the plurality of parsers that is configured for parsing the data type associated with the selected service queue and is dynamically selected and allocated during the parsing step,

wherein the parsing processor selects one of the parsers to use for parsing the next one of the reformatted messages based on the data type and at least one parsing rule applied to one or more characteristics of the next one of the reformatted messages and wherein the plurality of parsers includes at least two parsers adapted for parsing the data type associated with the selected service queue, and

wherein the parsing processor extracts a subset of information in the payload of the next one of the reformatted messages defined in the target output data model, whereby throughput of the parsing processor is enhanced by extracting only select information from each of the data messages.

8. The computer system of claim 7 further comprising:

a first key unit for inserting a unique key in the data message;

a second key unit for searching for the unique key in a history of processed keys; and

a third key unit for adding the unique key to the history of processed keys, if the unique key is not found in the history of processed keys.

9. The computer system of claim 7 wherein the receiving unit for receiving the data message comprises:

a first email unit for receiving an email message from a message server; and

a second email unit for processing the email message by extracting the data type and the payload.

13. The computer system of claim 7, wherein the data message is an email message and the data type is one of the following data types:

a configuration file, a patch list, a storage identification file, a performance record, a maintenance record, a bug list, an installed driver list, an installed software list, and a license information record.

16. The computer system of claim 7, wherein the parsing processor further functions, during the retrieving, to dynamically select a next one of the message service queues from which to retrieve a message for parsing.

17. The computer system of claim 7, wherein the another characteristic is a programming language format for the message payload and the selected one of the parsers is a parser configured for parsing files with the programming language format.

18. The computer system of claim 7, wherein the another characteristic defines the message payload as unformatted and, when the payload is determined to be unformatted, the parsing processor chooses the selected one of the parsers based on a volume of data in the message payload.