Methods and apparatus for creation of parsing rules

• US 7,343,604 B2
• Filed: 07/25/2003
• Issued: 03/11/2008
• Est. Priority Date: 07/25/2003
• Status: Active Grant

• Current Assignee: Meta Platforms, Inc. (f/k/a Facebook, Inc.)
• Original Assignee: International Business Machines Corporation
• Inventors: Grabarnik, Genady, Thoenen, David H., Perng, Chang-shing, Salahshour, Abdolreza, Ma, Sheng
• Primary Examiner(s): Thomson; William
• Assistant Examiner(s): Seye; Abdou K
• Application Number:
  US10/627,824
• Publication Number:
  US 20050022207A1
• Time in Patent Office: 1,691 Days
• Field of Search: None
• US Class Current: 719/313
• CPC Class Codes: G06F 8/427 Parsing

Abstract

Techniques for parsing rule creation are provided. A technique for constructing one or more message parsing rules may comprise the following steps. First, message data representing past messages, for example, associated with a network, an application and/or a system being analyzed, is obtained. For example, this may involve reading the past or historical message data from messages logs or having a system point to the message data in existing data storage. Parsing rules are then generated by a process from one or more existing rule templates and/or based on user selection and classification of at least a portion of a message. For example, the user may choose a message part and demonstratively classify the part, for example, as a positive or negative example. The generated rules may then be stored for access by a rule-based parsing system such as a message adaptation system. Prior to generation of the one or more parsing rules, a message structure may be established upon which generation of the rules may be based.

Claims

1. A method of constructing one or more message parsing rules in accordance with a user and a machine, comprising the steps of:

obtaining message data representing past messages, wherein the past messages contain management information for a network, an application, and a system being analyzed;

establishing an appropriate rule template for a message structure by scanning the message data, building a message skeleton, comparing previously generated rule templates to the message skeleton, providing potential rule template matches to the user for validation and choice of a selected rule template, determining whether the selected rule template contains enough information to identify individual messages, building a rule template through an iterative process between the user and the machine based on user selection of at least a portion of the message and designating the built rule template as a template to be used in the construction of one or more message parsing rules, when the selected rule template does not contain enough information, and designating the selected rule template as a template to be used in the construction of one or more message parsing rules; and

generating one or more message parsing rules by a process based on the obtained message data and the template to be used in the construction of one or more message parsing rules, and defining positive and negative examples in an unparseable message by the user, learning and creating possible rules covering positive examples at the machine, and adding a newly created rule to the one or more message parsing rules, wherein the one or more parsing rules are storable for access by a rule-based parsing system.

2. The method of claim 1, wherein the message data is obtained by at least one of:

(i) reading past message data from one or more messages logs; and

(ii) one of the network, the application and the system pointing to message data in existing data storage.

3. The method of claim 1, wherein the rule-based parsing system comprises a message adaptation system.

4. The method of claim 1, further comprising the step of establishing a message structure prior to the generating step.

5. The method of claim 1, wherein the iterative process further comprises demonstrative classification of the selected portion as one of a positive example and a negative example.

6. The method of claim 1, wherein the message skeleton comprises information relating to one or more of a message start, a message end, and a separator between fields.

7. The method of claim 1, wherein classification comprises user demonstration of at least one of a positive example and a negative example.

8. The method of claim 7, wherein classification further comprises the steps of:

the machine parsing message data sequentially until an unparseable message is encountered;

the machine displaying the unparseable message to the user;

the user selecting at least a portion of the unparseable message and marking the selected portion as one of a positive example and a negative example; and

the machine learning based on the example and creating one or more candidate rules.

9. The method of claim 8, further comprising the step of the machine revising the one or more candidate rules based on feedback from the user.

10. The method of claim 1, wherein each of the one or more generated parsing rules comprises a regular expression of a portion of a message.

11. The method of claim 1, wherein each of the one or more generated parsing rules comprises a transformation rule of a portion of a message.

12. The method of claim 11, wherein the transformation rule comprises a swing constant.

13. The method of claim 11, wherein the transformation rule comprises a permutation of one or more input tokens.

14. Apparatus for constructing one or more message parsing rules, comprising:

a memory; and

at least one machine-based processor coupled to the memory and operative to;

(i) obtain message data representing past messages, wherein the past messages contain management information for a network, an application, and a system being analyzed;

(ii) establish an appropriate rule template for a message structure by scanning the message data, building a message skeleton, comparing previously generated rule templates to the message skeleton, providing potential rule template matches to the user for validation and choice of a selected rule template, determining whether the selected rule template contains enough information to identify individual messages, building a rule template through an iterative process between the user and the machine based on user selection of at least a portion of the message and designating the built rule template as a template to be used in the construction of one or more message parsing rules, when the selected rule template does not contain enough information, and designating the selected rule template as a template to be used in the construction of one or more message parsing rules; and

(iii) generate one or more message parsing rules by a process based on the obtained message data and the template to be used in the construction of one or more message parsing rules, and defining positive and negative examples in an unparseable message by the user, learning and creating possible rules covering positive examples at the machine, and adding a newly created rule to the one or more message parsing rules, wherein the one or more parsing rules are storable for access by a rule-based parsing system.

15. The apparatus of claim 14, wherein the rule-based parsing system comprises a message adaptation system.

16. The apparatus of claim 14, wherein classification comprises user demonstration of at least one of a positive example and a negative example.

17. The apparatus of claim 16, wherein classification further comprises operations of:

the at least one machine-based processor parsing message data sequentially until an unparseable message is encountered;

the at least one machine-based processor displaying the unparseable message to a user;

the user selecting at least a portion of the unparseable message and marking the selected portion as one of a positive example and a negative example; and

the at least one machine-based processor learning based on the example and creating one or more candidate rules.

18. An article of manufacture for constructing one or more message parsing rules in accordance with a user and a machine, comprising a machine readable storage medium containing one or more programs which when executed implement the steps of:

obtaining message data representing past messages, wherein the past messages contain management information for a network, an application, and a system being analyzed;

establishing an appropriate rule template for a message structure by scanning the message data, building a message skeleton, comparing previously generated rule templates to the message skeleton, providing potential rule template matches to the user for validation and choice of a selected rule template, determining whether the selected rule template contains enough information to identify individual messages, building a rule template through an iterative process between the user and the machine based on user selection of at least a portion of the message and designating the built rule template as a template to be used in the construction of one or more message parsing rules, when the selected rule template does not contain enough information, and designating the selected rule template as a template to be used in the construction of one or more message parsing rules; and

generating one or more message parsing rules by a process based on the obtained message data and the template to be used in the construction of one or more message parsing rules, and defining positive and negative examples in an unparseable message by the user, learning and creating possible rules covering positive examples at the machine, and adding a newly created rule to the one or more message parsing rules, wherein the one or more parsing rules are storable for access by a rule-based parsing system.