METHOD FOR AUTOMATIC PARSING OF VARIABLE DATA FIELDS FROM TEXTUAL REPORT DATA

• US 20080005265A1
• Filed: 06/30/2006
• Published: 01/03/2008
• Est. Priority Date: 06/30/2006
• Status: Abandoned Application

• Current Assignee: Nokia Corporation
• Original Assignee: Nokia Corporation
• Inventors: Miettinen, Markus, Hatonen, Kimmo
• Application Number:
  US11/427,926
• Publication Number:
  US 20080005265A1
• Time in Patent Office: Days
• US Class Current: 709/217
• CPC Class Codes: G06F 40/186 Templates
• CPC Class Codes: G06F 40/205 Parsing

Abstract

A method and system for parsing textual report data found in free-text fields is disclosed. The textual report data may be included in log files that document a systems operation. A message template is created from reports or log data and used to automate the parsing of these variable data fields.

Claims

1. A method of parsing free-text data fields, the method comprising:

(a) detecting free-text message data located in the free-text data fields;

(b) separating the detected free-text message data into textual tokens;

(c) searching the free-text message data based on the textual tokens;

(d) detecting frequent patterns within the free-text message data;

(e) filtering the detected frequent patterns for arrangements of patterns;

(i) generating the message templates based on the arrangements of patterns; and

(g) parsing free-text message data based on the generated message templates.

2. The method of claim 1, wherein filtering the detected frequent patterns for arrangements in (e) further includes examining each detected frequent pattern, the examination including:

(i) analyzing each item of a detected frequent pattern;

(ii) determining the position of each item in the detected frequent pattern;

(iii) comparing the position of each item in the detected frequent pattern; and

(iv) determining if the items within the detected frequent pattern are consecutive and whether there are gaps of at most n positions between the items.

3. The method of claim 1, wherein the frequent patterns comprise closed sets.

4. The method of claim 1, wherein the frequent patterns comprise free sets.

5. The method of claim 1, wherein the frequent patterns comprise closed episodes.

6. The method of claim 1, wherein the frequent patterns comprise frequent episodes.

7. The method of claim 1, further comprising (h) displaying the detected frequent patterns.

8. The method of claim 1, wherein the textual tokens include words and punctuation.

9. The method of claim 8, wherein the words include a sequence of characters.

10. The method of claim 9, wherein the sequence of characters are contiguous.

11. The method of claim 1, wherein the detecting of frequent patterns in (d) comprises executing a data mining algorithm.

12. The method of claim 11, wherein the data mining algorithm comprises a frequent set mining algorithm.

13. The method of claim 11, wherein the data mining algorithm comprise a frequent episode mining algorithm.

14. A method of generating a message template for parsing free-text data fields, the method comprising:

(a) detecting free-text message data;

(b) detecting frequent patterns within the free-text message data;

(c) filtering the detected frequent patterns for arrangements of patterns; and

(d) creating the message template based on the arrangements of patterns.

15. The method of claim 14, wherein filtering the detected frequent patterns for arrangements in (c) further includes examining each detected frequent pattern, the examination including:

(i) analyzing each item of a detected frequent pattern;

(ii) determining the position of each item in the detected frequent pattern;

(iii) comparing the position of each item in the detected frequent pattern; and

(iv) determining if the items within the detected frequent pattern are consecutive and whether there are gaps of at most n positions between the items.

16. The method of claim 14, wherein the frequent patterns comprise closed sets.

17. The method of claim 14, wherein the frequent patterns comprise free sets.

18. The method of claim 14, wherein the frequent patterns comprise frequent episodes.

19. The method of claim 14, wherein the frequent patterns comprise closed episodes.

20. The method of claim 14, further comprising (e) displaying the detected frequent patterns.

21. The method of claim 14, wherein the detecting of frequent patterns in (b) comprises executing a frequency detection algorithm.

22. The method of claim 14, wherein the detecting of frequent patterns in (b) comprises executing a data mining algorithm.

23. The method of claim 22, wherein the data mining algorithm comprises the Apriori algorithm.

24. The method of claim 22, wherein the data mining algorithm comprises a frequent set mining algorithm.

25. A system for parsing free-text data fields, the system comprising:

(a) a storage medium;

(b) at least one processor coupled to the storage medium and programmed with computer-executable instruction for performing;

(i) detecting free-text message data located in the free-text data fields;

(ii) separating the detected free-text message data into textual tokens;

(iii) searching the free-text message data based on the textual tokens;

(iv) detecting frequent patterns within the free-text message data;

(v) filtering the detected frequent patterns for arrangements of patterns;

(vi) generating the message templates based on the arrangements of patterns; and

(vii) parsing free-text message data based on the generated message templates.

26. The system of claim 25, wherein filtering the detected frequent patterns for arrangements in (v) further includes examining each detected frequent pattern, the examination including:

(I) analyzing each item of a detected frequent pattern;

(II) determining the position of each item in the detected frequent pattern;

(III) comparing the position of each item in the detected frequent pattern; and

(IV) determining if the items within the detected frequent pattern are consecutive and whether there are gaps of at most n positions between the items.

27. A computer-readable medium having computer-executable instructions for performing steps comprising:

(a) detecting free-text message data;

(b) detecting frequent patterns within the free-text message data;

(c) filtering the detected frequent patterns for arrangements of patterns; and

(d) creating the message template based on the arrangements of patterns.

28. The computer-readable medium of claim 27, wherein filtering the detected frequent patterns for arrangements in (c) further includes examining each detected frequent pattern, the examination including:

(i) analyzing each item of a detected frequent pattern;

(ii) determining the position of each item in the detected frequent pattern;

(iii) comparing the position of each item in the detected frequent pattern; and

(iv) determining if the items within the detected frequent pattern are consecutive and whether there are gaps of at most n positions between the items.

29. The computer-readable medium of claim 27, wherein the frequent patterns comprise closed sets.

30. The computer-readable medium of claim 27, wherein the frequent patterns comprise free sets.

31. The computer-readable medium of claim 27, wherein the frequent patterns comprise closed episodes.

32. The computer-readable medium of claim 27, wherein the frequent patterns comprise frequent episodes.

33. An apparatus comprising:

a communication interface;

a storage medium; and

a processor coupled to the storage medium and programmed with computer-executable instructions to perform the steps comprising;

(a) detecting free-text message data located in the free-text data fields;

(b) separating the detected free-text message data into textual tokens;

(c) searching the free-text message data based on the textual tokens;

(d) detecting frequent patterns within the free-text message data;

(e) filtering the detected frequent patterns for arrangements of patterns;

(f) generating the message templates based on the arrangements of patterns; and

(g) parsing free-text message data based on the generated message templates.

34. The apparatus of claim 33, wherein filtering the detected frequent patterns for arrangements in (e) further includes examining each detected frequent pattern, the examination including:

(i) analyzing each item of a detected frequent pattern;

(ii) determining the position of each item in the detected frequent pattern;

(iii) comparing the position of each item in the detected frequent pattern; and

(iv) determining if the items within the detected frequent pattern are consecutive and whether there are gaps of at most n positions between the items.

35. An apparatus comprising:

(a) means for detecting free-text message data located in the free-text data fields;

(b) means for separating the detected free-text message data into textual tokens;

(c) means for searching the free-text message data based on the textual tokens;

(d) means for detecting frequent patterns within the free-text message data;

(e) means for filtering the detected frequent patterns for arrangements of patterns;

(f) means for generating the message templates based on the arrangements of patterns; and

(g) means for parsing free-text message data based on the generated message templates.

36. The apparatus of claim 35, wherein the means for detecting frequent patterns in (d) further comprises means for executing a data mining algorithm.