METHOD AND APPARATUS FOR AUTOMATICALLY STRUCTURING FREE FORM HETERGENEOUS DATA

• US 20080201279A1
• Filed: 02/15/2007
• Published: 08/21/2008
• Est. Priority Date: 02/15/2007
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Sailer, Anca, Wei, Xing, Kar, Gautam, Mahindru, Ruchi
• Granted Patent: US 8,996,587 B2
• Time in Patent Office: Days
• US Class Current: 706/12
• CPC Class Codes: G06F 40/205 Parsing
• CPC Class Codes: G06F 40/284 Lexical analysis, e.g. tokenisation or collocates
• CPC Class Codes: G06Q 30/02 Marketing; Price estimation or determination; Fundraising

Abstract

Techniques are provided for automatically structuring free form heterogeneous data. In one aspect of the invention, the techniques include obtaining free form heterogeneous data, segmenting the free form heterogeneous data into one or more units, automatically labeling the one or more units based on one or more machine learning techniques, wherein each unit is associated with a label indicating an information type, and structuring the one or more labeled units in a format to facilitate one or more operations that use at least a portion of the labeled units, e.g., information technology (IT) operations.

Claims

1. A method of automatically structuring free form heterogeneous data, comprising the steps of:

obtaining free form heterogeneous data;

segmenting the free form heterogeneous data into one or more units;

automatically labeling the one or more units based on one or more machine learning techniques, wherein each unit is associated with a label indicating an information type; and

structuring the one or more labeled units in a format to facilitate one or more operations that use at least a portion of the labeled units.

2. The method of claim 1, wherein the step of segmenting the free form heterogeneous data into one or more units comprises identifying punctuation in the free form data.

3. The method of claim 1, wherein the step of segmenting the free form heterogeneous data into one or more units comprises identifying one or more line breaks in the free form data.

4. The method of claim 1, wherein the step of segmenting the free form heterogeneous data into one or more units comprises identifying a part of speech in the free form data.

5. The method of claim 4, wherein the part of speech is a verb.

6. The method of claim 1, wherein the step of structuring the one or more labeled units in a format comprises structuring the one or more labeled units in a table.

7. The method of claim 1, wherein the step of structuring the one or more labeled units in a format comprises structuring the one or more labeled units in an extensible markup language (XML) format.

8. The method of claim 1, wherein the one or more information technology operations comprises processing at least one of existing free form data and newly obtained free form data.

9. A method of automatically structuring free form problem ticket data for facilitating technical assistance for one or more information technology (IT) operations, comprising the steps of:

obtaining free form problem ticket data;

segmenting the data;

storing the segmented data;

manually labeling a portion of the segmented data;

using the labeled data to generate an annotation model;

using the annotation model to automatically label a portion of unlabeled segmented data;

storing the automatically labeled data; and

structuring the stored data in a format, wherein the format facilitates technical assistance for one or more IT operations.

10. The method of claim 9, wherein the step of segmenting the data comprises identifying punctuation in the data.

11. The method of claim 9, wherein the step of segmenting the data comprises identifying one or more line breaks in the data.

12. The method of claim 9, wherein the step of segmenting the data comprises identifying a part of speech in the data.

13. The method of claim 12, wherein the part of speech is a verb.

14. The method of claim 9, wherein the step of manually labeling a portion of the segmented data comprises applying a label to the data, wherein the label comprises one of abstract, blank line, contact information (info), important step, no data, problem context problem description, problem type, root cause, severity level, and unimportant step.

15. The method of claim 9, wherein the step of using the labeled data to generate an annotation model comprises generating a semi-supervised learning process based on one or more machine learning techniques.

16. The method of claim 15, wherein the one or more machine learning techniques comprises a conditional random fields (CRFs) learning technique.

17. The method of claim 9, wherein the step of structuring the stored data in a format, wherein the format facilitates technical assistance for one or more IT operations comprises using the format to process existing free form problem ticket data offline.

18. The method of claim 9, wherein the step of structuring the stored data in a format, wherein the format facilitates technical assistance for one or more IT operations comprises using the format to process newly obtained free form problem ticket data online.

19. An apparatus for automatically structuring free form heterogeneous data, comprising:

a memory; and

at least one processor coupled to the memory and operative to;

obtain free form heterogeneous data;

segment the free form heterogeneous data into one or more units;

automatically label the one or more units based on one or more machine learning techniques, wherein each unit is associated with a label indicating an information type; and

structure the one or more labeled units in a format to facilitate one or more operations that use at least a portion of the labeled units.

20. The apparatus of claim 19, wherein the at least one processor is operative to segment the free form heterogeneous data into one or more units by identifying at least one of punctuation, one or more line breaks, and a part of speech in the free form data.

21. The apparatus of claim 19, wherein the at least one processor is operative to structure the one or more labeled units in a format to facilitate processing of at least one of existing free form data and newly obtained free form data.

22. A computer program product comprising a computer useable medium having computer useable program code for automatically structuring free form heterogeneous data, the computer program product including:

computer useable program code for obtaining free form heterogeneous data;

computer useable program code for segmenting the free form heterogeneous data into one or more units;

computer useable program code for automatically labeling the one or more units based on one or more machine learning techniques, wherein each unit is associated with a label indicating an information type; and

computer useable program code for structuring the one or more labeled units in a format to facilitate one or more operations that use at least a portion of the labeled units.

23. The computer program product of claim 22, wherein the computer usable program code for segmenting the free form heterogeneous data into one or more units comprises computer usable program code for identifying at least one of punctuation, one or more line breaks, and a part of speech in the free form data.

24. The computer program product of claim 22, wherein the computer usable program code for structuring the one or more labeled units in a format comprises computer usable program code for structuring the one or more labeled units in a format to facilitate processing of at least one of existing free form data and newly obtained free form data.

25. A computer program product comprising a computer useable medium having computer useable program code for automatically structuring free form problem ticket data for facilitating technical assistance for one or more information technology (IT) operations, the computer program product including:

computer useable program code for obtaining free form problem ticket data;

computer useable program code for segmenting the data;

computer useable program code for storing the data;

computer useable program code for manually labeling a portion of the segmented data;

computer useable program code for using the labeled data to generate an annotation model;

computer useable program code for using the annotation model to automatically label a portion of unlabeled segmented data;

computer useable program code for storing the automatically labeled data; and

computer useable program code for structuring the stored data in a format, wherein the format facilitates technical assistance for one or more IT operations.

26. The computer program product of claim 25, wherein the computer usable program code for segmenting the data comprises computer usable program code for identifying punctuation in the data.

27. The computer program product of claim 25, wherein the computer usable program code for segmenting the data comprises computer usable program code for identifying one or more line breaks in the data.

28. The computer program product of claim 25, wherein the computer usable program code for segmenting the data comprises computer usable program code for identifying a part of speech in the data.

29. The computer program product of claim 28, wherein the computer usable program code for identifying a part of speech in the data comprises computer usable program code for identifying a verb in the data.

30. The computer program product of claim 25, wherein the computer usable program code for manually labeling a portion of the segmented data comprises computer usable program code for applying a label to the data, wherein the label comprises one of abstract, blank line, contact information (info), important step, no data, problem context problem description, problem type, root cause, severity level, and unimportant step.

31. The computer program product of claim 25, wherein the computer usable program code for using the labeled data to generate an annotation model comprises computer usable program code for generating a semi-supervised learning process based on one or more machine learning techniques.

32. The computer program product of claim 31, wherein the computer usable program code for generating a semi-supervised learning process based on one or more machine learning techniques comprises computer usable program code for generating a semi-supervised learning process based on a conditional random fields (CRFs) learning technique.

33. The computer program product of claim 25, wherein the computer usable program code for structuring the stored data in a format, wherein the format facilitates technical assistance for one or more IT operations comprises computer usable program code for using the format to process existing free form problem ticket data offline.

34. The computer program product of claim 25, wherein the computer usable program code for structuring the stored data in a format, wherein the format facilitates technical assistance for one or more IT operations comprises computer usable program code for using the format to process newly obtained free form problem ticket data online.