System and method for concept based analysis of unstructured data

• US 7,627,588 B1
• Filed: 10/03/2003
• Issued: 12/01/2009
• Est. Priority Date: 05/07/2001
• Status: Expired due to Term

• Current Assignee: CONQ Incorporated
• Original Assignee: IxReveal Incorporated
• Inventors: Sha, David D., Mohan, Usha, Mohan, Rengaswamy
• Primary Examiner(s): Pham; Khanh B
• Application Number:
  US10/677,492
• Time in Patent Office: 2,251 Days
• Field of Search: 707/5, 707/102, 704/9, 702/66, 703/22
• US Class Current: 1/1
• CPC Class Codes: G06F 16/3329 Natural language query formulation or dialogue systems
• CPC Class Codes: G06F 16/3331 Query processing
• CPC Class Codes: G06F 16/3343 using phonetics
• CPC Class Codes: Y10S 707/99933 Query processing, i.e. searching
• CPC Class Codes: Y10S 707/99934 Query formulation, input preparation, or translation
• CPC Class Codes: Y10S 707/99935 Query augmenting and refining, e.g. inexact access
• CPC Class Codes: Y10S 707/99936 Pattern matching access
• CPC Class Codes: Y10S 707/99942 Manipulating data structure, e.g. compression, compaction, compilation

Abstract

A computer implemented system and method automates analysis and mining of concepts from unstructured data. At least one concept within at least one object is analyzed. Analysis involves: selecting an object for inclusion in an initial set of objects to be analyzed; using an algorithm to extract a concept from the initial set of objects to create an initial set of concepts; refining the concept based upon relationships to other concepts; and performing multi-dimensional analysis on the concept in the initial set of objects to analyze the concept.

Claims

1. A computer-readable storage medium storing codes that when executed by a processor perform a method, comprising:

receiving from a user a selection of a first category and a second category from among a plurality of categories associated with a set of objects, the first category being associated with a first subset of objects from among a plurality of subsets of objects, and the second category being associated with a second subset of objects from among the plurality of subsets of objects, each of the first subset and the second subset being included in the set of objects and including at least one object;

receiving from a user a selection of a concept from among a plurality of concepts associated with the first category;

performing multi-dimensional analysis on the concept to determine a presence or absence of the concept in each of the first subset and the second subset;

for each of the first subset and second subset, if the concept is present in that subset, providing an indication of a strength of presence of the concept in that subset; and

if the concept is absent from the second subset, providing an indication of an absence of the concept from the second subset, the indication of the absence including a reference to the concept and the second category.

2. The computer-readable storage medium of claim 1, wherein the concept is a first concept, the indication of a strength of presence is a first indication of a strength of presence, and the indication of an absence is a first indication of an absence, further comprising codes that when executed by a processor perform the following steps:

for each of the first subset of objects and the second subset of objects, determining, based on the multi-dimensional analysis, if a second concept is present in that subset;

performing multi-dimensional analysis on the second concept to determine a presence or absence of the second concept in each of the first subset and the second subset;

for each of the first subset and the second subset, if the second concept is present in that subset, providing a second indication of a strength of presence of the second concept in that subset; and

if the second concept is absent from that subset, providing a second indication of an absence of the second concept from that subset, the second indication of the absence including a reference to the second concept and a category that is associated with that subset.

3. The computer-readable storage medium of claim 2, further comprising codes that when executed by a processor perform the step of:

creating a business rule to be used in a transformation of data into a datastore.

4. The computer-readable storage medium of claim 1, wherein the concept is a first concept, further comprising codes that when executed by a processor perform the steps of:

determining, based on the multi-dimensional analysis, that a second concept from among the plurality of concepts should be deleted from the plurality of concepts;

deleting the second concept from the plurality of concepts;

determining if there exists a second category to be included in the plurality of categories; and

if the second category exists, repeating the detecting and deleting.

5. The computer-readable storage medium of claim 1, wherein the set of objects is comprised of at least one of:

structured data;

unstructured data; and

semi-structured data.

6. The computer-readable storage medium of claim 1, wherein the user selection of the first category is based on one or more of:

a location of the first subset of objects on a computer disk drive, a LAN, or a WAN;

a location of the first subset of objects within a database;

a URL address of the first subset of objects;

a presence of a predefined concept, keyword, or text in the first subset of objects; and

an absence of a predefined concept, keyword, or text in the first subset of objects.

7. The computer-readable storage medium of claim 1, wherein the first subset of objects comprises an electronic file.

8. The computer-readable storage medium of claim 7, wherein the electronic file is selected from the group consisting of a Program File (*.exe), Text File (*.txt, *.prn, *.csv), Word Document (*.doc), Rich Text Format (*.rtf), Windows Write (*.wri), Word for Macintosh (*.mcw), MS-DOS Text with Layout (*.asc), Text with Layout (*.ans), E-mails (*.eml), Outlook Address Book (*.olk), Personal Address Book (*.pab), WordPerfect file (*.wpd), Schedule+Contact (*.scd), Powerpoint (*.ppt), Harvard Graphics Show (*.sh3), Harvard Graphics Chart (*.ch3), Freelance Windows file (*.pre), Excel File (*.xl*), Adobe Acrobat File (*.pdf), Web Page (*.htm*, *.asp, *.jsp), Query File (*.*qy), Lotus 1-2-3 File (*.wk*), Quattro Pro/Dos File (*.wq1), Microsoft Works File (*.wks), Works for Window (*.wps), Microsoft Access Files (*.mdb), Dbase Files (*.dbf), SYLK Files (*.slk), Data Interchange Format File (*.dif), Backup File (*.bak), Quattro Pro 1.0/5.0 (win) (*.wbl), Text Recovered from any File (*.*), Graphic Interchange Format (*.gif), Windows Bitmap (*.bmp), JPEG file interchange format (*.jpg), Tag image file format (*.tif), portable network graphics (*.png), Kodac Photo CD (*.pcd), PC Paintbrush (*.pcx), Raster file (*.ras), Audio File (*.wav, *.snd, *.aif, *.aifc, *.aiff, *.wma, *.mp3), CD Audio Track (*.cda), Media Playlist (*.asx, *.wax, *.m3u, *.wvx), MIDI File (*.mid, *.rmi, *.midi), Movie File (*.mpeg, *.mpg, *.m1v, *.mp2, *.mpa, *.mpe), Video File (*.avi, *.wmv), Windows Media File (*.asf, *.wm, *.wma, *.wmv), and Tactile Sensing File in ASCII, LabView, or MATLAB formats.

9. The computer-readable storage medium of claim 1, further comprising codes that when executed by a processor perform the steps of:

extracting the concept from the set of objects, the extracting including one or more of;

generating a common format of the set of objects using an application programming interface (API);

generating a common format of the set of objects using an extraction algorithm;

determining an image intensity of the set of objects;

determining at least one boundary within the set of objects;

mapping audio waveforms within the set of objects to a text format; and

converting non-textual information within the set of objects into text.

10. The computer-readable storage medium of claim 1, wherein the set of objects is a first set of objects, and the performing multi-dimensional analysis includes one or more of:

determining a number of objects within the first subset of objects;

determining a frequency of occurrence of the concept within the first set of objects;

determining a frequency of occurrence of the concept within the first subset of objects;

determining a frequency of occurrence of the concept within a second set of objects different from the first set of objects;

determining a normalized frequency of occurrence of the concept within the first set of objects;

determining a normalized frequency of occurrence of the concept within the first subset of objects;

determining a normalized frequency of occurrence of the concept within the second set of objects;

determining an electronic path to the location of the first subset of objects;

determining a characteristic of the first subset of objects;

determining a concept type for the concept;

determining a definition for the first concept; and

if the first subset of objects contains the concept, determining a position of the concept within the first subset.

11. The computer-readable storage medium of claim 10, wherein the characteristic is one of:

a file extension

a creation date; and

an effective date and a status.

12. The computer-readable storage medium of claim 10, wherein the concept type is a P parent type.

13. The computer-readable storage medium of claim 10, wherein the performing multi-dimensional analysis includes slicing-and-dicing across at least one dimension of the set of objects.

14. The computer-readable storage medium of claim 13, wherein the at least one dimension is one of:

a time dimension, a geographical location dimension, an electronic location dimension, a person dimension, a multiple-person dimension, a business unit dimension, an organization dimension, a process dimension, a product dimension, a service dimension, a subject dimension, a category dimension, a concept dimension, a concept type dimension, a user viewpoint dimension, and an entity dimension in a structured database.

15. The computer-readable storage medium of claim 1, wherein the performing multi-dimensional analysis further comprises codes that when executed by a processor perform at least one of the following:

generating a report based on the analysis; and

displaying results of the analysis in one or more of;

a graphical format; and

a visual format.

16. The computer-readable storage medium of claim 1, further comprising codes that when executed by a processor perform the following step:

using the multi-dimensional analysis to discover a result, the result being one of;

a trend of concepts within the set of objects;

a pattern of concepts within the set of objects;

a concept exception within the set of objects; and

at least one relationship between concepts within the set of objects.

17. The computer-readable storage medium of claim 1, wherein the method further comprises codes that when executed by a processor perform at least one of:

storing the concept in a concept repository; and

sharing the concept with one or more other users.

18. The computer-readable storage medium of claim 1, wherein the using a processor to perform multi-dimensional analysis includes creating a hierarchy of the plurality of concepts based on user input.

19. The computer-readable storage medium of claim 1, wherein the concept is a first concept, further comprising codes that when executed by a processor perform the step of:

refining the concept based on a relationship to a second concept before performing the multi-dimensional analysis.

20. The computer-readable storage medium of claim 19, wherein the second concept is not within the set of objects.

21. The computer-readable storage medium of claim 19, wherein the concept is a first concept, the set of objects is a first set of objects, and the refining the first concept includes one or more of:

determining a relationship of the first concept to a second concept within an existing reference;

determining a relationship of the first concept to a second concept using a natural language processing (NLP) algorithm;

determining a frequency of occurrence of the first concept within the first set of objects;

determining a frequency of occurrence of the first concept within a second set of objects different from the first;

determining a normalized frequency of occurrence of the first concept within the first set of objects; and

determining a normalized frequency of occurrence of the first concept within the second set of objects.

22. The computer-readable storage medium of claim 1, further comprising codes that when executed by a processor perform the steps of:

receiving from a user a selection of a first subcategory and a second subcategory from among a plurality of subcategories associated with the first subset of objects, the first subcategory being associated with a first sub-subset of the first subset and the second subcategory being associated with a second sub-subset of the first subset, each of the first and second sub-subset including one or more objects;

performing multi-dimensional analysis on the concept to determine a presence or absence of the concept in each of the first sub-subset and the second sub-subset;

for each of the first sub-subset and the second sub-subset, if the concept is present in that sub-subset, providing an indication of a strength of presence of the concept in that sub-subset; and

if the concept is absent from that sub-subset, providing an indication of an absence of the concept from that sub-subset, the indication of the absence including a reference to the concept and the subcategory that is associated with that subset.

23. The computer-readable storage medium of claim 1, further comprising codes that when executed by a processor perform the steps of:

receiving from a user a selection of a subconcept from a plurality of subconcepts associated with the concept;

performing multi-dimensional analysis on the subconcept to determine a presence or absence of the subconcept in each of the first subset and second subset;

for each of the first subset and the second subset, if the subconcept is present that subset, providing an indication of a strength of presence of the subconcept in that subset; and

if the subconcept is not present in the second subset, providing an indication of an absence of the subconcept from the second subset, the indication of the absence including a reference to the subconcept and the second category.

24. The computer-readable storage medium of claim 1, further comprising codes that when executed by a processor perform the steps of:

receiving from a user a selection of a subconcept from a plurality of subconcepts associated with the concept;

receiving from a user a selection of a first subcategory and a second subcategory from among a plurality of subcategories associated with the first subset of objects, the first subcategory being associated with a first sub-subset of the first subset and the second subcategory being associated with a second sub-subset of the first subset, each of the first and second sub-subset including one or more objects;

performing multi-dimensional analysis on the subconcept to determine a presence or absence of the subconcept in each of the first sub-subset and the second sub-subset;

for each of the first sub-subset and the second sub-subset, if the subconcept is present in that sub-subset, providing an indication of a strength of presence of the subconcept in that sub-subset; and

if the subconcept is absent from that sub-subset, providing an indication of an absence of the subconcept from that sub-subset, the indication of the absence including a reference to the subconcept and the subcategory associated with that subset.

25. A method, comprising:

receiving one or more signals that define a plurality of categories associated with a set of data objects;

receiving a selection of a category from among the plurality of categories, the category being associated with a subset of objects, the subset being included in the set of objects and including at least one object;

receiving a selection of a concept from among a plurality of concepts associated with the category;

using a processor to perform multi-dimensional analysis on the concept, the multi-dimensional analysis including determining a strength of presence of the concept in the subset;

if the concept is absent from the subset, providing an indication of an absence of the concept from the subset; and

outputting to a display device a graphical representation of the strength of presence and the category.

26. The method of claim 25, wherein the using a processor to perform multi-dimensional analysis includes one or more of:

determining a frequency of occurrence of the concept within the set of objects;

determining a normalized frequency of occurrence of the concept within the set of objects;

determining a normalized frequency of occurrence of the concept within the subset of objects;

determining an electronic path to the location of the subset of objects;

determining a characteristic of the subset of objects; and

determining a concept type for the concept.

27. The method of claim 25, wherein the first subset is selected by a user, the selection being based on one or more of:

a location of the subset on a computer disk drive, a LAN, or a WAN;

a location of the subset within a database;

a URL address of the subset;

a presence of a predefined concept, keyword, or text in the subset; and

an absence of a predefined concept, keyword, or text in the subset.

28. The method of claim 25, wherein the using a processor to perform multi-dimensional analysis includes slicing-and-dicing across at least one dimension of the set of data objects.

29. The method of claim 25, wherein the concept is detected in the set of data objects, further comprising:

generating a common format of the set of data objects using an application programming interface (API);

generating a common format of the set of data objects using an extraction algorithm;

determining an image intensity of the set of data objects;

determining at least one boundary within the set of data objects;

mapping audio waveforms within the set of data objects to a text format; and

converting non-textual information within the set of data objects into text.

30. The method of claim 25, wherein the using a processor to perform multi-dimensional analysis includes creating a hierarchy of a plurality of concepts based on user input, the plurality of concepts including the concept.

31. The method of claim 25, wherein the concept is a first concept, further comprising:

refining the first concept based on a relationship to a second concept before using a processor to perform the multi-dimensional analysis.

32. The method of claim 31, wherein the set of data objects is a first set of data objects, and the second concept is present in a second set of data objects different from the first set of data objects.

33. The method of claim 31, wherein the concept is a first concept, the set of objects is a first set of data objects, and the refining the first concept includes one or more of:

determining a relationship of the first concept to a second concept within an existing reference;

determining a relationship of the first concept to a second concept using a natural language processing (NLP) algorithm;

determining a frequency of occurrence of the first concept within the first set of data objects;

determining a frequency of occurrence of the first concept within a second set of data objects different from the first;

determining a normalized frequency of occurrence of the first concept within the first set of data objects; and

determining a normalized frequency of occurrence of the first concept within the second set of data objects.

34. The method of claim 25, wherein the step of using a processor to perform multi-dimensional analysis further comprises at least one of the following steps:

generating reports based on the analysis; and

displaying results of the analysis in a visual format.