Concept-based method and system for dynamically analyzing unstructured information

• US 6,970,881 B1
• Filed: 03/01/2002
• Issued: 11/29/2005
• Est. Priority Date: 05/07/2001
• Status: Active Grant

• Current Assignee: Nathan T. Conk
• Original Assignee: Intelligenxia Incorporated
• Inventors: Mohan, Usha, Mohan, Rengaswamy
• Primary Examiner(s): AMSBURY, WAYNE P
• Application Number:
  US10/087,053
• Time in Patent Office: 1,369 Days
• Field of Search: 707/102
• US Class Current: 1/1
• CPC Class Codes: G06F 16/284 Relational databases
• CPC Class Codes: G06F 16/36 Creation of semantic tools, e.g. ontology or thesauri
• CPC Class Codes: Y10S 707/99943 Generating database or data structure, e.g. via user interface

Abstract

A method, operating model, system, data structure, computer program and computer program product for analyzing and categorizing unstructured information is provided such that conventional structured data access techniques can be utilized over unstructured objects. A analysis and categorization engine builds a set of concept groupings, each grouping consisting of related words and phrases. The concept groupings are augmented by user input. A set of categories is built. The analysis and categorization engine generates a vector representation of each object based on concepts and utilizes a statistical analysis to select concepts that represent each object and assign objects to categories. Information about users, objects, and categories is stored in an open architecture, such as a relational database. An object concept based search is provided to efficiently locate meaningful objects and to provide for updating of the object categorization based on search entries.

Claims

1. A system for accessing and analyzing unstructured objects, where the system provides structured information through which a user can access the unstructured objects, the structured information including a set of concepts where each concept comprises at least one word, the system comprising:

a first storage medium storing at least one unstructured object;

a second storage medium storing an analysis and categorization engine procedure that, when executed, accesses the unstructured objects and generates structured information about the objects;

a third storage medium storing the structured information in a form of at least one relational database data structure, wherein the at least one relational database data structure comprises a relational database table having a global seed concept ID field, a seed concept text field, and a created date field; and

a computer processor accessible to the user, having access to the structured information.

2. A system for accessing and analyzing unstructured objects, where the system provides structured information through which a user can access the unstructured objects, the structured information including a set of concepts where each concept comprises at least one word, the system comprising:

a first storage medium storing at least one unstructured object;

a second storage medium storing an analysis and categorization engine procedure that, when executed, accesses the unstructured objects and generates structured information about the objects;

a third storage medium storing the structured information in a form of at least one relational database data structure, wherein the at least one relational database data structure comprises a relational database table having a user ID field, a global seed concept ID field, a related concept ID field, a type of relationship field, and a status field; and

a computer processor accessible to the user, having access to the structured information.

3. A system for accessing and analyzing unstructured objects, where the system provides structured information through which a user can access the unstructured objects, the structured information including a set of concepts where each concept comprises at least one word, the system comprising:

a first storage medium storing at least one unstructured object;

a second storage medium storing an analysis and categorization engine procedure that, when executed, accesses the unstructured objects and generates structured information about the objects;

a third storage medium storing the structured information in a form of at least one relational database data structure, wherein the at least one relational database data structure comprises a relational database table having an object ID field, a concept ID field, a cross-reference time stamp field, a cross-reference type field, an index start time field, and a total hits field; and

a computer processor accessible to the user, having access to the structured information.

4. A system for accessing and analyzing unstructured objects, where the system provides structured information through which a user can access the unstructured objects, the structured information including a set of concepts where each concept comprises at least one word, the system comprising:

a first storage medium storing at least one unstructured object;

a second storage medium storing an analysis and categorization engine procedure that, when executed, accesses the unstructured objects and generates structured information about the objects;

a third storage medium storing the structured information in a form of at least one relational database data structure, wherein the at least one relational database data structure comprises a relational database table having a user object id field, a key concept id field, a probability field, and a rank field; and

a computer processor accessible to the user, having access to the structured information.

5. A system for accessing and analyzing unstructured objects, where the system provides structured information through which a user can access the unstructured objects, the structured information including a set of concepts where each concept comprises at least one word, the system comprising:

a first storage medium storing at least one unstructured object;

a second storage medium storing an analysis and categorization engine procedure that, when executed, accesses the unstructured objects and generates structured information about the objects;

a third storage medium storing the structured information in a form of at least one relational database data structure, wherein the at least one relational database data structure comprises a relational database table having a user ID field, a user object ID field, an object ID field, a user object hierarchy pointer field, and object status field, and an object score field; and

a computer processor accessible to the user, having access to the structured information.

6. A relational database structure, embodied on a computer-readable medium, for storing structured information about an object, the database structure comprising:

at least one relational database table having a global seed concept ID field and a seed concept text field, and a created date field whereby the structured information about the object is made accessible to a user.

7. The relational database data structure of claim 6, wherein the at least one relational database data structure comprises a relational database table having a user ID field, a global seed concept ID field, a related concept ID field, a type of relationship field, and a status field.

8. The relational database structure of claim 6, wherein the at least one relational database data structure comprises a relational database table having an object ID field, a concept ID field, a cross-reference time stamp field, a cross-reference type field, an index start time field, and a total hits field.

9. The relational database structure of claim 6, wherein the at least one relational database data structure comprises a relational database table having a user object id field, a key concept id field, a probability field, and a rank field.

10. The relational database data structure of claim 6, wherein the at least one relational database data structure comprises a relational database table having a user ID field, a user object ID field, an object ID field, a user object hierarchy pointer field, and object status field, and an object score field.

11. A computer-based method for automatically assigning at least one key concept to represent an unstructured object, comprising:

automatically selecting at least one concept from the unstructured object without requiring user input to identify the concept;

expanding each selected concept into at least one concept grouping, wherein a concept grouping contains elements consisting of a seed concept and at least one related concept;

scoring each concept grouping to indicate the relevance of the concept grouping to the unstructured object; and

applying probabilistic analysis to the scores of the concept groupings to identify at least one relevant concept grouping, wherein for each relevant concept grouping, the seed concept of the relevant concept grouping is a key concept of the unstructured object, whereby at least one key concept is assigned to represent the unstructured object.

12. The method of claim 11, wherein the selecting step comprises:

filtering undesired elements from the unstructured object to produce a filtered object; and

selecting at least one concept from the filtered object.

13. The method of claim 12, wherein the filtering step comprises:

removing predetermined characters, words, and phrases from the unstructured object to produce a filtered object.

14. The method of claim 11, wherein the expanding step comprises:

selecting at least one concept grouping from a set of preexisting concept groupings wherein a selected concept matches an element of the concept grouping.

15. The method of claim 11, wherein the expanding step comprises:

using preexisting information to create at least one concept grouping wherein a selected concept is an element of the concept grouping.

16. The method of claim 11, wherein the expanding step comprises:

using preexisting information to create at least one concept grouping wherein a selected concept is an element of the concept grouping and the preexisting information is selected from the group consisting of;

meaning words, synonyms, related words, and user-entered words.

17. The method of claim 11, wherein the scoring step comprises:

scoring each concept grouping based on the frequency that an element of the concept grouping appears in the unstructured object.

18. The method of claim 11, wherein the scoring step comprises:

scoring each concept grouping based on the position where an element of the concept grouping appears in the unstructured object.

19. The method of claim 11, wherein the scoring step comprises:

scoring each concept grouping based on part of speech usage of an element of the concept grouping in the unstructured object.

20. The method of claim 11, wherein the applying step comprises:

applying probabilistic analysis to the scores of the concept groupings to identify at least one relevant concept grouping, wherein the probabilistic analysis is Gaussian distribution analysis.

21. A computer-based method of processing a set of unstructured objects, comprising:

(1) selecting an unstructured object from the set of unstructured objects;

(2) automatically selecting at least one concept from the selected unstructured object without requiring user input to identify the concept;

(3) expanding each selected concept into at least one concept grouping, wherein a concept grouping contains elements consisting of a seed concept and at least one related concept;

(4) scoring each concept grouping to indicate the relevance of the concept grouping to the selected unstructured object;

(5) applying probabilistic analysis to the scores of the concept groupings to identify at least one relevant concept grouping, wherein for each relevant concept grouping, the seed concept of the relevant concept grouping is a key concept of the selected unstructured object and the score of the relevant concept grouping is the score of the key concept of the selected unstructured object, whereby at least one key concept is assigned to represent the selected unstructured object; and

(6) repeating steps (1)–

(5) for each unstructured object in the set of unstructured objects.

22. The method of claim 21, wherein step (4) comprises:

scoring each concept grouping based on the frequency that an element of the concept grouping appears in the set of unstructured objects.

23. The method of claim 21, further comprising:

(7) creating a category for the set of unstructured objects, wherein a category comprises a name and a description.

24. The method of claim 23, wherein step (7) comprises:

(a) assigning a category-relevance value to each key concept in the set of unstructured objects, wherein a category-relevance value of a key concept indicates the relevance of the key concept to the set of unstructured objects;

(b) using the category-relevance values to select a set of key concepts; and

(c) assigning the set of key concepts as the description of the category.

25. The method of claim 24, further comprising:

(d) assigning at least one key concept in the set of key concepts as the name of the category.

26. The method of claim 24, wherein step (a) comprises:

assigning a category-relevance value to each key concept in the set of unstructured objects based on the frequency that the key concept appears in the set of unstructured objects.

27. The method of claim 24, wherein step (a) comprises:

assigning a category-relevance value to each key concept in the set of unstructured objects based on the scores of the key concepts of the set of unstructured objects.

28. The method of claim 24, wherein step (b) comprises:

applying probabilistic analysis to the category-relevance values to select a set of key concepts wherein the probabilistic analysis is Gaussian distribution analysis.

29. The method of claim 21, wherein step (5) further comprises:

assigning the selected unstructured object to at least one category from a set of categories.

30. The method of claim 29 wherein the assigning step comprises:

(a) selecting a category from the set of categories;

(b) calculating an object-relevance value for the selected unstructured object and the selected category, wherein the object-relevance value indicates the relevance of the unstructured object to the selected category;

(c) assigning the selected unstructured object to the selected category if the object-relevance value is greater than a threshold; and

(d) repeating steps (a)–

(c) for each category in the set of categories.

31. The method of claim 30, wherein the calculating step comprises:

extracting category-concepts from the selected category wherein category-concepts are concepts extracted from the name and description of the selected category; and

calculating an object-relevance value based on the key concepts of the unstructured object, category-concepts of the selected category, and the frequency that category-concepts appear in the set of categories.

32. The method of claim 21, further comprising:

(7) using search text to retrieve from the set of unstructured objects a list of unstructured objects that are relevant to the search text.

33. The method of claim 32, wherein step (7) comprises:

(a) extracting search-concepts from the search text wherein search-concepts are concepts which represent the search text;

(b) calculating a search-relevance value for each unstructured object based on the search-concepts and the key concepts of the unstructured object, wherein a search-relevance value indicates the relevance of an unstructured object to the search text; and

(c) creating a list of unstructured objects wherein each unstructured object in the list has a search-relevance value greater than a threshold.

34. A computer program product comprising a computer useable medium having computer readable program code means embedded in said medium for causing a computer to process a set of unstructured objects, comprising:

first computer readable program code means for causing the computer to select an unstructured object from the set of unstructured objects;

second computer readable program code means for causing the computer to select at least one concept from the selected unstructured object without requiring user input to identify the concept;

third computer readable program code means for causing the computer to expand each selected concept into at least one concept grouping, wherein a concept grouping contains elements consisting of a seed concept and at least one related concept;

fourth computer readable program code means for causing the computer to score each concept grouping to indicate the relevance of the concept grouping to the selected unstructured object;

fifth computer readable program code means for causing the computer to apply probabilistic analysis to the scores of the concept groupings to identify at least one relevant concept grouping, wherein for each relevant concept grouping, the seed concept of the relevant concept grouping is a key concept of the selected unstructured object and the score of the relevant concept grouping is the score of the key concept of the selected unstructured object, whereby at least one key concept is assigned to represent the selected unstructured object; and

sixth computer readable program code means for causing the first through fifth computer readable program code means to assign a key concept to represent each unstructured object in the set of unstructured objects.

35. The computer program product of claim 34, further comprising:

seventh computer readable program code means for causing the computer to create a category for the set of unstructured objects, wherein a category comprises a name and a description.

36. The computer program product of claim 35, wherein the seventh computer readable program code means comprises:

eighth computer readable program code means for causing the computer to assign a category-relevance value to each key concept in the set of unstructured objects, wherein a category-relevance value of a key concept indicates the relevance of the key concept to the set of unstructured objects;

ninth computer readable program code means for causing the computer to use the category-relevance values to select a set of key concepts; and

tenth computer readable program code means for causing the computer to assign the set of key concepts as the description of the category.

37. The computer program product of claim 34, wherein the fifth computer readable program code means comprises:

seventh computer readable program code means for causing the computer to assign the selected unstructured object to at least one category from a set of categories.

38. The computer program product of claim 37 wherein the seventh computer readable program code means comprises:

eighth computer readable program code means for causing the computer to select a category from the set of categories;

ninth computer readable program code means for causing the computer to calculate an object-relevance value for the selected unstructured object and the selected category, wherein the object-relevance value indicates the relevance of the unstructured object to the selected category;

tenth computer readable program code means for causing the computer to assign the selected unstructured object to the selected category if the object-relevance value is greater than a threshold; and

eleventh computer readable program code means for causing the computer to cause the eighth through tenth computer program code means to repeat performance of their respective functions for each category in the set of categories.

39. The computer program product of claim 34, further comprising:

seventh computer readable program code means for causing the computer to use search text to retrieve a list of unstructured objects from the set of unstructured objects that are relevant to the search text.

40. The computer program product of claim 39, wherein the seventh computer readable program code means comprises:

eighth computer readable program code means for causing the computer to extract search-concepts from the search text wherein search-concepts are concepts which represent the search text;

ninth computer readable program code means for causing the computer to calculate a search-relevance value for each unstructured object based on the search-concepts and the key concepts of the unstructured object, wherein a search-relevance value indicates the relevance of an unstructured object to the search text; and

tenth computer readable program code means for causing the computer to create a list of unstructured objects wherein each unstructured object in the list has a search-relevance value greater than a threshold.