Method and system for semantic search and retrieval of electronic documents

• US 20060235843A1
• Filed: 01/31/2006
• Published: 10/19/2006
• Est. Priority Date: 01/31/2005
• Status: Abandoned Application

• Current Assignee: TextDigger, Inc.
• Original Assignee: TextDigger, Inc.
• Inventors: Musgrove, Timothy A., Walsh, Robin H.
• Application Number:
  US11/343,084
• Publication Number:
  US 20060235843A1
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/313 Selection or weighting of terms for indexing
• CPC Class Codes: G06F 16/3344 using natural language analysis
• CPC Class Codes: G06F 16/3346 using probabilistic model
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques

Abstract

A system and method for semantic search for electronic documents stored on a computer readable media, and providing a search result in response to a query. The system includes a corpus including a plurality of electronic documents that are domain tagged at a document level and analyzed based on the tags to identify word usage patterns. An index of word usage patterns is provided that indexes the plurality of documents in the corpus according to their word usage patterns. The system also includes a query pre-processing module that receives a query from a user, and analyzes the query to determine probable word usage patterns in the query. The system further includes a processor that uses the index to identify documents having word usage patterns that matches the probable word usage patterns in the query as a candidate electronic document, and retrieves the candidate electronic document.

Claims

1. A system for semantic search for electronic documents stored on a computer readable media, and providing a search result in response to a query, comprising:

a corpus including a plurality of electronic documents that are tagged at a document level to identify general domain of each electronic document, and are analyzed based at least partially on said tags to identify word usage patterns in said plurality of electronic documents;

an index of word usage patterns that indexes said plurality of documents in said corpus according to word usage patterns and said domain tags of said plurality of electronic documents;

a query pre-processing module that receives a query from a user, and analyzes said query to determine probable word usage patterns in said query; and

a processor that uses said index to identify at least one of said electronic documents having word usage patterns that matches said probable word usage patterns in said query as a candidate electronic document, and retrieves said candidate electronic document.

2. The system of claim 1, further including a post-processing module that analyzes said retrieved candidate electronic document to determine exactness of match between said probable word usage patterns of said query and word usage patterns of said candidate electronic document.

3. The system of claim 2, wherein said processor identifies a plurality of candidate electronic documents determined to have matching word usage patterns.

4. The system of claim 3, wherein said processor ranks said retrieved candidate electronic documents based on exactness of match, and provides candidate electronic documents with the highest ranking as a search result.

5. The system of claim 1, wherein said word usage patterns of said index are clustered based on similarity between said patterns.

6. The system of claim 1, wherein said query pre-processing module is further adapted to disambiguate word sense in said query.

7. The system of claim 6, wherein said query pre-processing module further at least one of elicits contextual information from a user, receives a selection of a word usage pattern or a set of synonyms from a user, and selects a ranked, probabilistic word usage pattern.

8. The system of claim 6, wherein said query pre-processing module further at least one of:

selects a topic and a sub-topic within a domain of said query;

recognizes an ontological element of said query;

select a synonym or a set of synonyms for at least one word in said query;

determines interrogative type of said query;

identifies a multiword term in said query;

identifies a proper name in said query;

corrects spelling and grammar of a multiple word pattern in said query; and

performs semantic analysis of common verbs and adjectives in said query.

9. The system of claim 2, wherein said post-processing module determines proximity of words of said query to each other in said candidate electronic document to determine exactness of match.

10. The system of claim 9, wherein said words of said query must be within a predetermined proximity range to each other within said electronic document in order for said electronic document to be provided as a search result.

11. The system of claim 10, wherein different types of words of said query are assigned different proximity ranges.

12. The system of claim 2, wherein said post-processing module determines word order for words of said query in said candidate electronic document in determining exactness of match.

13. The system of claim 12, wherein said post-processing module assigns a word placement score based on said determined word order match.

14. The system of claim 13, wherein said post-processing module reduces said word placement score a decreasing amount as number of intervening words between words of said query in said candidate electronic document increases.

15. The system of claim 2, wherein said post-processing module further at least one of:

recognizes an ontological element in said candidate electronic document;

selects a synonym or a set of synonyms in said candidate electronic document;

identifies a multiword term in said candidate electronic document;

identifies a proper name in said candidate electronic document;

corrects spelling and grammar of a multiple word pattern in said candidate electronic document; and

performs semantic analysis of common verbs and adjectives in said candidate electronic document.

16. The system of claim 1, wherein said processor is further adapted to provide paid search content together with a search result.

17. The system of claim 16, wherein said paid search content is analyzed and provided together with said search result only if said paid search content is determined to have word usage patterns matching word usage patterns of said query.

18. The system of claim 1, wherein said query pre-processing module includes a user interface adapted to at least one of:

provide a first entry field to receive input of said query, and includes a second entry field to receive input of context clue words;

provide to the user, a real-time cue as to which domains said system is construing said query to belong to;

render said query in a first color, and change said first color to a second color when said query is disambiguated; and

prompt the user to continue entering additional words related to said query to facilitate disambiguation thereof.

19. A computer implemented method for semantic search for electronic documents stored on a computer readable media, and providing a search result in response to a query, comprising:

providing a corpus including a plurality of electronic documents that are tagged at a document level to identify general domain of each electronic document, and are analyzed based at least partially on said tags to identify word usage patterns in said plurality of electronic documents;

providing an index of word usage patterns that indexes said plurality of electronic documents in said corpus according to word usage patterns and said domain tags of said plurality of electronic documents;

receiving a query from a user;

analyzing said query to derive probable word usage patterns in said query;

using said index to identify at least one of said electronic documents that has word usage patterns matching said probable word usage patterns in said query as a candidate electronic document; and

retrieving said candidate electronic document.

20. The method of claim 19, further including analyzing said retrieved candidate electronic document to determine exactness of match between said probable word usage patterns of said query and word usage patterns of said candidate electronic document.

21. The method of claim 20, further including identifying a plurality of candidate electronic documents that have matching word usage patterns.

22. The method of claim 21, further including ranking said retrieved candidate electronic documents based on exactness of match, and providing candidate electronic documents with the highest ranking as said search result.

23. The method of claim 19, wherein said plurality of electronic documents in said corpus are tagged essentially only at a document level.

24. The method of claim 19, further including clustering said word usage patterns based on similarity between said patterns.

25. The method of claim 20, further including disambiguating word sense in said query.

26. The method of claim 25, wherein analyzing said query includes at least one of eliciting contextual information from a user, receiving a selection of a word usage pattern or a set of synonyms from a user, and selecting a ranked, probabilistic word usage pattern.

27. The method of claim 25, wherein at least one of analyzing said query and analyzing said candidate electronic document includes at least one of:

selecting a topic and a sub-topic within a domain;

recognizing an ontological element;

selecting of a synonym or a set of synonyms;

determining interrogative type;

identifying a multiword term;

identifying a proper name;

correcting spelling and grammar of a multiple word pattern; and

performing semantic analysis of common verbs and adjectives.

28. The method of claim 25, wherein said processing of said candidate electronic document to determine exactness of match includes determining proximity of words of said query to each other in said candidate electronic document.

29. The method of claim 28, wherein said words of said query must be within a predetermined proximity range to each other within said electronic document in order to be provided as a search result.

30. The method of claim 29, wherein different types of words of said query are assigned different proximity ranges.

31. The method of claim 20, wherein said processing of said candidate electronic document to determine exactness of match includes determining word order match.

32. The method of claim 31, wherein determining word order match includes assignment of a word placement score based on said determined word order match.

33. The method of claim 32, wherein said word placement score is reduced a decreasing amount as number of intervening words increases.

34. The method of claim 19, further including providing paid search content together with said search result.

35. The method of claim 34, wherein said paid search content is analyzed and provided together with said search result only if said paid search content is determined to have word usage patterns matching word usage patterns of said query.

36. The method of claim 19, further including at least one of:

generating a first entry field to receive input of said query, and generating a second entry field to receive input of context clue words;

providing a real-time cue as to which domains said query is being searched;

rendering said query in a first color, and changing said first color to a second color when said query is disambiguated; and

prompting the user to continue entering additional words related to said query to facilitate disambiguation thereof.

37. A system for semantic search for electronic documents stored on a computer readable media, and providing a search result in response to a query, comprising:

a corpus of a plurality of electronic documents;

a tagging module that tags said plurality of electronic documents in said corpus at a document level to identify general domain of each electronic document;

a word usage module that determines word usage patterns in said plurality of electronic documents in said corpus based at least partially on said tags of said plurality of electronic documents; and

an indexing module that indexes said plurality of electronic documents in said corpus at least according to word usage patterns and domain tags.

38. The system of claim 37, further including a query pre-processing module that receives a query from a user, and analyzes said query to determine probable word usage patterns in said query.

39. The system of claim 38, further including a processor that identifies at least one indexed electronic document having word usage patterns that matches said probable word usage patterns in said query as a candidate electronic document, and retrieves said candidate electronic document.

40. The system of claim 39, further including a post-processing module that analyzes said retrieved candidate electronic document to determine exactness of match between said probable word usage patterns of said query and word usage patterns of said candidate electronic document.

41. The system of claim 38, wherein said query pre-processing module disambiguates word sense in said query to identify general domain of said query.

42. A computer implemented method for semantic search for electronic documents stored on a computer readable media, and providing a search result in response to a query, comprising:

providing a corpus of a plurality of electronic documents;

tagging said plurality of electronic documents in said corpus at a document level to identify general domain of each electronic document;

determining word usage patterns in said plurality of electronic documents in said corpus based at least partially on said tags of said plurality of electronic documents; and

generating an index of word usage patterns that indexes said plurality of documents in said corpus according to said word usage patterns and said domain tags of said plurality of electronic documents.

43. The method of claim 42, further including receiving a query from a user, and analyzing said query to derive probable word usage patterns in said query.

44. The method of claim 43, further including using said generated index to identify at least one of said electronic documents that has word usage patterns matching said probable word usage patterns in said query as a candidate electronic document, and retrieving said candidate electronic document.

45. The method of claim 44, further including analyzing said retrieved candidate electronic document to determine exactness of match between said probable word usage patterns of said query and word usage patterns of said candidate electronic document.

46. The method of claim 43, further including disambiguating word sense in said query to identify general domain of said query.

47. A computer readable medium with executable instructions for semantic search for electronic documents stored on a computer readable media, and providing a search result in response to a query, comprising:

instructions for receiving a query from a user;

instructions for analyzing said query to derive probable word usage patterns in said query;

instructions for accessing an index of word usage patterns that indexes a plurality of electronic documents according to word usage patterns in said plurality of electronic documents, said plurality of electronic documents being tagged at a document level to identify general domain of each electronic document;

instructions for identifying at least one of said electronic documents that has word usage patterns matching said probable word usage patterns in said query as a candidate electronic document; and

instructions for retrieving said candidate electronic document.

48. The computer readable medium of claim 47, further including instructions for analyzing said retrieved candidate electronic document to determine exactness of match between said probable word usage patterns of said query and word usage patterns of said candidate electronic document.

49. The computer readable medium of claim 48, further including instructions for identifying a plurality of candidate electronic documents that have matching word usage patterns.

50. The computer readable medium of claim 49, further including instructions for ranking said retrieved candidate electronic documents based on exactness of match, and providing candidate electronic documents with the highest ranking as a search result.

51. The computer readable medium of claim 47, further including instructions for clustering said word usage patterns based on similarity between said patterns.

52. The computer readable medium of claim 47, further including instructions for disambiguating word sense in said query.

53. The computer readable medium of claim 52, wherein instructions for analyzing said query includes instructions for at least one of eliciting contextual information from a user, receiving a selection of a word usage pattern or a set of synonyms from a user, and selecting a ranked, probabilistic word usage pattern.

54. The computer readable medium of claim 52, wherein at least one of said instructions for analyzing said query and instructions for analyzing said candidate electronic document includes instructions for at least one of:

selecting a topic and a sub-topic within a domain;

recognizing an ontological element;

selecting of a synonym or a set of synonyms;

determining interrogative type;

identifying a multiword term;

identifying a proper name;

correcting spelling and grammar of a multiple word pattern; and

performing semantic analysis of common verbs and adjectives.

55. The computer readable medium of claim 48, wherein said instructions for processing of said candidate electronic document to determine exactness of match includes instructions for determining proximity of words of said query to each other in said candidate electronic document.

56. The computer readable medium of claim 55, wherein said words of said query must be within a predetermined proximity range to each other within said electronic document in order to be provided as a search result.

57. The computer readable medium of claim 56, wherein different types of words of said query are assigned different proximity ranges.

58. The computer readable medium of claim 55, wherein said instructions for processing of said candidate electronic document to determine exactness of match includes instructions for determining word order.

59. The computer readable medium of claim 58, wherein instructions for determining word order match includes instructions for assignment of a word placement score based on said determined word order match.

60. The computer readable medium of claim 59, wherein said instructions for determining word placement score includes instructions for reducing said word placement score a decreasing amount as number of intervening words increases.

61. The computer readable medium of claim 47, further including instructions for providing paid search content together with a search result.

62. The computer readable medium of claim 61, further including instructions for providing said paid search content together with said search result only if said paid search content is determined to have word usage patterns matching word usage patterns of said query.

63. The computer readable medium of claim 47, further including instructions for at least one of:

generating a first entry field to receive input of said query, and instructions for generating a second entry field to receive input of context clue words;

providing a real-time cue as to which domains said query is being searched;

rendering said query in a first color, and changing said first color to a second color when said query is disambiguated; and

prompting the user to continue entering additional words related to said query to facilitate disambiguation thereof.

64. A computer readable medium with executable instructions for semantic search for electronic documents stored on a computer readable media, and providing a search result in response to a query, comprising:

instructions for accessing a corpus of a plurality of electronic documents;

instructions for tagging said plurality of electronic documents in said corpus at a document level to identify general domain of each electronic document;

instructions for determining word usage patterns in said plurality of electronic documents in said corpus based at least partially on said tags of said plurality of electronic documents; and

instructions for generating an index of word usage patterns that indexes said plurality of documents in said corpus according to said word usage patterns and said domain tags of said plurality of electronic documents.

65. The computer readable medium of claim 64, further including instructions for receiving a query from a user, and analyzing said query to derive probable word usage patterns in said query.

66. The computer readable medium of claim 65, further including instructions for using said generated index to identify at least one of said electronic documents that has word usage patterns matching said probable word usage patterns in said query as a candidate electronic document, and retrieving said candidate electronic document.

67. The computer readable medium of claim 66, further including instructions for analyzing said retrieved candidate electronic document to determine exactness of match between said probable word usage patterns of said query and word usage patterns of said candidate electronic document.

68. The computer readable medium of claim 65, further including instructions for disambiguating word sense in said query to identify general domain of said query.