A SYSTEM AND METHOD FOR WORD INDEXING IN A CAPTURE SYSTEM AND QUERYING THEREOF

• US 20110004599A1
• Filed: 09/01/2010
• Published: 01/06/2011
• Est. Priority Date: 08/31/2005
• Status: Active Grant

• Current Assignee: McAfee, LLC
• Original Assignee: McAfee, Inc. (McAfee, LLC)
• Inventors: Deninger, William, de la Iglesia, Erik
• Granted Patent: US 8,554,774 B2
• Time in Patent Office: Days
• US Class Current: 707/747
• CPC Class Codes: G06F 16/325 Hash tables

Abstract

Searching of objects captured by a capture system can be improved by eliminating irrelevant objects from a query. In one embodiment, the present invention includes receiving such a query for objects captured by a capture system, the query including at least one search term. This search term is then hashed to a term bit position using a hash function. Then objects can be eliminated if, in a word index associated with the object, the term bit position is not set.

Claims

1-36. -36. (canceled)

37. A method, comprising:

receiving a query to search a plurality of objects captured by a capture system, the query including at least one search term;

generating one or more search tokens from the at least one search term using a context-aware parser;

hashing the one or more search tokens to a term bit position using a hash function;

searching a first word index associated with a first object;

eliminating the first object from the query if a bit is not set in the term bit position of a first bit vector of the first word index.

38. The method of claim 37, wherein the context-aware parser uses a list of separators associated with a content type indicated by the query.

39. The method of claim 37, wherein the context-aware parser uses a list of patterns associated with a content type indicated by the query to sub-tokenize at least one of the search tokens.

40. The method of claim 37, wherein the hashing the one or more search tokens includes truncating at least one of the search tokens.

41. The method of claim 37, further comprising:

searching a second word index associated with a second object;

retrieving the second object if a bit is set in the term bit position of a second bit vector of the second word index.

42. The method of claim 37, wherein the eliminating the first object from the query comprises performing a search over a tag database, wherein each tag in the tag database comprises one of a plurality of word indexes.

43. The method of claim 37, wherein the bit has a value of “

1”

if the bit is set in the bit vector.

44. The method of claim 37, further comprising:

extracting text contained in the first object;

generating a plurality of text tokens from the extracted text using the context-aware parser;

hashing the plurality of text tokens to a plurality of bit position values; and

creating the first word index by setting bits of the first bit vector of the first word index, one or more of the bits corresponding to the plurality of bit position values, wherein the extracting, the generating, the hashing, and the creating occur prior to the receiving a query.

45. Logic encoded in one or more non-transitory media that includes code for execution and when executed by one or more processors is operable to perform operations comprising:

receiving a query to search a plurality of objects captured by a capture system, the query including at least one search term;

generating one or more search tokens from the at least one search term using a context-aware parser;

hashing the one or more search tokens to a term bit position using a hash function;

searching a first word index associated with a first object;

eliminating the first object from the query if a bit is not set in the term bit position of a first bit vector of the first word index.

46. The logic of claim 45, wherein the context-aware parser uses a list of separators associated with a content type indicated by the query.

47. The logic of claim 45, wherein the context-aware parser uses a list of patterns associated with a content type indicated by the query to sub-tokenize at least one of the search tokens.

48. The logic of claim 45, wherein the hashing the one or more search tokens includes truncating at least one of the search tokens.

49. The logic of claim 45, the one or more processors operable to perform further operations comprising:

searching a second word index associated with a second object;

retrieving the second object if a bit is set in the term bit position of a second bit vector of the second word index.

50. The logic of claim 45, wherein the eliminating the first object from the query comprises performing a search over a tag database, wherein each tag in the tag database comprises one of a plurality of word indexes.

51. An apparatus, comprising:

a query module; and

one or more processors operable to execute instructions associated with the query module, including;

receiving a query to search a plurality of objects captured by a capture system, the query including at least one search term;

generating one or more search tokens from the at least one search term using a context-aware parser;

hashing the one or more search tokens to a term bit position using a hash function;

searching a first word index associated with a first object;

eliminating the first object from the query if a bit is not set in the term bit position of a first bit vector of the first word index.

52. The apparatus of claim 51, wherein the context-aware parser uses a list of separators associated with a content type indicated by the query.

53. The apparatus of claim 51, wherein the context-aware parser uses a list of patterns associated with a content type indicated by the query to sub-tokenize at least one of the search tokens.

54. The apparatus of claim 51, wherein the hashing the one or more search tokens includes truncating at least one of the search tokens.

55. The apparatus of claim 51, the one or more processors operable to execute further instructions comprising:

searching a second word index associated with a second object;

retrieving the second object if a bit is set in the term bit position of a second bit vector of the second word index.

56. The apparatus of claim 55, further comprising:

a word index module, wherein the one or more processors are operable to execute further instructions associated with the word index module including;

extracting text contained in the second object;

generating a plurality of text tokens from the extracted text using the context-aware parser;

hashing the plurality of text tokens to a plurality of bit position values; and

creating the second word index by setting bits of the second bit vector of the second word index, one or more of the bits corresponding to the plurality of bit position values, wherein the extracting, the generating, the hashing, and the creating occur prior to the receiving a query.