System and method for word indexing in a capture system and querying thereof

• US 8,554,774 B2
• Filed: 09/01/2010
• Issued: 10/08/2013
• Est. Priority Date: 08/31/2005
• Status: Active Grant

• Current Assignee: McAfee, LLC
• Original Assignee: McAfee, Inc. (McAfee, LLC)
• Inventors: Deninger, William, de la Iglesia, Erik
• Primary Examiner(s): Lin, Shew-Fen
• Assistant Examiner(s): Tang, Jieying
• Application Number:
  US12/873,860
• Publication Number:
  US 20110004599A1
• Time in Patent Office: 1,133 Days
• Field of Search: 707/999.002, 707/999.003, 707/747, 707/769, 711/216
• US Class Current: 707/747
• CPC Class Codes: G06F 16/325 Hash tables

Abstract

Searching of objects captured by a capture system can be improved by eliminating irrelevant objects from a query. In one embodiment, the present invention includes receiving such a query for objects captured by a capture system, the query including at least one search term. This search term is then hashed to a term bit position using a hash function. Then objects can be eliminated if, in a word index associated with the object, the term bit position is not set.

Claims

1. A method, comprising:

receiving a query to search a plurality of objects captured by a capture system, the query including a search term;

generating a search token from the search term using a context-aware parser, wherein the context-aware parser uses a list of patterns associated with a content type indicated by the query to generate sub-tokenized search tokens from the search token;

hashing the sub-tokenized search tokens to one or more term bit positions using a hash function;

searching a first word index associated with a first object;

eliminating the first object from the query if a bit is not set in each of the one or more term bit positions of a first bit vector of the first word index, wherein each bit that is set in the first bit vector represents at least one token generated from the first object, and wherein the hashing of the sub-tokenized search tokens includes truncating several of the sub-tokenized search tokens such that they stem to a same token.

2. The method of claim 1, wherein the context-aware parser uses a list of separators associated with the content type indicated by the query.

3. The method of claim 1, further comprising:

searching a second word index associated with a second object;

retrieving the second object if a bit is set in each of the one or more term bit positions of a second bit vector of the second word index.

4. The method of claim 1, wherein the eliminating the first object from the query comprises performing a search over a tag database, wherein each tag in the tag database comprises one of a plurality of word indexes.

5. The method of claim 1, wherein the bit has a value of “

1”

if the bit is set in the bit vector.

6. The method of claim 1, further comprising:

extracting text contained in the first object;

generating a plurality of text tokens from the extracted text using the context-aware parser;

hashing the plurality of text tokens to a plurality of bit position values; and

creating the first word index by setting bits of the first bit vector of the first word index, one or more of the bits corresponding to the plurality of bit position values, wherein the extracting, the generating, the hashing, and the creating occur prior to the receiving a query.

7. One or more non-transitory computer readable media containing logic encoded therein for performing operations when the logic is executed by one or more processors, the operations comprising:

receiving a query to search a plurality of objects captured by a capture system, the query including a search term;

generating a search token from the search term using a context-aware parser, wherein the context-aware parser uses a list of patterns associated with a content type indicated by the query to generate sub-tokenized search tokens from the search token;

hashing the sub-tokenized search tokens to one or more term bit positions using a hash function;

searching a first word index associated with a first object;

eliminating the first object from the query if a bit is not set in each of the one or more term bit positions of a first bit vector of the first word index, wherein each bit that is set in the first bit vector represents at least one token generated from the first object, and wherein the hashing of the sub-tokenized search tokens includes truncating several of the sub-tokenized search tokens such that they stem to a same token.

8. The one or more non-transitory computer readable media of claim 7, wherein the context-aware parser uses a list of separators associated with the content type indicated by the query.

9. The one or more non-transitory computer readable media of claim 7, the logic performing further operations when executed by the one or more processors, comprising:

searching a second word index associated with a second object;

retrieving the second object if a bit is set in each of the one or more term bit positions of a second bit vector of the second word index.

10. The one or more non-transitory computer readable media of claim 7, wherein the eliminating the first object from the query comprises performing a search over a tag database, wherein each tag in the tag database comprises one of a plurality of word indexes.

11. An apparatus, comprising:

a query module; and

one or more processors configured to execute instructions associated with the query module such that the apparatus is configured for;

receiving a query to search a plurality of objects captured by a capture system, the query including a search term;

generating a search token from the search term using a context-aware parser, wherein the context-aware parser uses a list of patterns associated with a content type indicated by the query to generate sub-tokenized search tokens from the search token;

hashing the sub-tokenized search tokens to one or more term bit positions using a hash function;

searching a first word index associated with a first object;

eliminating the first object from the query if a bit is not set in each of the one or more term bit positions of a first bit vector of the first word index, wherein each bit that is set in the first bit vector represents at least one token generated from the first object, and wherein the hashing of the sub-tokenized search tokens includes truncating several of the sub-tokenized search tokens such that they stem to a same token.

12. The apparatus of claim 11, wherein the context-aware parser uses a list of separators associated with the content type indicated by the query.

13. The apparatus of claim 11, the one or more processors configured to execute further instructions comprising:

searching a second word index associated with a second object;

retrieving the second object if a bit is set in each of the one or more term bit positions of a second bit vector of the second word index.

14. The apparatus of claim 13, further comprising:

a word index module, wherein the one or more processors are configured to execute further instructions associated with the word index module including;

extracting text contained in the second object;

generating a plurality of text tokens from the extracted text using the context-aware parser;

hashing the plurality of text tokens to a plurality of bit position values; and

creating the second word index by setting bits of the second bit vector of the second word index, one or more of the bits corresponding to the plurality of bit position values, wherein the extracting, the generating, the hashing, and the creating occur prior to the receiving a query.