System and method for word indexing in a capture system and querying thereof

• US 7,818,326 B2
• Filed: 08/31/2005
• Issued: 10/19/2010
• Est. Priority Date: 08/31/2005
• Status: Expired due to Fees

• Current Assignee: McAfee, LLC
• Original Assignee: McAfee, Inc. (McAfee, LLC)
• Inventors: Deninger, William, de la Iglesia, Erik
• Primary Examiner(s): Alam; Hosain T
• Assistant Examiner(s): Tang; Jieying
• Application Number:
  US11/218,167
• Publication Number:
  US 20070050334A1
• Time in Patent Office: 1,875 Days
• Field of Search: 707/2, 707/3, 707/999.002, 707/999.03, 707/747, 707/769, 711/216
• US Class Current: 707/747
• CPC Class Codes: G06F 16/325 Hash tables

Abstract

Searching of objects captured by a capture system can be improved by eliminating irrelevant objects from a query. In one embodiment, the present invention includes receiving such a query for objects captured by a capture system, the query including at least one search term. This search term is then hashed to a term bit position using a hash function. Then objects can be eliminated if, in a word index associated with the object, the term bit position is not set.

Claims

1. A method to be performed by a processor in an electronic environment, comprising:

receiving a query for an object captured by a capture system, the query including at least one search term, wherein a content type of the object is determined and the content type is positioned in a content field of a tag associated with the object;

hashing the at least one search term to a term bit position using a hash function; and

eliminating at least one other object from the query using a word index associated with the at least one other object, the word index comprising a plurality of bits, if a bit in the term bit position is not set, wherein eliminating at least one other object from the query comprises performing a search over a tag database containing tags associated with the captured objects.

2. The method of claim 1, further comprising retrieving at least one object, wherein a second word index associated with the retrieved object included a set bit in the term bit position.

3. The method of claim 1, wherein each tag in the tag database comprises an index field, the index field including a plurality of bits, wherein at least one of the bits is set.

4. The method of claim 3, wherein the at least one set bit has a value of “

1”

.

5. A method to be performed by a processor in an electronic environment, the method comprising:

capturing an object being transmitted over a network, wherein a content type of the object is determined and the content type is positioned in a content field of a tag associated with the object;

extracting text contained in the captured object;

generating a plurality of tokens from the extracted text;

hashing the plurality of tokens to a plurality of bit position values; and

creating a word index to be associated with the captured object by setting bits of a bit vector corresponding to the plurality of bit position values, wherein generating the plurality of tokens comprises tokenizing the extracted text using a context-aware parser, and wherein the context-aware parser uses a list of patterns associated with a content type of the captured object to sub-tokenize at least one of the tokens generated from the extracted text.

6. The method of claim 5, wherein the context-aware parser uses a list of separators associated with a content type of the captured object.

7. The method of claim 5, further comprising inserting the word index into a tag associated with the captured object.

8. The method of claim 7, further comprising storing the captured object in a canonical object store and storing the tag associated with the captured object in a tag database.

9. The method of claim 5, further comprising:

receiving a query including a search term;

hashing the search term to a term bit position of the bit vector; and

eliminating the captured object from the query by observing that a bit in the term bit position is not set in the word index contained in the tag associated with the captured object.

10. A capture system comprising:

one or more capture modules including a network interface card to capture an object being transmitted over a network, wherein a content type of the object is determined and the content type is positioned in a content field of a tag associated with the object;

a text extractor to extract text contained in the captured object;

a tokenizer to generate a plurality of tokens from the extracted text;

a sub-tokenizer to sub-tokenize the plurality of tokens comprises using a context-aware parser; and

an index generator to create a word index by hashing the plurality of tokens to a plurality of bit position values and setting bits of a bit vector corresponding to the plurality of bit position values, wherein the context-aware parser uses a list of patterns associated with a content type of the captured object to sub-tokenize at least one of the tokens generated from the extracted text.

11. The capture system of claim 10, wherein the context-aware parser uses a list of separators associated with a content type of the captured object.

12. The capture system of claim 10, wherein the index generator also inserts the word index into a tag associated with the captured object.

13. The capture system of claim 12, further comprising an object store to store the captured object and a tag database to store the tag associated with the captured object.

14. A machine-readable medium having stored thereon data representing instructions that, when executed by a processor, cause the processor to perform operations comprising:

receiving a query for an object captured by a capture system, the query including at least one search term, wherein a content type of the object is determined and the content type is positioned in a content field of a tag associated with the object;

hashing the at least one search term to a term bit position using a hash function; and

eliminating at least one other object from the query using a word index associated with the at least one other object, the word index comprising a plurality of bits, if a bit in the term bit position is not set, wherein eliminating the at least one other object from the query comprises performing a search over a tag database containing tags associated with the capture objects.

15. The machine-readable medium of claim 14, wherein the instructions further cause the processor to retrieve at least one object, wherein a second word index associated with the retrieved object included a set bit in the term bit position.

16. The machine-readable medium of claim 14, wherein each tag in the tag database comprises an index field, the index field including a plurality of bits, wherein at least one of the bits is set.

17. The machine-readable medium of claim 16, wherein the at least one set bit has a value of “

1”

.

18. A machine-readable medium having stored thereon data representing instructions that, when executed by a processor of a capture system, cause the processor to perform operations comprising:

extracting text contained in a captured object, wherein a content type of the object is determined and the content type is positioned in a content field of a tag associated with the object;

generating a plurality of tokens from the extracted text;

hashing the plurality of tokens to a plurality of bit position values; and

creating a word index to be associated with the captured object by setting bits of a bit vector corresponding to the plurality of bit position values, wherein generating the plurality of tokens comprises tokenizing the extracted text using a context-aware parser, and wherein the context-aware parser uses a list of patterns associated with a content type of the captured object to sub-tokenize at least one of the tokens generated from the extracted text.

19. The machine-readable medium of claim 18, wherein the context-aware parser uses a list of separators associated with a content type of the captured object.

20. The machine-readable medium of claim 18, wherein the instructions further cause the processor to insert the word index into a tag associated with the captured object.

21. The machine-readable medium of claim 20, wherein the instructions further cause the processor to store the captured object in a canonical object store and storing the tag associated with the captured object in a tag database.

22. The machine-readable medium of claim 18, wherein the instructions further cause the processor perform operations comprising:

receiving a query including a search term;

hashing the search term to a term bit position of the bit vector; and

eliminating the captured object from the query by observing that a bit in the term bit position is not set in the word index contained in the tag associated with the captured object.