Word indexing in a capture system

• US 20070050334A1
• Filed: 08/31/2005
• Published: 03/01/2007
• Est. Priority Date: 08/31/2005
• Status: Active Grant

• Current Assignee: McAfee, LLC
• Original Assignee: McAfee, Inc. (McAfee, LLC)
• Inventors: Deninger, William, Iglesia, Erik de la
• Granted Patent: US 7,818,326 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/325 Hash tables

Abstract

Searching of objects captured by a capture system can be improved by eliminating irrelevant objects from a query. In one embodiment, the present invention includes receiving such a query for objects captured by a capture system, the query including at least one search term. This search term is then hashed to a term bit position using a hash function. Then objects can be eliminated if, in a word index associated with the object, the term bit position is not set.

Claims

1. A method comprising:

receiving a query for objects captured by a capture system, the query including at least one search term;

hashing the at least one search term to a term bit position using a hash function; and

eliminating at least one object from the query using a word index associated with the object, the word index comprising a plurality of bits, wherein a bit in the term bit position is not set.

2. The method of claim 1, further comprising retrieving at least one object, wherein a second word index associated with the retrieved object included a set bit in the term bit position.

3. The method of claim 1, wherein eliminating at least one object from the query comprises performing a search over a tag database containing tags associated with the capture objects.

4. The method of claim 3, wherein each tag in the tag database comprises an index field, the index field including a plurality of bits, wherein at least one of the bits is set.

5. The method of claim 4, wherein the at least one set bit has a value of “

1.”

6. A method performed by a capture system, the method comprising:

capturing an object being transmitted over a network;

extracting text contained in the captured object;

generating a plurality of tokens form the extracted text;

hashing the plurality of tokens to a plurality of bit position values; and

creating a word index to be associated with the captured object by setting bits of a bit vector corresponding to the plurality of bit position values.

7. The method of claim 6, wherein generating the plurality of tokens comprises tokenizing the extracted text using a context-aware parser.

8. The method of claim 6, wherein the context-aware parser uses a list of separators associated with a content type of the captured object.

9. The method of claim 6, wherein the context-aware parser uses a list of patterns associated with a content type of the captured object to sub-tokenize a token generated from the extracted text.

10. The method of claim 6, further comprising inserting the word index into a tag associated with the captured object.

11. The method of claim 10, further comprising storing the captured object in a canonical object store and storing the tag associated with the captured object in a tag database.

12. The method of claim 6, further comprising:

receiving a query including a search term;

hashing the search term to a term bit position of the bit vector; and

eliminating the captured object from the query by observing that a bit in the term bit position is not set in the word index contained in the tag associated with the captured object.

13. A capture system comprising:

a user interface configured to;

receive a query for objects captured by the capture system, the query including at least one search term;

hash the at least one search term to a term bit position using a hash function; and

eliminate at least one object from the query using a word index associated with the object, the word index comprising a plurality of bits, wherein a bit in the term bit position is not set.

14. The capture system of claim 13, wherein the user interface is further configured to retrieve at least one object, wherein a second word index associated with the retrieved object included a set bit in the term bit position.

15. The capture system of claim 13, wherein eliminating at least one object from the query comprises performing a search over a tag database containing tags associated with the capture objects.

16. The capture system of claim 15, wherein each tag in the tag database comprises an index field, the index field including a plurality of bits, wherein at least one of the bits is set.

17. The capture system of claim 15, wherein the at least one set bit has a value of “

1.”

18. A capture system:

one or more capture modules to capture an object being transmitted over a network;

a text extractor to extract text contained in the captured object;

a tokenizer to generate a plurality of tokens form the extracted text; and

an index generator to create a word index by hashing the plurality of tokens to a plurality of bit position values, and setting bits of a bit vector corresponding to the plurality of bit position values.

19. The capture system of claim 18, further comprising a sub-tokenizer to sub-tokenize the plurality of tokens comprises using a context-aware parser.

20. The capture system of claim 19, wherein the context-aware parser uses a list of separators associated with a content type of the captured object.

21. The capture system of claim 19, wherein the context-aware parser uses a list of patterns associated with a content type of the captured object to sub-tokenize a token generated from the extracted text.

22. The capture system of claim 18, wherein the index generator also inserts the word index into a tag associated with the captured object.

23. The capture system of claim 22, further comprising an object store for storing the captured object and a tag database for storing the tag associated with the captured object.

24. The capture system of claim 18, further comprising:

a user interface configured to;

receive a query for objects captured by the capture system, the query including at least one search term;

hash the at least one search term to a term bit position using a hash function; and

eliminate at least one object from the query using a word index associated with the object, the word index comprising a plurality of bits, wherein a bit in the term bit position is not set.

25. A machine-readable medium having stored thereon data representing instructions that, when executed by a processor, cause the processor to perform operations comprising:

receiving a query for objects captured by a capture system, the query including at least one search term;

hashing the at least one search term to a term bit position using a hash function; and

eliminating at least one object from the query using a word index associated with the object, the word index comprising a plurality of bits, wherein a bit in the term bit position is not set.

26. The machine-readable medium of claim 25, wherein the instructions further cause the processor to retrieve at least one object, wherein a second word index associated with the retrieved object included a set bit in the term bit position.

27. The machine-readable medium of claim 25, wherein eliminating at least one object from the query comprises performing a search over a tag database containing tags associated with the capture objects.

28. The machine-readable medium of claim 27, wherein each tag in the tag database comprises an index field, the index field including a plurality of bits, wherein at least one of the bits is set.

29. The machine-readable medium of claim 28, wherein the at least one set bit has a value of “

1.”

30. A machine-readable medium having stored thereon data representing instructions that, when executed by a processor of a capture system, cause the processor to perform operations comprising:

extracting text contained in the captured object;

generating a plurality of tokens form the extracted text;

hashing the plurality of tokens to a plurality of bit position values; and

creating a word index to be associated with the captured object by setting bits of a bit vector corresponding to the plurality of bit position values.

31. The machine-readable medium of claim 30, wherein generating the plurality of tokens comprises tokenizing the extracted text using a context-aware parser.

32. The machine-readable medium of claim 31, wherein the context-aware parser uses a list of separators associated with a content type of the captured object.

33. The machine-readable medium of claim 31, wherein the context-aware parser uses a list of patterns associated with a content type of the captured object to sub-tokenize a token generated from the extracted text.

34. The machine-readable medium of claim 30, wherein the instructions further cause the processor to insert the word index into a tag associated with the captured object.

35. The machine-readable medium of claim 34, wherein the instructions further cause the processor to store the captured object in a canonical object store and storing the tag associated with the captured object in a tag database.

36. The machine-readable medium of claim 30, wherein the instructions further cause the processor perform operations comprising:

receiving a query including a search term;

hashing the search term to a term bit position of the bit vector; and

eliminating the captured object from the query by observing that a bit in the term bit position is not set in the word index contained in the tag associated with the captured object.