Method and system for processing text

• US 8,266,179 B2
• Filed: 09/30/2009
• Issued: 09/11/2012
• Est. Priority Date: 09/30/2009
• Status: Active Grant

• Current Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Original Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Inventors: Forman, George, Kirshenbaum, Evan R.
• Primary Examiner(s): Woo, Isaac M
• Application Number:
  US12/570,309
• Publication Number:
  US 20110078152A1
• Time in Patent Office: 1,077 Days
• Field of Search: 707600-899
• US Class Current: 707/791
• CPC Class Codes: G06F 16/325 Hash tables
• CPC Class Codes: G06F 16/334 Query execution G06F16/335 takes precedence
• CPC Class Codes: G06F 40/284 Lexical analysis, e.g. tokenisation or collocates

Abstract

An exemplary embodiment of the present invention provides a method of processing an electronic text document. The method includes obtaining a character from the document. The method also includes obtaining a hash input code from a character map, the hash input code corresponding to the character. The method also includes modifying a hash value based on the hash input code if the hash input code indicates that the character is part of a token, or asserting the hash value if the hash input code indicates that character is not part of a token.

Claims

1. A computer-implemented method for processing an electronic text document by performing a scan, the method comprising:

obtaining a character from the document;

obtaining a hash input code from a character map, the hash input code corresponding to the character;

modifying a hash value based on the hash input code if the hash input code indicates that the character is part of a token; and

asserting the hash value if the hash input code indicates that character is not part of a token.

2. The computer implemented method of claim 1, wherein the character map comprises a plurality of hash input codes that are deterministic randomized values, and at least one hash input code is indicative of a corresponding character being a non-token character.

3. The computer implemented method of claim 2, wherein the hash input codes are identical for interchangeable characters.

4. The computer implemented method of claim 1, wherein asserting the hash value comprises modifying a feature vector value at a position in the feature vector corresponding to the hash value.

5. The computer implemented method of claim 1, comprising identifying a document class of the electronic text document based, at least in part, on the hash value, and automatically performing an action by a computer based on the identified document class.

6. The computer implemented method of claim 1, wherein asserting the hash value comprises obtaining a weight value from a position in an array of weights corresponding with the hash value.

7. The computer implemented method of claim 1, wherein asserting the hash value comprises incrementing a token count.

8. The computer implemented method of claim 1, wherein asserting the hash value comprises obtaining a weight value from a position in an array of weights corresponding with the hash value and adding the weight to a classification score.

9. The computer implemented method of claim 1, wherein the scan comprises obtaining a character class associated with the character from the character map and asserting the hash if the character class does not match a most recent character class.

10. The computer implemented method of claim 1, wherein asserting the hash comprises combining the hash value with a previously asserted hash value to obtain a second hash value corresponding with a two token phrase and asserting the second hash value.

11. The computer implemented method of claim 1, wherein asserting the hash comprises storing the hash value in an index record, the method comprising adding the index record in a reverse index.

12. A computer system, comprising:

a processor that is configured to execute machine-readable instructions;

a storage device that is configured to store data, the data comprising a character map that includes character indicators and associated hash input codes; and

a memory device that stores instructions that are executable by the processor, the instructions comprising;

a hash input code finder configured to receive a character from an electronic text document and obtain a hash input code corresponding to the character from the character map;

a hash generator configured to modify a hash value based on the hash input code if the hash input code indicates that the character is part of a token; and

a hash handler configured to assert the hash if the hash input code indicates that the character is not part of a token.

13. The computer system of claim 12, wherein the character map comprises a plurality of hash input codes that are deterministic randomized values, and at least one hash input code is indicative of a corresponding character being a non-token character.

14. The computer system of claim 12, wherein asserting the hash comprises adding the hash value to an index record, and the system comprises an index compiler configured to add the index record to a reverse index.

15. The computer system of claim 14, wherein:

the hash input code finder is configured to receive another character from a search phrase comprising one or more search terms and obtain another hash input code corresponding to the other character from the character map;

the hash generator is configured to modify a second hash value based on the other hash input code; and

the hash handler is configured to store the second hash value to a search vector.

16. The computer system of claim 15, comprising a comparator configured to compare the search vector to the reverse index to identify a matching document within the reverse index that includes one or more of the search terms.

17. The computer system of claim 12, wherein asserting the hash comprises modifying a feature vector based on the hash value, and the computer system comprises a classification score generator configured to compute a classification score for the document based, at least in part, on the feature vector.

18. A tangible, computer-readable medium, comprising code configured to direct a processor to:

receive a character obtained from an electronic text document and obtain a corresponding hash input code from the character map;

modify a hash value based on the hash input code if the hash input code indicates that the character is part of a token; and

assert the hash value if the hash input code indicates that the character is not part of a token.

19. The tangible, computer-readable medium of claim 18, wherein asserting the hash value comprises modifying a feature vector value at a location in the feature vector corresponding to the hash value.

20. The tangible, computer-readable medium of claim 18, wherein asserting the hash comprises adding the hash value to an index record, and the tangible, computer-readable medium comprises code configured to direct the processor to sort the index record and add the index record to a reverse index.