METHOD AND SYSTEM FOR TEXT INTERPRETATION AND NORMALIZATION

• US 20100257172A1
• Filed: 04/01/2010
• Published: 10/07/2010
• Est. Priority Date: 04/01/2009
• Status: Active Grant

• Current Assignee: Touchstone Systems LLC
• Original Assignee: Touchstone Systems LLC
• Inventors: Khalid, Shiraz, Lambert, Jerry
• Granted Patent: US 8,812,459 B2
• Time in Patent Office: Days
• US Class Current: 707/739
• CPC Class Codes: G06F 40/20 Natural language analysis semantic analysis of natural language G06F40/30
• CPC Class Codes: G06F 40/284 Lexical analysis, e.g. tokenisation or collocates

Abstract

A system and method for text interpretation and normalization is presented. The method for text interpretation and normalization may include receiving a reference data entry that includes one or more strings of text and one or more associated numeric codes, creating a plurality of tokens from the one or more strings of text, each token being tied to an associated numeric code, formatting the plurality tokens with operations codes (opcodes) that provides additional information about the tokens, retrieving configuration data including the plurality of tokens, the opcodes, and numeric codes associated with the tokens, selecting one inbound, non-reference string for interpretation, comparing tokens from the configuration data to the non-reference string to determine the best matching token, and applying, using the processor, the numeric code associated with the best matching token to the non-reference string in order to normalize the non-reference string.

Claims

1. A method for text interpretation and normalization, comprising:

receiving, using a processor and a memory, a reference data entry that includes one or more strings of text and one or more associated numeric codes, wherein each associated numeric code is associated with one or more strings of text;

creating, using the processor, one or more tokens from the one or more strings of text, wherein each token is tied to an associated numeric code; and

formatting, using the processor, the one or more tokens with an operations code (opcode) that provides additional information about the token, wherein the one or more tokens may be used to interpret non-reference data and associate the non-reference data to one of the one or more associated numeric codes.

2. The method of claim 1 wherein the creating includes splitting the reference data entry into separate words.

3. The method of claim 2 wherein the creating includes randomly selecting a random number of characters from a randomly selected location in one of the separate words.

4. The method of claim 1 further comprising comparing the one or more tokens to each other and other existing tokens, if any, to determine if the one or more tokens are unique.

5. The method of claim 4 further comprising, wherein if one token of the one or more tokens is determined to not be unique, discarding the non-unique token and creating a new, different token if possible.

6. The method of claim 1 further comprising repeating the receiving, creating and formatting for additional reference data entries.

7. The method of claim 1 further comprising storing the tokens, opcodes and associated numeric codes as configuration data in a configuration database, wherein the configuration data is used to interpret non-reference data.

8. A computer-readable medium comprising instructions that are executed by a processor to perform the method of claim 1.

9. A method for text interpretation and normalization, comprising:

retrieving, using a processor and a memory, configuration data including a plurality of tokens, operation codes (opcodes) that provide additional information about the tokens, and numeric codes associated with the tokens;

selecting, using the processor, one inbound, non-reference string for interpretation;

comparing, using the processor, tokens from the configuration data to the non-reference string to determine the best matching token; and

applying, using the processor, the numeric code associated with the best matching token to the non-reference string in order to normalize the non-reference string.

10. The method of claim 9 further comprising comparing tokens from the configuration data to the non-reference string to determine matching tokens, wherein the comparing to determine the best matching token compares the matching tokens to determine the best match.

11. The method of claim 9 further comprising outputting the inbound, non-reference string with the applied numeric code.

12. The method of claim 11 wherein the outputting writes the inbound, non-reference string with the applied numeric code into a database.

13. The method of claim 9 further comprising repeating the selecting, comparing and applying for additional non-reference strings.

14. A computer-readable medium comprising instructions that are executed by a processor to perform the method of claim 9.

15. A method for text interpretation and normalization, comprising:

receiving, using a processor and a memory, a reference data entry that includes one or more strings of text and one or more associated numeric codes, wherein each associated numeric code is associated with one or more strings of text;

creating, using the processor, a plurality of tokens from the one or more strings of text, wherein each token is tied to an associated numeric code;

formatting, using the processor, the plurality tokens with operations codes (opcodes) that provides additional information about the tokens, wherein the plurality of tokens may be used to interpret non-reference data and associate the non-reference data to one of the one or more associated numeric codes;

retrieving, using the processor, configuration data including the plurality of tokens, the opcodes, and numeric codes associated with the tokens;

selecting, using the processor, one inbound, non-reference string for interpretation;

comparing, using the processor, tokens from the configuration data to the non-reference string to determine the best matching token; and

applying, using the processor, the numeric code associated with the best matching token to the non-reference string in order to normalize the non-reference string.

16. The system of claim 15 wherein the creating includes splitting the reference data entry into separate words and randomly selecting a random number of characters from a randomly selected location in one of the separate words.

17. The system of claim 15 wherein the instructions further include instructions for comparing the one or more tokens to each other and other existing tokens, if any, to determine if the one or more tokens are unique.

18. The system of claim 15 further comprising a database, wherein the instructions further include instructions for storing the tokens, opcodes and associated numeric codes as configuration data in the database, wherein the configuration data is used to interpret non-reference data.

19. The system of claim 15 wherein the memory further includes a computer program stored therein that includes instructions that are executed by the processor for interpreting non-reference strings by:

retrieving configuration data including the plurality of tokens, the opcodes, and numeric codes associated with the tokens;

selecting one inbound, non-reference string for interpretation;

comparing tokens from the configuration data to the non-reference string to determine the best matching token; and

applying the numeric code associated with the best matching token to the non-reference string in order to normalize the non-reference string.

20. The method of claim 19 further comprising comparing tokens from the configuration data to the non-reference string to determine matching tokens, wherein the comparing to determine the best matching token compares the matching tokens to determine the best match.

15-1. A system for text interpretation and normalization, comprising:

a computer including a processor and memory, wherein the memory includes a computer program stored therein that includes instructions that are executed by the processor for creating tokens by;

receiving a reference data entry that includes one or more strings of text and one or more associated numeric codes, wherein each associated numeric code is associated with one or more strings of text;

creating a plurality of tokens from the one or more strings of text, wherein each token is tied to an associated numeric code; and

formatting the plurality tokens with operations codes (opcodes) that provides additional information about the tokens, wherein the plurality of tokens may be used to interpret non-reference data and associate the non-reference data to one of the one or more associated numeric codes.