System and method for tokening documents

• US 7,275,069 B2
• Filed: 04/26/2004
• Issued: 09/25/2007
• Est. Priority Date: 04/26/2004
• Status: Expired due to Fees

• Current Assignee: Intel Corporation
• Original Assignee: Tarari, Inc. (Broadcom, Inc.)
• Inventors: McMillen, Robert J., Lemoine, Eric, Hundley, Douglas
• Primary Examiner(s): Cottingham; John
• Assistant Examiner(s): Lu; Kuen S.
• Application Number:
  US10/831,956
• Publication Number:
  US 20050240911A1
• Time in Patent Office: 1,247 Days
• Field of Search: 707/104.1, 707/3, 707/6, 717/114
• US Class Current: 707/737
• CPC Class Codes: G06F 40/143 Markup, e.g. Standard Generalized Markup Language [SGML] or Document Type Definition [DTD]
• CPC Class Codes: G06F 40/221 Parsing markup language streams streaming G06F40/149
• CPC Class Codes: G06F 8/425 Lexical analysis
• CPC Class Codes: Y10S 707/99933 Query processing, i.e. searching
• CPC Class Codes: Y10S 707/99936 Pattern matching access
• CPC Class Codes: Y10S 707/99945 Object-oriented database structure processing

Abstract

A system for tokenizing a document, such as, for example, an XML document. A classifier is configured to assign the at least one character to at least one of a plurality of character classes. Each of a plurality of token logic units is configured to concurrently perform a comparison as specified by an instruction. A comparison may comprise comparing the at least one character class to an operand. An execution unit is configured to select an action from the instruction in response to performing the comparisons and to perform the action. A method of tokenizing a document includes assigning at least one character from a document to at least one of a plurality of character classes and concurrently performing a plurality of comparisons. At least one of the plurality of comparisons comprises comparing the assigned character class to the character from the document. At least one action to be performed is selected based on at least one result produced by performing the comparisons, and the selected action is subsequently performed.

Claims

1. A method of tokenizing a document the method comprising:

receiving at least a portion of a document, the portion comprising at least one character;

assigning the at least one character to at least one of a plurality of character classes; and

concurrently performing a plurality of comparisons defined by at least one instruction wherein performing at least one of the plurality of comparisons comprises comparing the assigned character classes to an operand of the instruction;

selecting at least one of a plurality of executable actions based on at least one result of performing the plurality of comparisons;

executing the at least one of a plurality of executable actions; and

storing tokenizing state information to a memory based on executing the action.

2. The method of claim 1, wherein performing at least one of the plurality of comparisons comprises comparing the at least one character to said operand associated with said instruction.

3. The method of claim 1, wherein performing at least one of the plurality of comparisons comprises comparing saved tokenizing state information to said operand associated with said instruction.

4. The method of claim 1, wherein storing the tokenizing state information comprises outputting a token to the memory.

5. The method of claim 1, wherein at least one of the steps of assigning and performing is performed using an FPGA (Field Programming Gate Arrays).

6. A system for tokenizing a document the system comprising:

a memory configured to store at least a portion of the document, the portion comprising at least one character,

wherein the memory is further configured to store a plurality of instructions wherein each of the plurality of instructions defines a plurality of comparisons and a plurality of actions and

wherein each comparison comprises an operand and wherein the memory is further configured to store tokenizing state information;

an instruction pointer configured to identify one of the plurality of instructions;

a classifier configured to assign the at least one character to one of a plurality of character classes;

a plurality of token logic units configured to operate, at least in part, concurrently,

wherein each of the plurality of token logic units is configured to perform one of the plurality of comparisons of the identified instruction so as to produce an output and

wherein at least one of the plurality of token logic units is configured to perform the respective one of the plurality of comparisons by comparing the one of the plurality of character classes to the operand of the respective one of the plurality of comparisons; and

an execution unit configured to select an action from the plurality of actions in response to the output of one of the plurality of token logic units and wherein the execution unit is further configured to execute the selected action and wherein the execution unit is configured to store tokenizing state information to the memory based on the selected action.

7. The system of claim 6, wherein the plurality of token logic units comprises at least three token logic units.

8. The system of claim 6, wherein the classifier is configured to identify the at least one character based on XML.

9. The system of claim 6, wherein at least one of the plurality of token logic units is configured to compare the at least one character to the operand of one of the plurality of comparisons.

10. The system of claim 6, wherein at least one of the plurality of token logic units is configured to compare tokenizing state information to the operand of the at least one of the plurality of comparisons.

11. The system of claim 6, wherein said memory comprises a first memory configured to store the token and a second memory configured to store the instructions.

12. An integrated circuit comprising a computer readable medium having stored thereon a software defining a process that when being executed causes a logic associated therewith to perform the acts of:

receiving at least one character from a document;

assigning the at least one character to at least one of a plurality of character classes;

loading an instruction wherein the instruction comprises a plurality of comparisons and a plurality of actions and wherein each of the plurality of comparisons comprises at least one operand;

concurrently performing at least some of the plurality of comparisons wherein performing said at least one of the plurality of comparisons comprises comparing the at least one of the plurality of character classes with the at least one operand of the instruction;

selecting at least one of the plurality of actions to perform based on at least one result of the comparing; and

executing the at least one of the plurality of actions; and

storing tokenizing state information to a memory in response to executing the action.

13. The integrated circuit of claim 12, wherein performing at least one of the plurality of comparisons comprises comparing the at least one character to the at least one operand.

14. The integrated circuit of claim 12, wherein performing at least one of the plurality of comparisons comprises comparing tokenizing state information to the at least one operand.

15. The integrated circuit of claim 12, wherein storing the tokenizing state information comprises outputting a token to the memory.

16. The integrated circuit of claim 12 comprising an FPGA (Field Programming Gate Arrays).

17. A computer implemented system for tokenizing a document, the system comprising:

means for receiving at least a portion of a document, the portion comprising at least one character;

means for assigning the at least one character to at least one of a plurality of character classes;

means for identifying at least one instruction;

means for concurrently comparing at least two of the plurality of character classes to an operand defined by the instruction;

means for selecting at least one action from a plurality of actions defined by the instruction to perform in response to the means for comparing; and

means for executing the at least one action,

wherein the executing means is configured to store tokenizing state information to a memory based on executing the at least one action.

18. The system of claim 17, wherein the comparing comprises comparing the at least one character to an operand.

19. The system of claim 17, wherein the comparing comprises comparing saved tokenizing state information to an operand.

20. The system of claim 17, wherein storing tokenizing state information comprises storing a token.

21. The system of claim 17, wherein at least one of the means for assigning, comparing, selecting, and executing comprises an FPGA (Field Programming Gate Arrays).