Method and Apparatus for Lexical Analysis Using Parallel Bit Streams

• US 20080030383A1
• Filed: 08/06/2007
• Published: 02/07/2008
• Est. Priority Date: 08/07/2006
• Status: Active Grant

• Current Assignee: International Characters Inc.
• Original Assignee: International Characters Inc.
• Inventors: Cameron, Robert
• Granted Patent: US 8,392,174 B2
• Time in Patent Office: Days
• US Class Current: 341/55
• CPC Class Codes: G06F 16/83 Querying
• CPC Class Codes: G06F 40/00 Handling natural language data speech analysis or synthesis, speech recognition G10L
• CPC Class Codes: G06F 40/284 Lexical analysis, e.g. tokenisation or collocates
• CPC Class Codes: H03M 7/00 Conversion of a code where information is represented by a given sequence or number of digits to a code where the same , similar or subset of information is represented by a different sequence or number of digits

Abstract

One embodiment of the present invention is a method for lexical analysis of a character stream including: (a) generating one or more parallel property bit streams in response to the character stream; (b) generating one or more lexical item streams in response to the one or more parallel property bit streams; and (c) generating one or more token streams in response to the one or more lexical item streams.

Claims

1. A method for lexical analysis of a character stream comprising:

generating a plurality of parallel property bit streams in response to the character stream; and

generating a token stream in response to the parallel property bit streams.

2. :

The method of claim 1 wherein;

(a) generating a plurality of parallel property bit streams in response to the character stream comprises;

generating a basis set of bit streams comprising a plurality of parallel property bit streams in response to the character stream; and

generating a plurality of lexical item streams in response to the basis set of bit streams; and

(b) generating a token stream in response to the parallel property bit streams comprises;

generating a token stream in response to the lexical item streams.

3. :

The method of claim 2 wherein generating a token stream comprises;

identifying positions of one or more fixed-length tokens of a particular type by occurrences of a predetermined bit value in a particular lexical item stream.

4. :

The method of claim 2 wherein generating a token stream comprises;

identifying positions and lengths of one or more tokens of a particular class of variable length tokens by maximal length contiguous runs of bits in a particular lexical item stream.

5. :

The method of claim 2 wherein generating a token stream comprises;

identifying positions of one or more tokens of a particular class of variable length tokens by occurrences of a predetermined number of bits each consisting of a first bit in an opening delimiter stream and a second bit in a closing delimiter stream.

6. :

The method of claim 2 which further comprises generating a validation data stream in response to the character stream and/or one or more of the lexical item streams.

7. :

The method of claim 6 wherein the character stream is an XML character stream.

8. :

The method of claim 6 which includes validating Unicode.

9. :

The method of claim 7 which includes transcoding the character stream.