Embedded interaction code decoding for a liquid crystal display

• US 20070003150A1
• Filed: 06/30/2005
• Published: 01/04/2007
• Est. Priority Date: 06/30/2005
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Wang, Qiang, Dang, Yingnong, Wang, Jian, Chen, Liyong, Zhang, Chunhui, Xu, An
• Granted Patent: US 7,528,848 B2
• Time in Patent Office: Days
• US Class Current: 382/237
• CPC Class Codes: G06F 3/03545 Pens or stylus

Abstract

A method and system for decoding data on a medium that is captured by an input device is described. The method decodes a symbol design, such as an embedded interaction code (EIC), on a display screen, such as a liquid crystal display (LCD). An image that is captured by an input device, such as a universal pen-type input device with a camera, is received. The captured image is processed to output an extracted bit stream. Position data and/or metadata information is decoded from the extracted bit stream and a position of the captured image with respect to a screen of the LCD is determined based upon the decoded position data. The EIC pattern may be recognized from LCD pixel grids and a bit stream of EIC bits may be extracted based upon the recognized EIC pattern.

Claims

1. A method for decoding an embedded interactive code (EIC) pattern on a liquid crystal display (LCD), the method comprising steps of:

receiving a captured image;

processing the captured image to output an extracted bit stream; and

decoding position data from the extracted bit stream; and

determining a position of the captured image with respect to a screen of the LCD based upon the decoded position data.

2. The method of claim 1, wherein the step of processing includes steps of:

preprocessing the captured image to determine pixels corresponding to LCD pixel grids associated with the EIC pattern;

defining the LCD pixel grids based upon the pixels;

recognizing the EIC pattern from the LCD pixel grids; and

extracting a bit stream of EIC bits based upon the recognized EIC pattern to obtain the extracted bit stream.

3. The method of claim 2, wherein the step of preprocessing includes steps of:

separating pixels that lie on edges and vertexes of the LCD pixel grids as a first mask;

binary eroding the first mask to obtain a second mask, the second mask including fewer pixels than the first mask;

separating connected regions of the second mask in accordance with a first threshold to obtain a third mask and a fourth mask;

binary eroding the fourth mask to obtain a fifth mask;

defining a second threshold corresponding to a region of the fifth mask;

defining a sixth mask as pixels that belong to the regions of the fifth mask; and

merging the third and the sixth masks to define the LCD pixel grids.

4. The method of claim 3, wherein pixels that lie on edges or vertexes of the LCD pixel grids have a mask value of 1.

5. The method of claim 4, wherein the step of binary eroding the first mask includes steps of:

for each pixel of the first mask with a mask value of 1, determining whether at least three neighboring pixels of the pixel have a mask value of 1, and

assigning a mask value of 1 to the pixel in the second mask when the pixel and at least three neighboring pixels have a mask value of 1.

6. The method of claim 3, wherein the step of separating connected regions of the second mask includes steps of:

for each pixel of the second mask determining whether the pixel belongs to a first region or a second region based upon the first threshold, and

assigning a mask value of 1 to the pixel in the third mask and a mask value of 0 to the pixel in the fourth mask when the pixel belongs to the first region.

7. The method of claim 6, wherein the step of separating connected regions of the second mask further includes a step of assigning a mask value of 0 to the pixel in the third mask and a mask value of 1 to the pixel in the fourth mask when the pixel belongs to the second region.

8. The method of claim 3, wherein the step of binary eroding the fourth mask includes steps of:

for each pixel of the fourth mask with a mask value of 1, determining whether at least three neighboring pixels of the pixel have a mask value of 1, and

assigning a mask value of 1 to the pixel in the fifth mask when the pixel and at least three neighboring pixels have a mask value of 1.

9. The method of claim 2, wherein the step of recognizing the EIC pattern includes steps of:

determining an orientation of the EIC pattern in the captured image;

retrieving horizontal and vertical alignment information for the orientation; and

calculating EIC bits for the EIC pattern.

10. The method of claim 9, wherein the step of determining the orientation of the EIC pattern includes steps of:

sampling a plurality of points of each LCD pixel grid;

determining which LCD pixel grids are valid;

calculating an average grey-level difference of four orientations for each valid LCD pixel grid;

selecting a threshold number of differences for each orientation of all valid LCD pixel grids;

calculating an average grey-level difference of the selected threshold number for each orientation;

determining an orientation with the smallest average grey-level difference of the selected threshold number;

redefining a reference frame of the LCD pixel grids based upon the determined orientation; and

recalculating the average grey-level difference of all valid LCD pixel grids in the redefined reference frame.

11. The method of claim 10, wherein the calculation of the grey-level difference is defined by:

$\{\begin{array}{c}{D}_{i,j,0}=\frac{\left(G_{i,j,0}+G_{i,j,3}+G_{i,j,5}\right)-\left(G_{i,j,2}+G_{i,j,4}+G_{i,j,7}\right)}{3}\\ D_{i,j,1}=\frac{\left(G_{i,j,0}+G_{i,j,1}+G_{i,j,2}\right)-\left(G_{i,j,5}+G_{i,j,6}+G_{i,j,7}\right)}{3}\\ D_{i,j,2}=\frac{\left(G_{i,j,2}+G_{i,j,4}+G_{i,j,7}\right)-\left(G_{i,j,0}+G_{i,j,3}+G_{i,j,5}\right)}{3}\\ D_{i,j,3}=\frac{\left(G_{i,j,5}+G_{i,j,6}+G_{i,j,7}\right)-\left(G_{i,j,0}+G_{i,j,1}+G_{i,j,2}\right)}{3}\end{array}.$

12. The method of claim 9, wherein the step of calculating EIC bits for the EIC pattern includes steps of:

identifying the smallest LCD pixel grid based upon the smallest average grey level difference of four information LCD pixels for each LCD pixel grid;

retrieving EIC bits corresponding to the smallest LCD pixel grid; and

constructing bits array based upon the retrieved EIC bits.

13. A system for decoding an embedded interaction code (EIC) pattern on a liquid crystal display (LCD), the system comprising:

an input component configured to receive a captured image;

a processor component configured to process the captured image and output an extracted bit stream; and

a decoding component configured to decode position data from the extracted bit stream to determine a position of the captured image with respect to a screen of the LCD.

14. The system of claim 13, wherein the processor component includes:

a preprocessing component configured to determine pixels corresponding to LCD pixel grids associated with the EIC pattern;

a definition component configured to define the LCD pixel grids based upon the determined pixels;

a recognition, component configured to recognize the EIC pattern from the LCD pixel grids; and

an extracting component configured to extract a bit stream of EIC bits based upon the recognized EIC pattern.

15. The system of claim 14, wherein the preprocessing component includes:

a pixel separator configured to separate pixels that lie on edges and vertexes of the LCD pixel grids as a first mask;

a binary erosion component configured to erode the first mask to obtain a second mask, the second mask including fewer pixels than the first mask;

a separation component configured to separate connected regions of the second mask in accordance with a first threshold to obtain a third mask and a fourth mask;

a mask definer configured to define a sixth mask as pixels that belong to regions of the fifth mask; and

a mask merger configured to merge the third and the sixth masks to define the LCD pixel grids,

wherein the binary erosion component is further configured to erode the fourth mask to obtain a fifth mask,

wherein a second threshold is defined to correspond to a region of the fifth mask.

16. The system of claim 14, wherein the recognition component includes:

an orientation component configured to determine an orientation of the EIC pattern in the captured image;

an alignment retriever configured to retrieve horizontal and vertical alignment information for the orientation; and

a calculation component configured to calculate EIC bits for the EIC pattern.

17. The system of claim 13, wherein the input component is configured to receive the captured image from a camera.

18. The system of claim 17, wherein the camera is associated with a pen-style input device.

19. A software architecture stored on one or more computer readable media for processing data representative of an embedded interaction code pattern, comprising:

at least one component configured to process a captured image to output an extracted bit stream, to decode position data from the extracted bit stream, and to determine a position of the captured image with respect to a screen of an LCD; and

at least one application program interface to access the component.

20. The software architecture of claim 19, wherein the at least one component is further configured to preprocess the captured image to determine pixels corresponding to LCD pixel grids associated with the EIC pattern, to define the LCD pixel grids based upon the pixels, to recognize the EIC pattern from the LCD pixel grids, and to extract a bit stream of EIC bits based upon the recognized EIC pattern.