Camera-pen-tip mapping and calibration

• US 7,136,054 B2
• Filed: 01/06/2004
• Issued: 11/14/2006
• Est. Priority Date: 01/06/2004
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Dang, Yingnong, Wang, Jian, Chen, Liyong, Ma, Xiaoxu
• Primary Examiner(s): Hjerpe, Richard
• Assistant Examiner(s): Abdulselam, Abbas
• Application Number:
  US10/753,023
• Publication Number:
  US 20050146518A1
• Time in Patent Office: 1,043 Days
• Field of Search: 345/179, 345/181, 345/183, 178/19.01, 178/19.05, 178/18.09, 715/858, 715/859, 348/222.1, 348/208.3, 382/313, 361/686, 73/17.9, 335/03
• US Class Current: 345/179
• CPC Class Codes: G06F 3/0321 by optically sensing the absolute position with respect to a regularly patterned surface forming a passive digitiser, e.g. pen optically detecting position indicative tags printed on a paper sheet constructional details of pen-shaped pointing devices G06F3/03545, G06F3/03542, G06F3/037
• CPC Class Codes: G06F 3/03545 Pens or stylus

Abstract

X-y positions of a digital pen'"'"'s tip may be determined by using a calibration parameter to map the x-y positions of the respective centers of images captured by the pen'"'"'s camera. The calibration parameter may be generated by iteratively calculating estimates of the calibration parameter. Calibration input data may be produced by a user placing the pen tip in a fixed location on a surface, which may be a positionally encoded medium, such as paper, and then rotating the pen and/or moving the opposite end of the pen in various directions to capture multiple images for use in generating the calibration parameter. A user may perform such a calibration procedure without the need for complicated calibration equipment typically used in connection with conventional calibration techniques.

Claims

1. A method of calibrating a digital pen, which includes a pen tip and a camera, and of mapping a location of a center of a camera-captured image to a location of the pen tip, the method comprising:

generating a calibration parameter based on a plurality of images captured by the digital pen'"'"'s camera by placing the pen in a plurality of different orientations while keeping the pen tip at a substantially fixed location L_pentip on a positionally encoded surface, wherein the calibration parameter is generated based on a plurality of iteratively generated estimates of the calibration parameter; and

using the calibration parameter to map the location of the center of the camera-captured image to the location of the pen tip so that a stroke written by a user on the positionally encoded surface can be recovered at a location within an electronic copy of the positionally encoded surface, wherein the position of the recovered stroke within the electronic copy corresponds to a plurality of locations of the pen tip as the user was writing the stroke on the positionally encoded surface.

2. The method of claim 1, further comprising:

calculating a transform F<sub>S→

P</sub> and an inverse transform

$F_{P\to <br><br>S}=F_{S\to <br><br>P}^{-1},$

by in-array decoding and image matching, for each image of the plurality of captured images and then calculating;

$L_{\mathrm{pentip}}^i=F_{S\to <br><br>P}^i·<br><br>L_{\mathrm{virtual}-\mathrm{pentip}},i=1,2,\mathrm{\cdots <br><br>}<br><br>,N,\mathrm{and}$ $L_{\mathrm{virtual}-\mathrm{pentip}}^i=F_{P\to <br><br>S}^i·<br><br>\left(L_{\mathrm{pentip}}+\mathrm{\Delta <br><br>}<br><br><br><br>L_i\right),i=1,2,\mathrm{\cdots <br><br>}<br><br>,N$

where N is a number of captured images used for generating the calibration parameter and Δ

L_i is an offset between an actual pen tip location in an ith frame and L_pentip.

3. The method of claim 2, further comprising:

initializing {overscore (L)}_{virtual-pentip as (}0,

     0), where {overscore (L)}_{virtual-pentip} is an estimated value of L_{virtual-pentip}.

4. The method of claim 3, further comprising:

in the equation

$L_{\mathrm{pentip}}^i=F_{S\to <br><br>P}^i·<br><br>L_{\mathrm{virtual}-\mathrm{pentip}},i=1,2,\mathrm{\cdots <br><br>}<br><br>,N,$

setting L_{virtual-pentip} as {overscore (L)}_{virtual-pentip} to get L_pentipⁱ, i=1,2, . . . , N and, by averaging L_pentipⁱ, estimating L_pentip as;

${\stackrel{\_}{L}}_{\mathrm{pentip}}=\frac{1}{N}<br><br>\underset{i=1}{\overset{i=N}{\sum <br><br>}}<br><br>L_{\mathrm{pentip}}^i.$

5. The method of claim 4, further comprising:

in the equation

$L_{\mathrm{virtual}-\mathrm{pentip}}^i=F_{P\to <br><br>S}^i·<br><br>\left(L_{\mathrm{pentip}}+\mathrm{\Delta <br><br>}<br><br><br><br>L_i\right),i=1,2,\mathrm{\cdots <br><br>}<br><br>,N,$

setting L_pentip as {overscore (L)}_pentip to get L_{virtual-pentip}ⁱ, i=1,2, . . . ,N and, by averaging L_{virtual-pentip}, estimating L_{virtual-pentip} as;

${\stackrel{\_}{L}}_{\mathrm{virtual}-\mathrm{pentip}}=\frac{1}{N}<br><br>\underset{i=1}{\overset{i=N}{\sum <br><br>}}<br><br>L_{\mathrm{virtual}-\mathrm{pentip}}^i.$

6. The method of claim 5, further comprising:

repeating the steps of claims 4 and 5 a plurality of times such that the estimate of L_{virtual-pentip} converges to a substantially more accurate result

$L_{\mathrm{virtual}-\mathrm{pentip}}^{\mathrm{accurate}}$

and the estimate of L_pentip converges to a substantially more accurate result

$L_{\mathrm{pentip}}^{\mathrm{accurate}}.$

7. The method of claim 6, further comprising:

using

$L_{\mathrm{virtual}-\mathrm{pentip}}^{\mathrm{accurate}}$

as the calibration parameter for mapping the location of the center of the camera-captured image to the location of the pen tip.

8. The method of claim 1, further comprising:

using a virtual pen tip to map a recovered camera-captured stroke to a recovered pen-tip stroke, wherein a location of the virtual pen-tip on an image-sensing plane of the digital pen depends on a predetermined relationship between the tip of the pen and the pen'"'"'s camera, wherein the predetermined relationship is based on the configuration of the pen.

9. A computer-readable medium containing computer-executable instructions that generate a calibration parameter for a digital pen, which includes a pen tip and a camera, wherein mapping, the location of the image center to the location of the tip of the pen on a positionally encoded surface, depends on the calibration parameter, by performing steps comprising:

generating the calibration parameter based on a plurality of images captured by the digital pen'"'"'s camera by placing the pen in a plurality of different orientations while keeping the pen tip at a fixed location L_pentip on the positionally encoded surface, wherein the calibration parameter is generated based on a plurality of iteratively generated estimates of the calibration parameter.

10. The computer-readable medium of claim 9, containing further computer-executable instructions that calculate a transform F_S→

P and an inverse transform

$F_{P\to <br><br>S}=F_{S\to <br><br>P}^{-1},$

by image matching, for each image of the plurality of captured images and that calculate;

$L_{\mathrm{pentip}}^i=F_{S\to <br><br>P}^i·<br><br>L_{\mathrm{virtual}-\mathrm{pentip}},i=1,2,\mathrm{\cdots <br><br>}<br><br>,N,\mathrm{and}$ $L_{\mathrm{virtual}-\mathrm{pentip}}^i=F_{P\to <br><br>S}^i·<br><br>\left(L_{\mathrm{pentip}}+\mathrm{\Delta <br><br>}<br><br><br><br>L_i\right),i=1,2,\mathrm{\cdots <br><br>}<br><br>,N$

where N is a number of captured images used for generating the calibration parameter and Δ

L_i is an offset between an actual pen tip location in an ith frame and L_pentip.

11. The computer-readable medium of claim 10, containing further computer-executable instructions that initialize {overscore (L)}_{virtual-pentip} as (0, 0), where {overscore (L)}_{virtual-pentip} is an estimated value of L_{virtual-pentip}.

12. The computer-readable medium of claim 11, containing further computer-executable instructions for, in the equation

$L_{\mathrm{pentip}}^i=F_{S\to }^{}$

P i ·

L virtual - pentip , i = 1 , 2 , ⋯

⁢

, N ,

setting L_{virtual-pentip} as {overscore (L)}_{virtual-pentip} to get L_pentipⁱ, i=1,2, . . . ,N and, by averaging L_pentipⁱ, estimating L_pentip as;

${\stackrel{\_}{L}}_{\mathrm{pentip}}=\frac{1}{N}<br><br>\underset{i=1}{\overset{i=N}{\sum <br><br>}}<br><br>L_{\mathrm{pentip}}^i.$

13. The computer-readable medium of claim 12, containing further computer-executable instructions for, in the equation

$L_{\mathrm{virtual}-\mathrm{pentip}}^i=F_{P\to }^{}$

S i ·

( L pentip + Δ

⁢

⁢

L i ) , i = 1 , 2 , ⋯

⁢

, N ,

setting L_pentip as {overscore (L)}_pentip to get L_{virtual-pentip}ⁱ, i=1,2, . . . ,N and, by averaging L_{virtual-pentip}ⁱ, estimating L_{virtual-pentip} as;

${\stackrel{\_}{L}}_{\mathrm{virtual}-\mathrm{pentip}}=\frac{1}{N}<br><br>\underset{i=1}{\overset{i=N}{\sum <br><br>}}<br><br>L_{\mathrm{virtual}-\mathrm{pentip}}^i.$

14. The computer-readable medium of claim 13, containing further computer-executable instructions for repeating the calculations of claims 12 and 13 a plurality of times such that the estimate of L_{virtual-pentip} converges to a substantially more accurate result

$L_{\mathrm{virtual}-\mathrm{pentip}}^{\mathrm{accurate}}$

and the estimate of L_pentip converges to a substantially more accurate result

$L_{\mathrm{pentip}}^{\mathrm{accurate}}.$

15. The computer-readable medium of claim 14, containing further computer-executable instructions for using

$L_{\mathrm{virtual}-\mathrm{pentip}}^{\mathrm{accurate}}$

as the calibration parameter L_{virtual-pentip} for mapping the location of the image center to the location of the tip of the pen.

16. A computer-readable medium containing computer-executable instructions that calibrate a digital pen, which includes a pen tip and a camera, and that map a location of a center of a camera-captured image to a location of the pen tip, by performing steps comprising:

generating a calibration parameter based on a plurality of images captured by the digital pen'"'"'s camera by placing the pen in a plurality of different orientations while keeping the pen tip at a substantially fixed location L_pentip on a positionally encoded surface, wherein the calibration parameter is generated based on a plurality of iteratively generated estimates of the calibration parameter; and

using the calibration parameter to map the location of the center of the camera-captured image to the location of the pen tip so that a stroke written by a user on the positionally encoded surface can be recovered at a location within an electronic copy of the positionally encoded surface, wherein the position of the recovered stroke within the electronic copy corresponds to a plurality of locations of the pen tip as the user was writing the stroke on the positionally encoded surface.

17. The computer-readable medium of claim 16, containing further computer-executable instructions for using a virtual pen tip to map a recovered camera-captured stroke to a recovered pen-tip stroke, wherein a location of the virtual pen-tip on an image-sensing plane of the digital pen depends on a predetermined relationship between the tip of the pen and the pen'"'"'s camera, wherein the predetermined relationship is based on the configuration of the pen.

18. A system that calibrates a digital pen, which includes a pen tip and a camera, and that maps a location of a center of a camera-captured image to a location of the pen tip, the system comprising:

a calibration module that generates a calibration parameter based on a plurality of images captured by the digital pen'"'"'s camera with the pen in a plurality of different orientations while the pen tip is at a substantially fixed location L_pentip on the positionally encoded surface, wherein the calibration parameter is generated based on a plurality of iteratively generated estimates of the calibration parameter; and

a mapping module that uses the calibration parameter to map the location of the center of the camera-captured image to the location of the pen tip so that a stroke written by a user on the positionally encoded surface can be recovered at a location within an electronic copy of the positionally encoded surface, wherein the position of the recovered stroke within the electronic copy corresponds to a plurality of locations of the pen tip as the user was writing the stroke on the positionally encoded surface.

19. The system of claim 18, wherein the calibration module calculates a transform F_S→

P and an inverse transform

$F_{P\to <br><br>S}=F_{S\to <br><br>P}^{-1},$

by image matching, for each image of the plurality of captured images and then calculates;

$L_{\mathrm{pentip}}^i=F_{S\to <br><br>P}^i·<br><br>L_{\mathrm{virtual}-\mathrm{pentip}},i=1,2,\mathrm{\cdots <br><br>}<br><br>,N,\mathrm{and}$ $L_{\mathrm{virtual}-\mathrm{pentip}}^i=F_{P\to <br><br>S}^i·<br><br>\left(L_{\mathrm{pentip}}+\mathrm{\Delta <br><br>}<br><br><br><br>L_i\right),i-1,2,\mathrm{\cdots <br><br>}<br><br>,N$

where N is a number of captured images used for generating the calibration parameter and Δ

L_i, an offset between an actual pen tip location in an ith frame and L_pentip.

20. The system of claim 19, wherein the calibration module initializes {overscore (L)}_{virtual-pentip} as (0, 0), where {overscore (L)}_{virtual-pentip} is an estimated value of L_{virtual-pentip}.

21. The system of claim 20 wherein, in the equation

$\begin{array}{c}{L}_{\mathrm{pentip}}^i=\end{array}$

F S ->

P i ·

L virtual - pentip , i = ⁢

1 , 2 , …

⁢

, N , L virtual - pentip

is set as {overscore (L)}_{virtual-pentip} to getL_pentipⁱ, i=1,2, . . . ,N and, L_pentip is estimated, by averaging L_pentipⁱ, as;

${\stackrel{\_}{L}}_{\mathrm{pentip}}=\frac{1}{N}<br><br>\underset{i=1}{\overset{i=N}{\sum <br><br>}}<br><br><br><br>L_{\mathrm{pentip}}^i.$

22. The system of claim 21, wherein, in the equation

$L_{\mathrm{virtual}-\mathrm{pentip}}^i=F_{P->}^{}$

S i ·

( L pentip + Δ

⁢

⁢

L i ) , i = 1 , 2 , …

⁢

, ⁢

N , L pentip

is set as {overscore (L)}_pentip to get L_{virtual-pentip}ⁱ, i=1,2, . . . ,N and L_{virtual pentip} is estimated, by averaging L_{virtual-pentip}ⁱ as;

${\stackrel{\_}{L}}_{\mathrm{virtual}-\mathrm{pentip}}=\frac{1}{N}<br><br>\underset{i=1}{\overset{i=N}{\sum <br><br>}}<br><br><br><br>L_{\mathrm{virtual}-\mathrm{pentip}}^i.$

23. The system of claim 22, wherein, the calibration module repeats the calculations of claims 21 and 22 a plurality of times such that the estimate Of L_{virtual-pentip}, converges to a substantially more accurate result

$L_{\mathrm{virtual}-\mathrm{pentip}}^{\mathrm{accurate}}$

and the estimate of L_pentip converges to a substantially more accurate result

$L_{\mathrm{pentip}}^{\mathrm{accurate}}.$

24. The system of claim 23, wherein, the calibration module outputs

$L_{\mathrm{virtual}-\mathrm{pentip}}^{\mathrm{accurate}}$

from the calibration module as the calibration parameter for use by the mapping module to map the recovered image-center stroke to the recovered pen-tip stroke.

25. The system of claim 18, wherein the mapping module uses a virtual pen tip to map a recovered camera-captured stroke to a recovered pen-tip stroke, wherein a location of the virtual pen-tip on an image-sensing plane of the digital pen depends on a predetermined relationship between the tip of the pen and the pen'"'"'s camera, wherein the predetermined relationship is based on the configuration of the pen.