System and method for transforming an ordinary computer monitor screen into a touch screen

• US 6,774,889 B1
• Filed: 10/24/2000
• Issued: 08/10/2004
• Est. Priority Date: 10/24/2000
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Zhang, Zhengyou, Shan, Ying
• Primary Examiner(s): Liang, Regina
• Assistant Examiner(s): Nguyen, Jennifer T.
• Application Number:
  US09/696,251
• Time in Patent Office: 1,386 Days
• Field of Search: 345/156, 345/157, 345/158, 345/9, 345/169, 345/173, 345/863, 345/865, 345/178, 345/181, 178/18.1, 382/168-172
• US Class Current: 345/173
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386

Abstract

A system and method for turning a regular computer monitor screen into a touch screen using an ordinary camera. It includes an image-screen mapping procedure to correct for the non-flatness of the computer screen. It also includes a segmentation method to distinguish the foreground, for example an indicator such as a finger, from the background of a computer screen. Additionally, this system and method includes a robust technique of finding the tip point location of the indicator (such as the finger tip). The screen coordinates of the tip point are then used to control the position of the system indicator.

Claims

1. A computer-implemented process for transforming a computer screen into a touch screen using a camera, comprising using a computer to perform the following process actions:

inputting images of the computer screen;

computing a projective mapping between the images and the screen which is capable of determining the screen coordinates of a point in the images, wherein the process action of computing a projective mapping between the image coordinates and the screen coordinates, comprises an action of determining a homography from at least four pairs of image-screen correspondences;

locating a tip of an indicator in said images, said indicator being superimposed on the screen in the image;

determining the position of the tip of the indicator on the screen from the images of the screen using said projective mapping; and

correcting for the curvature of the screen said connection comprising;

sequentially displaying a set of original calibration points on the screen which have known screen coordinates and which together form a grid;

capturing each of the calibration points in an image;

for each displayed original calibration point,

mapping said original calibration point as depicted in the image to its corresponding screen coordinates using the projective mapping to derive an estimated calibration point,

comparing the estimated calibration point with the original calibration point to define a residual vector;

using bilinear interpolation to compute residual vectors for all screen points not associated with a displayed calibration point to create a residual vector field;

identifying the rough location of the indicator tip using the projective mapping; and

refining the location of the indicator tip by applying the residual vector associated with the rough location.

2. The process of claim 1, wherein the process action of determining a homography from at least four pairs of image-screen correspondences comprises a process action of having a user choose the screen coordinates as the four corners of the screen and selecting corresponding points on a displayed version of the image of the screen.

3. The process of claim 1, wherein the process action of determining a homography from at least four pairs of image-screen correspondences comprises a process action of having a user choosing the screen coordinates as the four corners of the screen and detecting corresponding points on an image of the screen via an edge detection procedure.

4. The process of claim 1, wherein the process action of locating the tip of the indicator in the inputted images, comprises the actions of:

inputting an image of the computer screen lacking a depiction of an indicator pointed at the screen;

extracting a model of the screen, referred to as a background model, from the input image lacking the depiction of the indicator, said background model being indicative of the screen when no indicator is pointed at it;

inputting a second image of the screen which depicts an indicator pointed at the screen;

extracting a model of the indicator, referred to as a foreground model, from the second image;

inputting one or more subsequently-captured images of the screen; and

for each subsequently-captured image,

employing the background and foreground models to identify pixels of the subsequently-captured image that correspond to the indicator,

finding a location in the subsequently-captured image which corresponds to the tip of the indicator among the identified pixels.

5. The process of claim 4 wherein the process action of extracting a background model of the screen comprises:

capturing an image of the screen while it displays colors typical of the screen images used;

histogramming all pixels in the image on the basis of pixel color intensity; and

calculating a Gaussian distribution for the histogram to determine the mean pixel intensity and the variance therefrom.

6. The process of claim 4 wherein the process action of extracting a foreground model further comprises:

displaying an area on the screen;

having a user point to a said displayed area using said indicator;

capturing an image of the screen while the user is pointing to said displayed area;

histogramming all pixels of the captured image in the displayed area on the basis of pixel color intensity; and

calculating a Gaussian distribution for the histogram to determine the mean pixel intensity and the variance therefrom.

7. The process of claim 4 wherein the process action of employing the background and foreground models to identify pixels of the subsequently-captured image that correspond to the indicator comprises the process actions of:

for each pixel,

determining the probability it is a background pixel using the background model;

determining the probability it is a foreground pixel using the foreground model;

dividing the probability it is a background pixel by the probability it is a foreground pixel and if this number is greater than one designating the pixel as a background pixel, and if this number is less than one designating this pixel as a foreground pixel.

8. The process of claim 1 wherein the process action for sequentially displaying the set of original calibration points on the screen comprises sequentially displaying a set of circular dots, the centers of which correspond to the screen coordinates of each original calibration point, wherein each dot appears as an ellipse in the image of the dot;

and wherein

the process action for mapping the original calibration point as depicted in the image to its corresponding screen coordinates comprises using the centroid of the ellipse associated with the calibration point as the calibration point depicted in the image.

9. The process of claim 1 wherein the action of finding the location of the indicator comprises the actions of:

calculating a cumulative total number of pixels that belong to the foreground on a scan line by scan line basis starting from the top of the captured image;

analyzing the number of pixels representing foreground pixels in each scan line to identify the scan line where the foreground pixels first appear and increase in cumulative total thereafter, said identified scan line being designated as the preliminary tip location scan line;

scanning a prescribed number of lines above and below the identified tip location scan line to find the start and end of the foreground pixels in the horizontal direction;

fitting a line through the center point of each series of foreground pixels along each of the scan lines;

scanning all pixels within the prescribed number of lines to find the indicator boundary pixels; and

designating the location of the indicator tip as the location where the previously determined centerline intersects a boundary pixel of the indicator in the image.

10. The process of claim 9 further comprising the process action of using a Kalman filter to refine indicator tip location.

11. The process of claim 9 wherein the process action of determining the position of the tip of the indicator on the screen from the images of the screen comprises an action of mapping the location of the pixel of the image corresponding to the indicator tip to the corresponding screen coordinates using the projective mapping.

12. A computer-readable medium having computer-executable instructions for transforming a computer screen into a touch screen using a camera, said computer-executable instructions comprising:

inputting images of the computer screen;

computing a homography which is capable of determining the screen coordinates of a point in the images, said homography being computed from at least four pairs of image-screen point correspondences;

locating a tip of an indicator in said images, said indicator being superimposed on the screen in the image;

determining the position of the tip of the indicator on the screen from the images of the screen using said homography; and

refining the determined position of the tip of the indicator on the screen by correcting for errors in the homography said correcting comprising;

sequentially displaying a set of original calibration points on the screen which have known screen coordinates and which together form a grid;

capturing each of the calibration points in a separate image;

for each displayed original calibration point,

mapping said original calibration point as depicted in the image to its corresponding screen coordinates using the homography to derive an estimated calibration point,

comparing the estimated calibration point with the original calibration point to define a residual vector;

using bilinear interpolation to compute residual vectors for all screen points not associated with a displayed calibration point to create a residual vector field;

identifying the rough location of the indicator tip using the projective mapping; and

refining the location of the indicator tip by applying the residual vector associated with the rough location.

13. The computer-readable medium of claim 12 wherein the sub-module for sequentially displaying the set of original calibration points on the screen comprises sequentially displaying a set of circular dots the center of which correspond to the screen coordinates of the original calibration point, wherein said dot appears as an ellipse in the image of the dot;

and wherein

the sub-module for mapping the original calibration point as depicted in the image to its corresponding screen coordinates comprises using the centroid of the ellipse associated with the calibration point as the calibration point depicted in the image.

14. A computer-implemented process for compensating for the non-flatness of a computer monitor screen when computing screen coordinates corresponding to a point in an image of a screen, comprising using a computer to perform the following process actions:

computing a homography from the image of the screen which is capable of roughly determining the screen coordinates of a point in the image of the screen;

sequentially displaying a set of original calibration points on the screen which have known screen coordinates and which together form a grid;

for each displayed original calibration point,

mapping said original calibration point as depicted in the image to its corresponding screen coordinates using the homography to derive an estimated calibration point,

comparing the estimated calibration point with the original calibration point to define a residual vector; and

using the residual vectors to compensate for mapping errors cause by the non-flatness of the screen when computing screen coordinates corresponding to a point in an image of the screen.

15. The process of claim 14, wherein the process action of using the residual vectors to compensate for mapping errors comprises the process actions of:

using bilinear interpolation to compute residual vectors for all screen points of interest not associated with a displayed calibration point;

identifying the rough screen coordinates of a point in the image using the homography; and

refining the location of the screen coordinates of the point in the image by applying the residual vector associated with the rough screen coordinates of the point.