System and method for transforming an ordinary computer monitor into a touch screen

• US 20040207600A1
• Filed: 04/30/2004
• Published: 10/21/2004
• Est. Priority Date: 10/24/2000
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Zhang, Zhengyou, Shan, Ying
• Granted Patent: US 7,342,572 B2
• Time in Patent Office: Days
• US Class Current: 345/156
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386

Abstract

A system and method for turning a regular computer monitor screen into a touch screen using an ordinary camera. It includes an image-screen mapping procedure to correct for the non-flatness of the computer screen. It also includes a segmentation method to distinguish the foreground, for example an indicator such as a finger, from the background of a computer screen. Additionally, this system and method includes a robust technique of finding the tip point location of the indicator (such as the finger tip). The screen coordinates of the tip point are then used to control the position of the system indicator.

Claims

1. A computer-implemented process for transforming a computer screen into a touch screen using a camera, comprising using a computer to perform the following process actions:

inputting images of the computer screen;

computing a projective mapping between the images and the screen which is capable of determining the screen coordinates of a point in the images;

locating a tip of an indicator in said images, said indicator being superimposed on the screen in the image; and

determining the position of the tip of the indicator on the screen from the images of the screen using said projective mapping.

2. The process of claim 1, wherein the process action of computing a projective mapping between the image coordinates and the screen coordinates, comprises an action of determining a homography from at least four pairs of image-screen correspondences.

3. The process of claim 2, wherein the process action of determining a homography from at least four pairs of image-screen correspondences comprises a process action of having a user choose the screen coordinates as the four corners of the screen and selecting corresponding points on a displayed version of the image of the screen.

4. The process of claim 2, wherein the process action of determining a homography from at least four pairs of image-screen correspondences comprises a process action of having a user choosing the screen coordinates as the four corners of the screen and detecting corresponding points on an image of the screen via an edge detection procedure.

5. The process of claim 1, wherein the process action of locating the tip of the indicator in the inputted images, comprises the actions of:

inputting an image of the computer screen lacking a depiction of an indicator pointed at the screen;

extracting a model of the screen, referred to as a background model, from the input image lacking the depiction of the indicator, said background model being indicative of the screen when no indicator is pointed at it;

inputting a second image of the screen which depicts an indicator pointed at the screen;

extracting a model of the indicator, referred to as a foreground model, from the second image;

inputting one or more subsequently-captured images of the screen; and

for each subsequently-captured image,

employing the background and foreground models to identify pixels of the subsequently-captured image that correspond to the indicator,

finding a location in the subsequently-captured image which corresponds to the tip of the indicator among the identified pixels.

6. The process of claim 5 wherein the process action of extracting a background model of the screen comprises:

capturing an image of the screen while it displays colors typical of the screen images used;

histogramming all pixels in the image on the basis of pixel color intensity; and

calculating a Gaussian distribution for the histogram to determine the mean pixel intensity and the variance therefrom.

7. The process of claim 5 wherein the process action of extracting a foreground model further comprises:

displaying an area on the screen;

having a user point to a said displayed area using said indicator;

capturing an image of the screen while the user is pointing to said displayed area;

histogramming all pixels of the captured image in the displayed area on the basis of pixel color intensity; and

calculating a Gaussian distribution for the histogram to determine the mean pixel intensity and the variance therefrom.

8. The process of claim 5 wherein the process action of employing the background and foreground models to identify pixels of the subsequently-captured image that correspond to the indicator comprises the process actions of:

for each pixel,

determining the probability it is a background pixel using the background model;

determining the probability it is a foreground pixel using the foreground model;

dividing the probability it is a background pixel by the probability it is a foreground pixel and if this number is greater than one designating the pixel as a background pixel, and if this number is less than one designating this pixel as a foreground pixel.

9. The process of claim 1 wherein the process action of determining the position of the tip comprises an action for correcting for the curvature of the screen.

12. The process of claim 1 wherein the action of finding the location of the indicator comprises the actions of:

calculating a cumulative total number of pixels that belong to the foreground on a scan line by scan line basis starting from the top of the captured image;

analyzing the number of pixels representing foreground pixels in each scan line to identify the scan line where the foreground pixels first appear and increase in cumulative total thereafter, said identified scan line being designated as the preliminary tip location scan line;

scanning a prescribed number of lines above and below the identified tip location scan line to find the start and end of the foreground pixels in the horizontal direction;

fitting a line through the center point of each series of foreground pixels along each of the scan lines;

scanning all pixels within the prescribed number of lines to find the indicator boundary pixels; and

designating the location of the indicator tip as the location where the previously determined centerline intersects a boundary pixel of the indicator in the image.

13. The process of claim 12 further comprising the process action of using a Kalman filter to refine indicator tip location.

14. The process of claim 12 wherein the process action of determining the position of the tip of the indicator on the screen from the images of the screen comprises an action of mapping the location of the pixel of the image corresponding to the indicator tip to the corresponding screen coordinates using the projective mapping.

23. The process of claim 1 further comprising a process action for using the determined position of the tip of the indicator to control the position of a system indicator.

30. The computer-implemented process of claim 1 further comprising the process action of using the screen coordinates of the tip of the indicator to control the position of a system indicator.

10. Cancelled.

11. Cancelled.

15. A system for transforming a computer screen into a touch screen using a camera, the system comprising:

a camera positioned so as to have an unobstructed view of the computer screen;

a general purpose computing device; and

a computer program comprising program modules executable by the computing device, wherein the computing device is directed by the program modules of the computer program to,

input images of the computer screen from the camera;

compute a projective mapping between the images and the screen to determine the screen coordinates of a point in the images;

locate an indicator tip superimposed on the screen in the images; and

determine the position of the indicator on the screen from the images of the screen using said projective mapping.

16. The system of claim 15, wherein the program module for locating the tip of the indicator in the inputted images, comprises sub-modules for:

inputting an image of the computer screen lacking a depiction of an indicator pointed at the screen;

extracting a background model of the screen from the input image lacking the depiction of the indicator, said background model being indicative of the screen when no indicator is pointed at it;

inputting a second image of the screen which depicts an indicator pointed at the screen;

extracting a foreground model, from the second image;

inputting one or more subsequently-captured images of the screen; and

for each subsequently-captured image,

employing the background and foreground models to identify pixels of the subsequently-captured image that correspond to the indicator,

finding a location in the subsequently-captured image which corresponds to the tip of the indicator among the identified pixels.

17. The system of claim 16 wherein the sub-module for extracting a background model of the screen further comprises sub-modules for:

capturing an image of the screen while it displays colors typical of the screen images used;

histogramming all pixels in the image on the basis of pixel color intensity for each of the red, green and blue color channels to produce a background histrogram for each channel; and

calculating a Gaussian distribution for each background histogram to determine the mean pixel intensity and the variance therefrom for each color channel.

18. The system of claim 16 wherein the sub-module for extracting a foreground model of the screen further comprises sub-modules for:

displaying an area on the screen having a different color from the rest of the screen;

having a user point to a said displayed area using said indicator;

capturing an image of the screen while the user is pointing to said displayed area;

histogramming all pixels of the captured image in the displayed area on the basis of pixel color intensity for each of the red, green and blue color channels to produce a foreground histogram for each channel; and

calculating a Gaussian distribution for each foreground histogram to determine the mean pixel intensity and the variance therefrom for each color channel.

19. The system of claim 16 wherein the sub-modules for employing the background model and foreground models to identify pixels of the subsequently-captured image that correspond to the indicator, comprises sub-modules for:

for each pixel,

determining the probability it is a background pixel for each channel using the background model;

determining the probability it is a foreground pixel for each channel using the foreground model;

multiplying the probabilities it is a background pixel for each red, green and blue channel to determine the combined probability a pixel is a background pixel;

multiplying the probabilities it is a foreground pixel for each red, green and blue channel to determine the combined probability a pixel is a background pixel;

dividing the combined probability it is a background pixel by the combined probability it is a foreground pixel and if this number is greater than one designate the pixel as a background pixel, and if this number is less than one designate this pixel as a foreground pixel.

20. The system of claim 15 wherein the camera has an unobstructed view of the computer screen and is skewed no more than about 30 degrees from a normal projecting from the center of the screen.

21. The system of claim 15 wherein the camera is positioned to have an unobstructed view of the whole computer screen and a minimum non-screen background.

22. The system of claim 15 further comprising a pointing device capable of controlling the location of a system indicator on a computer monitor screen of the computing device, and wherein said computer program further comprises a program module for disabling the ability of the pointing device to control the system indicator whenever the indicator is found to be superimposed on the screen.

31. The system of claim 15 further comprising a program module to use the position of the indicator to control the position of a system indicator.

24. A computer-readable medium having computer-executable instructions for transforming a computer screen into a touch screen using a camera, said computer-executable instructions comprising:

inputting images of the computer screen;

computing a homography which is capable of determining the screen coordinates of a point in the images, said homography being computed from at least four pairs of image-screen point correspondences;

locating a tip of an indicator in said images, said indicator being superimposed on the screen in the image; and

determining the position of the tip of the indicator on the screen from the images of the screen using said homography.

25. The computer-readable medium of claim 24 further comprising an instruction for refining the determined position of the tip of the indicator on the screen by correcting for errors in the homography.

26. The computer-readable medium of claim 25, wherein the instruction for refining the determined position of the tip of the indicator comprises instruction sub-modules for:

sequentially displaying a set of original calibration points on the screen which have known screen coordinates and which together form a grid;

capturing each of the calibration points in a separate image;

for each displayed original calibration point,

mapping said original calibration point as depicted in the image to its corresponding screen coordinates using the homography to derive an estimated calibration point,

comparing the estimated calibration point with the original calibration point to define a residual vector;

using bilinear interpolation to compute residual vectors for all screen points not associated with a displayed calibration point to create a residual vector field;

identifying the rough location of the indicator tip using the projective mapping; and

refining the location of the indicator tip by applying the residual vector associated with the rough location.

27. The computer-readable medium of claim 26 wherein the sub-module for sequentially displaying the set of original calibration points on the screen comprises sequentially displaying a set of circular dots the center of which correspond to the screen coordinates of the original calibration point, wherein said dot appears as an ellipse in the image of the dot;

and wherein

the sub-module for mapping the original calibration point as depicted in the image to its corresponding screen coordinates comprises using the centroid of the ellipse associated with the calibration point as the calibration point depicted in the image.

32. The system of claim 24 further comprising a computer-executable instruction comprising using the screen coordinates to control the position of a system indicator.

28. Cancelled.

29. Cancelled.

33. A system for transforming a computer screen into a touch screen, the system comprising:

a camera positioned so as to have an view of a computer screen;

a general purpose computing device; and

a computer program comprising program modules executable by the computing device, wherein the computing device is directed by the program modules of the computer program to,

establish a mapping between images taken by the camera of the indicator superimposed on the computer screen;

extract a background model from the images;

extract a foreground model of an indicator from the images;

locate the tip of the indicator in screen coordinates using said background and foreground models.

34. The system of claim 33 further comprising a program module to use the screen coordinates of the tip of the indicator to control the position of a system indicator.

35. The system of claim 34 wherein the system indicator is a cursor.

36. A process for transforming a computer screen into a touch screen, comprising the process actions of:

positioning a camera so as to capture images of a computer screen with an indicator superimposed thereon;

establishing a mapping between the images and the screen;

locating a tip of an indicator in said images;

determining the position of the tip of the indicator on the screen from the images of the screen using said mapping; and

using said position of the tip of the indicator to control computer actions.