System and method for transforming an ordinary computer monitor into a touch screen

• US 7,342,572 B2
• Filed: 04/30/2004
• Issued: 03/11/2008
• Est. Priority Date: 10/24/2000
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Zhang, Zhengyou, Shan, Ying
• Primary Examiner(s): Hjerpe; Richard
• Assistant Examiner(s): Nguyen; Jennifer T
• Application Number:
  US10/836,798
• Publication Number:
  US 20040207600A1
• Time in Patent Office: 1,411 Days
• Field of Search: 345/156, 345173-180, 345/158, 345/169
• US Class Current: 345/173
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386

Abstract

A system and method for turning a regular computer monitor screen into a touch screen using an ordinary camera. It includes an image-screen mapping procedure to correct for the non-flatness of the computer screen. It also includes a segmentation method to distinguish the foreground, for example an indicator such as a finger, from the background of a computer screen. Additionally, this system and method includes a robust technique of finding the tip point location of the indicator (such as the finger tip). The screen coordinates of the tip point are then used to control the position of the system indicator.

Claims

1. A computer-implemented process for transforming a computer screen into a touch screen using a camera, comprising using a computer to perform the following process actions:

inputting images of the computer screen taken by a camera positioned so as to have a frontal view of the computer screen;

computing a projective mapping between the images and the screen which is capable of determining the screen coordinates of a point in the images;

locating a tip of an indicator in said images, said indicator being superimposed on the screen in the image; and

determining the position of the tip of the indicator on the screen from the images of the screen using said projective mapping, wherein the process action of locating the tip of the indicator in the inputted images, comprises the actions of;

inputting an image of the computer screen lacking a depiction of an indicator pointed at the screen;

extracting a model of the screen, referred to as a background model, from the input image lacking the depiction of the indicator, said background model being indicative of the screen when no indicator is pointed at it;

inputting a second image of the screen which depicts an indicator pointed at the screen;

extracting a model of the indicator, referred to as a foreground model, from the second image;

inputting one or more subsequently-captured images of the screen; and

for each subsequently-captured image,

employing the background and foreground models to identify pixels of the subsequently-captured image that correspond to the indicator,

finding a location in the subsequently-captured image which corresponds to the tip of the indicator among the identified pixels.

2. The process of claim 1, wherein the process action of computing a projective mapping between the image coordinates and the screen coordinates, comprises an action of determining a homography from at least four pairs of image-screen correspondences.

3. The process of claim 2, wherein the process action of determining a homography from at least four pairs of image-screen correspondences comprises a process action of having a user choose the screen coordinates as the four corners of the screen and selecting corresponding points on a displayed version of the image of the screen.

4. The process of claim 2, wherein the process action of determining a homography from at least four pairs of image-screen correspondences comprises a process action of having a user choosing the screen coordinates as the four corners of the screen and detecting corresponding points on an image of the screen via an edge detection procedure.

5. The process of claim 1 wherein the process action of extracting a background model of the screen comprises:

capturing an image of the screen while it displays colors typical of the screen images used;

histogramming all pixels in the image on the basis of pixel color intensity; and

calculating a Gaussian distribution for the histogram to determine the mean pixel intensity and the variance therefrom.

6. The process of claim 1 wherein the process action of extracting a foreground model further comprises:

displaying an area on the screen;

having a user point to a said displayed area using said indicator;

capturing an image of the screen while the user is pointing to said displayed area;

histogramming all pixels of the captured image in the displayed area on the basis of pixel color intensity; and

calculating a Gaussian distribution for the histogram to determine the mean pixel intensity and the variance therefrom.

7. The process of claim 1 wherein the process action of employing the background and foreground models to identify pixels of the subsequently-captured image that correspond to the indicator comprises the process actions of:

for each pixel,

determining the probability it is a background pixel using the background model;

determining the probability it is a foreground pixel using the foreground model;

dividing the probability it is a background pixel by the probability it is a foreground pixel and if this number is greater than one designating the pixel as a background pixel, and if this number is less than one designating this pixel as a foreground pixel.

8. The process of claim 1 wherein the process action of determining the position of the tip comprises an action for correcting for the curvature of the screen.

9. The process of claim 1 further comprising a process action for using the determined position of the tip of the indicator to control the position of a system indicator.

10. The computer-implemented process of claim 1 further comprising the process action of using the screen coordinates of the tip of the indicator to control the position of a system indicator.

11. A computer-implemented process for transforming a computer screen into a touch screen using a camera, comprising using a computer to perform the following process actions:

inputting images of the computer screen taken by a camera positioned so as to have a frontal view of the computer screen;

computing a projective mapping between the images and the screen which is capable of determining the screen coordinates of a point in the images;

locating a tip of an indicator in said images, said indicator being superimposed on the screen in the image; and

determining the position of the tip of the indicator on the screen from the images of the screen using said projective mapping, wherein the action of finding the location of the indicator comprises the actions of;

calculating a cumulative total number of pixels that belong to the foreground on a scan line by scan line basis starting from the top of the captured image;

analyzing the number of pixels representing foreground pixels in each scan line to identify the scan line where the foreground pixels first appear and increase in cumulative total thereafter, said identified scan line being designated as the preliminary tip location scan line;

scanning a prescribed number of lines above and below the identified tip location scan line to find the start and end of the foreground pixels in the horizontal direction;

fitting a line through the center point of each series of foreground pixels along each of the scan lines;

scanning all pixels within the prescribed number of lines to find the indicator boundary pixels; and

designating the location of the indicator tip as the location where the previously determined centerline intersects a boundary pixel of the indicator in the image.

12. The process of claim 11 further comprising the process action of using a Kalman filter to refine indicator tip location.

13. The process of claim 11 wherein the process action of determining the position of the tip of the indicator on the screen from the images of the screen comprises an action of mapping the location of the pixel of the image corresponding to the indicator tip to the corresponding screen coordinates using the projective mapping.

14. A system for transforming a computer screen into a touch screen using a camera, the system comprising:

a camera positioned so as to have an unobstructed frontal view of the computer screen;

a general purpose computing device; and

a computer program comprising program modules executable by the computing device, wherein the computing device is directed by the program modules of the computer program to,

input images of the computer screen from the camera positioned so as to have a frontal view of the computer screen;

compute a projective mapping between the images and the screen to determine the screen coordinates of a point in the images;

locate an indicator tip superimposed on the screen in the images, wherein the program module for locating the indicator tip in the inputted images, comprises sub-modules for;

inputting an image of the computer screen lacking a depiction of an indicator pointed at the screen;

extracting a background model of the screen from the input image lacking the depiction of the indicator, said background model being indicative of the screen when no indicator is pointed at it;

inputting a second image of the screen which depicts an indicator pointed at the screen;

extracting a foreground model, from the second image;

inputting one or more subsequently-captured images of the screen; and

for each subsequently-captured image,

employing the background and foreground models to identify pixels of the subsequently-captured image that correspond to the indicator,

finding a location in the subsequently-captured image which corresponds to the tip of the indicator among the identified pixel; and

determine the position of the indicator on the screen from the images of the screen using said projective mapping.

15. The system of claim 14 wherein the sub-module for extracting a background model of the screen further comprises sub-modules for:

capturing an image of the screen while it displays colors typical of the screen images used;

histogramming all pixels in the image on the basis of pixel color intensity for each of the red, green and blue color channels to produce a background histrogram for each channel; and

calculating a Gaussian distribution for each background histogram to determine the mean pixel intensity and the variance therefrom for each color channel.

16. The system of claim 14 wherein the sub-module for extracting a foreground model of the screen further comprises sub-modules for:

displaying an area on the screen having a different color from the rest of the screen;

having a user point to a said displayed area using said indicator;

capturing an image of the screen while the user is pointing to said displayed area;

histogramming all pixels of the captured image in the displayed area on the basis of pixel color intensity for each of the red, green and blue color channels to produce a foreground histogram for each channel; and

calculating a Gaussian distribution for each foreground histogram to determine the mean pixel intensity and the variance therefrom for each color channel.

17. The system of claim 14 wherein the sub-modules for employing the background model and foreground models to identify pixels of the subsequently-captured image that correspond to the indicator, comprises sub-modules for:

for each pixel,

determining the probability it is a background pixel for each channel using the background model;

determining the probability it is a foreground pixel for each channel using the foreground model;

multiplying the probabilities it is a background pixel for each red, green and blue channel to determine the combined probability a pixel is a background pixel;

multiplying the probabilities it is a foreground pixel for each red, green and blue channel to determine the combined probability a pixel is a background pixel;

dividing the combined probability it is a background pixel by the combined probability it is a foreground pixel and if this number is greater than one designate the pixel as a background pixel, and if this number is less than one designate this pixel as a foreground pixel.

18. The system of claim 14 wherein the camera has an unobstructed view of the computer screen and is skewed no more than about 30 degrees from a normal projecting from the center of the screen.

19. The system of claim 14 wherein the camera is positioned to have an unobstructed view of the whole computer screen and a minimum non-screen background.

20. The system of claim 14 further comprising a pointing device capable of controlling the location of a system indicator on a computer monitor screen of the computing device, and wherein said computer program further comprises a program module for disabling the ability of the pointing device to control the system indicator whenever the indicator is found to be superimposed on the screen.

21. The system of claim 14 further comprising a program module to use the position of the indicator to control the position of a system indicator.