System and method for transforming an ordinary computer monitor into a touch screen
First Claim
1. A computer-implemented process for transforming a computer screen into a touch screen using a camera, comprising using a computer to perform the following process actions:
- inputting images of the computer screen taken by a camera positioned so as to have a frontal view of the computer screen;
computing a projective mapping between the images and the screen which is capable of determining the screen coordinates of a point in the images;
locating a tip of an indicator in said images, said indicator being superimposed on the screen in the image; and
determining the position of the tip of the indicator on the screen from the images of the screen using said projective mapping, wherein the process action of locating the tip of the indicator in the inputted images, comprises the actions of;
inputting an image of the computer screen lacking a depiction of an indicator pointed at the screen;
extracting a model of the screen, referred to as a background model, from the input image lacking the depiction of the indicator, said background model being indicative of the screen when no indicator is pointed at it;
inputting a second image of the screen which depicts an indicator pointed at the screen;
extracting a model of the indicator, referred to as a foreground model, from the second image;
inputting one or more subsequently-captured images of the screen; and
for each subsequently-captured image,employing the background and foreground models to identify pixels of the subsequently-captured image that correspond to the indicator,finding a location in the subsequently-captured image which corresponds to the tip of the indicator among the identified pixels.
1 Assignment
0 Petitions
Accused Products
Abstract
A system and method for turning a regular computer monitor screen into a touch screen using an ordinary camera. It includes an image-screen mapping procedure to correct for the non-flatness of the computer screen. It also includes a segmentation method to distinguish the foreground, for example an indicator such as a finger, from the background of a computer screen. Additionally, this system and method includes a robust technique of finding the tip point location of the indicator (such as the finger tip). The screen coordinates of the tip point are then used to control the position of the system indicator.
-
Citations
21 Claims
-
1. A computer-implemented process for transforming a computer screen into a touch screen using a camera, comprising using a computer to perform the following process actions:
-
inputting images of the computer screen taken by a camera positioned so as to have a frontal view of the computer screen; computing a projective mapping between the images and the screen which is capable of determining the screen coordinates of a point in the images; locating a tip of an indicator in said images, said indicator being superimposed on the screen in the image; and determining the position of the tip of the indicator on the screen from the images of the screen using said projective mapping, wherein the process action of locating the tip of the indicator in the inputted images, comprises the actions of; inputting an image of the computer screen lacking a depiction of an indicator pointed at the screen; extracting a model of the screen, referred to as a background model, from the input image lacking the depiction of the indicator, said background model being indicative of the screen when no indicator is pointed at it; inputting a second image of the screen which depicts an indicator pointed at the screen; extracting a model of the indicator, referred to as a foreground model, from the second image; inputting one or more subsequently-captured images of the screen; and for each subsequently-captured image, employing the background and foreground models to identify pixels of the subsequently-captured image that correspond to the indicator, finding a location in the subsequently-captured image which corresponds to the tip of the indicator among the identified pixels. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8, 9, 10)
-
-
11. A computer-implemented process for transforming a computer screen into a touch screen using a camera, comprising using a computer to perform the following process actions:
-
inputting images of the computer screen taken by a camera positioned so as to have a frontal view of the computer screen; computing a projective mapping between the images and the screen which is capable of determining the screen coordinates of a point in the images; locating a tip of an indicator in said images, said indicator being superimposed on the screen in the image; and determining the position of the tip of the indicator on the screen from the images of the screen using said projective mapping, wherein the action of finding the location of the indicator comprises the actions of; calculating a cumulative total number of pixels that belong to the foreground on a scan line by scan line basis starting from the top of the captured image; analyzing the number of pixels representing foreground pixels in each scan line to identify the scan line where the foreground pixels first appear and increase in cumulative total thereafter, said identified scan line being designated as the preliminary tip location scan line; scanning a prescribed number of lines above and below the identified tip location scan line to find the start and end of the foreground pixels in the horizontal direction; fitting a line through the center point of each series of foreground pixels along each of the scan lines; scanning all pixels within the prescribed number of lines to find the indicator boundary pixels; and designating the location of the indicator tip as the location where the previously determined centerline intersects a boundary pixel of the indicator in the image. - View Dependent Claims (12, 13)
-
-
14. A system for transforming a computer screen into a touch screen using a camera, the system comprising:
-
a camera positioned so as to have an unobstructed frontal view of the computer screen; a general purpose computing device; and a computer program comprising program modules executable by the computing device, wherein the computing device is directed by the program modules of the computer program to, input images of the computer screen from the camera positioned so as to have a frontal view of the computer screen; compute a projective mapping between the images and the screen to determine the screen coordinates of a point in the images; locate an indicator tip superimposed on the screen in the images, wherein the program module for locating the indicator tip in the inputted images, comprises sub-modules for; inputting an image of the computer screen lacking a depiction of an indicator pointed at the screen; extracting a background model of the screen from the input image lacking the depiction of the indicator, said background model being indicative of the screen when no indicator is pointed at it; inputting a second image of the screen which depicts an indicator pointed at the screen; extracting a foreground model, from the second image; inputting one or more subsequently-captured images of the screen; and for each subsequently-captured image, employing the background and foreground models to identify pixels of the subsequently-captured image that correspond to the indicator, finding a location in the subsequently-captured image which corresponds to the tip of the indicator among the identified pixel; and determine the position of the indicator on the screen from the images of the screen using said projective mapping. - View Dependent Claims (15, 16, 17, 18, 19, 20, 21)
-
Specification