System and method for providing a mobile input device
First Claim
Patent Images
1. A computer-implemented process for using a quadrangular panel and a pointer overlaying the panel to control a remote display, comprising using a computer to perform the following process actions:
- inputting an image of a quadrangular panel and a pointer;
locating the position of the quadrangular panel in said image;
computing a projective mapping between the coordinates of said image and the coordinates of at least one remote display;
locating a tip of the pointer in said image; and
determining coordinates on the remote display corresponding to the location of the tip of the pointer on the quadrangular panel as depicted in the image using said projective mapping.
2 Assignments
0 Petitions
Accused Products
Abstract
A vision-based gesture interface system, which employs an arbitrary quadrangle-shaped panel and a pointer tip like a fingertip as an intuitive input device. Taking advantage of the panel, the system can fulfill many tasks such as controlling a remote and large display, and simulating a physical keyboard. Users can naturally use their fingers and other pointer tips to issue commands and type texts. The system is facilitated by accurately and reliably tracking the panel and the pointer tip and detecting clicking and dragging actions.
-
Citations
41 Claims
-
1. A computer-implemented process for using a quadrangular panel and a pointer overlaying the panel to control a remote display, comprising using a computer to perform the following process actions:
-
inputting an image of a quadrangular panel and a pointer;
locating the position of the quadrangular panel in said image;
computing a projective mapping between the coordinates of said image and the coordinates of at least one remote display;
locating a tip of the pointer in said image; and
determining coordinates on the remote display corresponding to the location of the tip of the pointer on the quadrangular panel as depicted in the image using said projective mapping. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28)
registering the panel location in a first image at time T=0 where T is the beginning of an application;
registering the panel location in a second image at some subsequent time T+1;
calculating the homography between the panel location in said first and second images at time T=0 and time T+1;
using the homography to map image pixels in the panel at time T=0 to time T=T+1 resulting in a virtual image that would be observed when there is no pointer overlaying the panel;
subtracting the virtual image from a third input image to give a difference image;
determining the areas with large color change in the difference image; and
searching for the tip pointer by fitting an elliptic curve to the edges.
-
-
7. The process of claim 2, wherein the process action of locating the position of the quadrangular panel in said image comprises the process actions of:
-
locating the four sides of the panel by scanning the image and determining the four lines corresponding to the largest gradients in color and intensity in the image of the panel over a prescribed threshold; and
locating the four corners of the panel in the image by specifying each corner as an intersection of two sides.
-
-
8. The process of claim 1, wherein the process action of computing a projective mapping between the coordinates of said image and the coordinates of at least one remote display comprises:
determining a homography from at least four pairs of image-remote display correspondences.
-
9. The process of claim 8, wherein the process action of determining a homography from at least four pairs of image-remote display correspondences comprises a process action of mapping the four corners of the panel in the image to the four corners of the at least one remote display.
-
10. The process of claim 9 wherein the process action of determining a homography from at least four pairs of image-remote display correspondences comprises a process action of mapping the four corners of the panel in the image to the four corners of an area less than the at least one remote display.
-
11. The process of claim 1 wherein the action of locating the pointer tip comprises the actions of:
-
scanning the image in a horizontal fashion to determine the pixel location where the color intensity of the panel changes significantly on a scan line by scan line basis starting from the top of the identified panel location in the image;
designating the location in each scanned line where the color intensity of the panel changes significantly as a pointer edge point;
fitting an elliptical curve through the pointer edge points;
determining a major axis of the elliptical curve; and
designating the location of the pointer tip as the location in the image where the major axis of the elliptical curve intersects the boundary of the elliptical curve.
-
-
12. The process of claim 11 wherein the action of locating the location of the pointer tip comprises the action of using previous pointer tip position data to limit the search for the new pointer tip location to within a prescribed area.
-
13. The process of claim 12 wherein the prescribed area is 30 by 30 pixels.
-
14. The process of claim 11 further comprising the process action of using a Kalman filter to refine indicator tip location.
-
15. The process of claim 11 wherein the process action of determining the position of the tip of the indicator on the panel from the images of the panel comprises an action of mapping the location of the pixel of the image corresponding to the indicator tip to the corresponding remote display coordinates using the projective mapping.
-
16. The process of claim 11 wherein the location of the pointer tip is defined in sub-pixels.
-
17. The process of claim 1 wherein the process action of determining coordinates on the remote display corresponding to the location of the tip of the pointer on the quadrangular panel as depicted in the image using said projective mapping further comprises the process action of determining the coordinates on more than one remote display.
-
18. The process of claim 1 further comprising the process actions of comparing the pointer tip location to a previous pointer tip location;
-
interpreting the pointer tip location; and
using the pointer tip position to control the position of a system indicator.
-
-
19. The process of claim 18 wherein the process action of interpreting the pointer tip location comprises interpreting the pointer tip location to simulate a mouse and wherein said system indicator is a mouse curser.
-
20. The process of claim 18 wherein the process action of interpreting the pointer tip location comprises the process action of interpreting the pointer tip location to simulate a keyboard and wherein said system indicator is a keyboard.
-
21. The process of claim 18 wherein the process action of interpreting the pointer tip location to simulate a mouse comprises the process actions of
simulating two mouse button pressing modes; - and
simulating two mouse motion types.
- and
-
22. The process of claim 21 wherein the process action of simulating two mouse button pressing modes comprises:
-
simulating a clicking mode which simulates a left mouse button down and then up automatically; and
simulating a dragging mode which simulates the left button down until released.
-
-
23. The process of claim 22 wherein the clicking mode is simulated by the following process actions:
-
designating a state variable, S, that maintains two states, up and down, to simulate the nature state of a button;
initializing the state variable S to up;
setting the state variable S to down when the pointer tip location has been at a fixed place for a prescribed first period of time to simulate mouse button depression; and
automatically setting the state variable to S to up after a second prescribed period of time to simulate mouse button release.
-
-
24. The process of claim 23 wherein the first prescribed period of time is 1 second and the second prescribed period of time is 0.1 second.
-
25. The process of claim 23 wherein the dragging mode is simulated by the following process actions:
-
designating a state variable, D, that maintains two states, up and down, to simulate the nature state of a button;
initializing the state variable D to the opposite of the state of variable S;
changing the state variable D to when the pointer tip location has been at a fixed place for a prescribed third period of time and wherein when a D-state change from up to down is detected a mouse button pressing action is simulated, and when a D-state change from down to up is detected a mouse button pressing releasing action is simulated.
-
-
26. The process of claim 25 wherein the third prescribed period of time is 1 second.
-
27. The process of claim 21 wherein the absolute mouse motion is simulated by mapping the panel to the whole at least one remote display.
-
28. The process of claim 21 wherein the relative mouse motion is determined by the process action of measuring the velocity of the pointer tip.
-
29. A system for using a quadrangular panel and a pointer overlaying the panel to control a remote display using a camera, the system comprising:
-
a camera positioned so as to have a view of the quadrangular panel;
a general purpose computing device; and
a computer program comprising program modules executable by the computing device, wherein the computing device is directed by the program modules of the computer program to, input images of the panel from the camera;
locate a quadrangular panel in said images;
compute a projective mapping between the images and the coordinates of at least remote display to determine the remote display coordinates of a point in the images;
locate a pointer tip superimposed on the panel in the images; and
determine the coordinates in the images on the remote display corresponding to the location of the tip of the pointer on the panel as depicted in the image using the projective mapping. - View Dependent Claims (30, 31, 32, 33)
locating the four sides of the panel by determining four lines corresponding to the four largest gradients in color and intensity in the image of the panel; and
locating the four corners of the panel in the image by specifying each corner as the location of the intersection of two lines.
-
-
33. The process of claim 29 further comprising a process action for using the determined position of the tip of the indicator to control the position of a system indicator.
-
34. A computer-readable medium having computer-executable instructions for a vision-based gesture interface to control a remote display using a camera, said computer-executable instructions comprising:
-
inputting an image of a quadrangular panel and a pointer;
locating the position of the quadrangular panel in said image;
computing a projective mapping between the coordinates of said image and the coordinates of at least one remote display;
locating a tip of the pointer in said image; and
determining coordinates on the remote display corresponding to the location of the tip of the pointer on the quadrangular panel as depicted in the image using said projective mapping.
-
-
35. A system for providing a mobile input device comprising:
-
a video input of a quadrangular-shaped panel;
a panel tracker which tracks said quadrangle-shaped panel by outputting the positions of its four corners;
a pointer tip tracker that identifies the location of a pointer on said panel;
an action/gesture detector that detects the location of the pointer on the panel;
and a remote display that is controlled by said panel tracker, said pointer tracker and said action detector. - View Dependent Claims (36, 37, 38, 39, 40, 41)
a curve fitter that fits an ellipse to the edge points of the pointer tip.
-
-
38. The system of claim 35 wherein said pointer tip tracker further comprises a color tracker.
-
39. The color tracker of claim 38 wherein the color tracker locates a potential pointer tip region that matches the predefined color tip model when no tip is being tracked.
-
40. The color tracker of claim 38 wherein the color tracker further comprises a predefined color tip model to help locate said tip in said image.
-
41. The color tracker of claim 38 wherein the color tracker further comprises a color tip model learned from previous pointer tip location data to help locate said tip in said image.
Specification