System and method for providing a mobile input device

• US 20030004678A1
• Filed: 06/18/2001
• Published: 01/02/2003
• Est. Priority Date: 06/18/2001
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Wu, Ying, Zhang, Zhengyou, Shan, Ying, Shafer, Steven A. N.
• Granted Patent: US 6,594,616 B2
• Time in Patent Office: Days
• US Class Current: 702/152
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: G06F 3/0426 tracking fingers with respect to a virtual keyboard projected or printed on the surface virtual keyboards on touch screens G06F3/04886
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text

Abstract

A vision-based gesture interface system, which employs an arbitrary quadrangle-shaped panel and a pointer tip like a fingertip as an intuitive input device. Taking advantage of the panel, the system can fulfill many tasks such as controlling a remote and large display, and simulating a physical keyboard. Users can naturally use their fingers and other pointer tips to issue commands and type texts. The system is facilitated by accurately and reliably tracking the panel and the pointer tip and detecting clicking and dragging actions.

Claims

1. A computer-implemented process for using a quadrangular panel and a pointer overlaying the panel to control a remote display, comprising using a computer to perform the following process actions:

inputting an image of a quadrangular panel and a pointer;

locating the position of the quadrangular panel in said image;

computing a projective mapping between the coordinates of said image and the coordinates of at least one remote display;

locating a tip of the pointer in said image; and

determining coordinates on the remote display corresponding to the location of the tip of the pointer on the quadrangular panel as depicted in the image using said projective mapping.

2. The process of claim 1, wherein the process action of locating the position of the panel in said image comprises the process action of using a color intensity gradient to locate the edges of the panel in said image.

3. The process of claim 2, wherein the process action of locating the position of the panel in said image comprises the process action of using previous available panel position data to limit the search area of the panel in the image by searching for panel edge segments in a prescribed range about each previous panel corner.

4. The process of claim 2, wherein the process action of locating the position of the panel comprises finding the location of the corners of the panel by finding the location where the edges of the panel intersect.

5. The process of claim 4, wherein the location of the corners are defined in terms of sub-pixels.

6. The process of claim 2, wherein the process action of locating the position of the quadrangular panel in said image comprises the process actions of:

locating the four sides of the panel by scanning the image and determining the four lines corresponding to the largest gradients in color and intensity in the image of the panel over a prescribed threshold; and

locating the four corners of the panel in the image by specifying each corner as an intersection of two sides.

7. The process of claim 1, wherein the process action of computing a projective mapping between the coordinates of said image and the coordinates of at least one remote display comprises:

determining a homography from at least four pairs of image-remote display correspondences.

8. The process of claim 7, wherein the process action of determining a homography from at least four pairs of image-remote display correspondences comprises a process action of mapping the four corners of the panel in the image to the four corners of the at least one remote display.

9. The process of claim 1 wherein the action of locating the pointer tip comprises the actions of:

scanning the image in a horizontal fashion to determine the pixel location where the color intensity of the panel changes significantly on a scan line by scan line basis starting from the top of the identified panel location in the image;

designating the location in each scanned line where the color intensity of the panel changes significantly as a pointer edge point;

fitting an elliptical curve through the pointer edge points;

determining a major axis of the elliptical curve; and

designating the location of the pointer tip as the location in the image where the major axis of the elliptical curve intersects the boundary of the elliptical curve.

10. The process of claim 9 the action of locating the location of the pointer tip comprises the action of using previous pointer tip position data to limit the search for the new pointer tip location to within a prescribed area.

11. The process of claim 10 wherein the prescribed area is 30 by 30 pixels.

12. The process of claim 9 further comprising the process action of using a Kalman filter to refine indicator tip location.

13. The process of claim 9 wherein the process action of determining the position of the tip of the indicator on the panel from the images of the panel comprises an action of mapping the location of the pixel of the image corresponding to the indicator tip to the corresponding remote display coordinates using the projective mapping.

14. The process of claim 9 wherein the location of the pointer tip is defined in sub-pixels.

15. The process of claim 1 wherein the process action of determining coordinates on the remote display corresponding to the location of the tip of the pointer on the quadrangular panel as depicted in the image using said projective mapping further comprises the process action of determining the coordinates on more than one remote display.

16. The process of claim 1 further comprising the process actions of

comparing the pointer tip location to a previous pointer tip location;

interpreting the pointer tip location; and

using the pointer tip position to control the position of a system indicator.

17. The process of claim 16 wherein the process action of interpreting the pointer tip location comprises interpreting the pointer tip location to simulate a mouse and wherein said system indicator is a mouse curser.

18. The process of claim 16 wherein the process action of interpreting the pointer tip location comprises the process action of interpreting the pointer tip location to simulate a keyboard and wherein said system indicator is a keyboard.

19. The process of claim 16 wherein the process action of interpreting the pointer tip location to simulate a mouse comprises the process actions of

simulating two mouse button pressing modes;

and

simulating two mouse motion types.

20. The process of claim 19 wherein the process action of simulating two mouse button pressing modes comprises:

simulating a clicking mode which simulates a left mouse button down and then up automatically; and

simulating a dragging mode which simulates the left button down until released.

21. The process of claim 20 wherein the clicking mode is simulated by the following process actions:

designating a state variable, S, that maintains two states, up and down, to simulate the nature state of a button;

initializing the state variable S to up;

setting the state variable S to down when the pointer tip location has been at a fixed place for a prescribed first period of time to simulate mouse button depression; and

automatically setting the state variable to S to up after a second prescribed period of time to simulate mouse button release.

22. The process of claim 21 wherein the first prescribed period of time is 1 second and the second prescribed period of time is 0.1 second.

23. The process of claim 21 wherein the dragging mode is simulated by the following process actions:

designating a state variable, D, that maintains two states, up and down, to simulate the nature state of a button;

initializing the state variable D to the opposite of the state of variable S;

changing the state variable D to when the pointer tip location has been at a fixed place for a prescribed third period of time and wherein

when a D-state change from up to down is detected a mouse button pressing action is simulated, and

when a D-state change from down to up is detected a mouse button pressing releasing action is simulated.

24. The process of claim 23 wherein the third prescribed period of time is 1 second.

25. The process of claim 19 wherein the absolute mouse motion is simulated by mapping the panel to the whole at least one remote display.

26. The process of claim 19 wherein the relative mouse motion is determined by the process action of measuring the velocity of the pointer tip.

27. The process of claim 8 wherein the process action of determining a homography from at least four pairs of image-remote display correspondences comprises a process action of mapping the four corners of the panel in the image to the four corners of an area less than the at least one remote display.

28. The process of claim 5 further comprising pointer tip tracking reinitialization process actions comprising:

registering the panel location in a first image at time T=0 where T is the beginning of an application;

registering the panel location in a second image at some subsequent time T+1;

calculating the homography between the panel location in said first and second images at time T=0 and time T+1;

using the homography to map image pixels in the panel at time T=0 to time T=T+1 resulting in a virtual image that would be observed when there is no pointer overlaying the panel;

subtracting the virtual image from a third input image to give a difference image;

determining the areas with large color change in the difference image; and

searching for the tip pointer by fitting an elliptic curve to the edges.

30. The system of claim 29, wherein the program module for computing a projective mapping comprises a sub-module for determining a homography from at least four pairs of image-remote display correspondences.

31. The system of claim 30, wherein the sub-module for determining a homography comprises a module for identifying the screen coordinates of the four corners of the remote display and detecting corresponding points on an image of the panel via an edge detection procedure.

32. The system of claim 31, wherein the module for locating the four corners of the image of the panel comprises:

locating the four sides of the panel by determining four lines corresponding to the four largest gradients in color and intensity in the image of the panel; and

locating the four corners of the panel in the image by specifying each corner as the location of the intersection of two lines.

33. The process of claim 29 further comprising a process action for using the determined position of the tip of the indicator to control the position of a system indicator.

36. The system of claim 35 wherein the pointer tip location coordinates are mapped to the coordinates of said remote display using a homography.

37. The system of claim 35 wherein said pointer tip tracker further comprises:

a curve fitter that fits an ellipse to the edge points of the pointer tip.

38. The system of claim 35 wherein said pointer tip tracker further comprises a color tracker.

39. The color tracker of claim 38 wherein the color tracker locates a potential pointer tip region that matches the predefined color tip model when no tip is being tracked.

40. The color tracker of claim 38 wherein the color tracker further comprises a predefined color tip model to help locate said tip in said image.

41. The color tracker of claim 38 wherein the color tracker further comprises a color tip model learned from previous pointer tip location data to help locate said tip in said image.

29. A system for using a quadrangular panel and a pointer overlaying the panel to control a remote display using a camera, the system comprising:

a camera positioned so as to have a view of the quadrangular panel;

a general purpose computing device; and

a computer program comprising program modules executable by the computing device, wherein the computing device is directed by the program modules of the computer program to,

input images of the panel from the camera;

locate a quadrangular panel in said images;

compute a projective mapping between the images and the coordinates of at least remote display to determine the remote display coordinates of a point in the images;

locate a pointer tip superimposed on the panel in the images; and

determine the coordinates in the images on the remote display corresponding to the location of the tip of the pointer on the panel as depicted in the image using the projective mapping.

34. A computer-readable medium having computer-executable instructions for a vision-based gesture interface to control a remote display using a camera, said computer-executable instructions comprising:

inputting an image of a quadrangular panel and a pointer;

locating the position of the quadrangular panel in said image;

computing a projective mapping between the coordinates of said image and the coordinates of at least one remote display;

locating a tip of the pointer in said image; and

determining coordinates on the remote display corresponding to the location of the tip of the pointer on the quadrangular panel as depicted in the image using said projective mapping.

35. A system for providing a mobile input device comprising:

a video input of a quadrangular-shaped panel;

a panel tracker which tracks said quadrangle-shaped panel by outputting the positions of its four corners;

a pointer tip tracker that identifies the location of a pointer on said panel;

an action/gesture detector that detects the location of the pointer on the panel;

and a remote display that is controlled by said panel tracker, said pointer tracker and said action detector.