Surface UI for gesture-based interaction

• US 20060036944A1
• Filed: 08/10/2004
• Published: 02/16/2006
• Est. Priority Date: 08/10/2004
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Wilson, Andrew D.
• Granted Patent: US 8,560,972 B2
• Time in Patent Office: Days
• US Class Current: 715/702
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text

Abstract

Disclosed is a unique system and method that facilitates gesture-based interaction with a user interface. The system involves an object sensing configured to include a sensing plane vertically or horizontally located between at least two imaging components on one side and a user on the other. The imaging components can acquire input images taken of a view of and through the sensing plane. The images can include objects which are on the sensing plane and/or in the background scene as well as the user as he interacts with the sensing plane. By processing the input images, one output image can be returned which shows the user objects that are in contact with the plane. Thus, objects located at a particular depth can be readily determined. Any other objects located beyond can be “removed” and not seen in the output image.

Claims

1. A surface UI image processing system comprising:

at least one non-diffuse sensing plane positioned in space to receive input from an entity;

at least two imaging components positioned to face one side of the sensing plane and that render at least a first input image and a second input image, respectively; and

a detection component that detects one or more dimensions of the first and second images to facilitate rendering a touch image, the touch image comprising a combination of the at least first and second input images.

2. The system of claim 1, further comprising:

a component that removes one or more effects of lens distortion from at least one input image; and

a rectification component that resolves one or more first parameters of the sensing plane with one or more second parameters of the at least one input image.

3. The system of claim 2, the first and second parameters are collected offline and stored.

4. The system of claim 2, the first parameters comprise measurements corresponding to four corners of the sensing plane.

5. The system of claim 2, the second parameters comprise measurements corresponding to four corners of the at least one input image.

6. The system of claim 2, the rectification component transforms the first input image and the second input image such that the physical points therein refer to the same physical points on the sensing plane.

7. The system of claim 1, further comprising an edge-detection filter that is applied to at least the first and second input images to highlight one or more edge contours of the first and second input images, respectively, to thereby yield at least a first and a second edge image.

8. The system of claim 6, further comprising an analysis component that multiplies the at least first and second edge images to facilitate producing the touch image that reveals where the edge contours of the at least first and second input images overlap.

9. The system of claim 7, wherein a presence of overlapping edge contours indicates that at least one input object is in contact with the sensing plane.

10. The system of claim 1, wherein the input in contact with the sensing plane has zero displacement from one view of the object to another.

11. The system of claim 1, further comprising a pixel-wise comparison component that performs pixel-wise multiplication of the input images to facilitate generating the touch image.

12. The system of claim 1, the imaging components comprise at least one of a video camera and a still camera.

13. The system of claim 1, the sensing plane comprises an acrylic plastic sheet or display screen that is at least partially transparent to allow the imaging components to see through the acrylic plastic sheet or display screen and to allow the touch image to be projected onto it.

14. The system of claim 1, a surface of the sensing plane facing the entity is not active for interaction.

15. The system of claim 1, the sensing plane comprises an interaction plane located a distance away from the sensing plane.

16. The system of claim 1, the entity is positioned opposite of the imaging components on the other side of the sensing plane.

17. The system of claim 1, the sensing plane is located vertically or horizontally with respect to the entity and the imaging components.

18. The system of claim 1, further comprising a block matching component that examines patterns of objects to determine whether overlap exists between the input images.

19. The system of claim 1, further comprising an IR component that employs IR illumination to identify IR reflective objects on or near the sensing plane to facilitate determining overlap between the input images.

20. The system of claim 19, further comprising an IR illuminant placed behind a plane and on the same side of the imaging components to illuminate the plane surface evenly, IR pass-filters that can be applied to a lens of each of the imaging components, and an IR cut-filter applied to a projector.

21. The system of claim 1, further comprising a calibration component that manually calibrates parameters of the input images and the sensing plane with respect to current positions of the imaging components for offline storage.

38. A computer readable medium having stored thereon computer-executable components of claim 1.

22. An image processing method that facilitates gesture-based interaction comprising:

acquiring at least first and second input images from first and second imaging components, respectively;

remapping the first and second input images with respect to a sensing plane;

determining and highlighting contours of the input images to yield highlighted images; and

combining the highlighted images to obtain a touch image whereby objects contacting the sensing plane remain and background images seen in the input images are excluded.

23. The method of claim 22, further comprising providing at least one non-diffuse sensing plane positioned in space to receive input from a user;

and

positioning the first and second imaging components to face toward the sensing plane on a side opposite from the user.

24. The method of claim 22, remapping the first and second images with respect to the sensing plane comprises:

removing lens distortion from the input images; and

warping the first input image and the second input image individually to the sensing plane to obtain a one-to-one correspondence of physical points between each input image and the sensing plane.

25. The method of claim 24, removing lens distortion and warping are performed at the same time.

26. The method of claim 22, determining contours of the input images comprising applying an edge detection filter to the input images to yield edge images.

27. The method of claim 22, combining the highlighted images to obtain a touch image comprises multiplying the input images pixel-wise such that surviving pixels are left to create the touch image.

28. The method of claim 22, further comprising projecting the touch image back to the user in real-time.

29. The method of claim 22, further comprising calibrating the sensing plane in each current imaging component view to find corner parameters of the sensing plane;

and storing calibration data offline.

30. The method of claim 29, further comprising repeating calibration of sensing plane when the imaging components move from their current positions.

31. The method of claim 22, the imaging components comprises cameras.

32. The method of claim 22, further comprising analyzing location of objects in the touch image to facilitate object tracking or cursor control.

33. The method of claim 22, further comprising normalizing the touch image to remove effects due to non-uniformity of illumination in a pixel-wise manner.

34. The method of claim 22, further comprising arbitrarily defining the sensing plane to coincide with a display screen.

35. The method of claim 22, further comprising mounting at least a third imaging component that is triggered to take a high resolution snapshot of at least a portion of object(s) in the sensing plane to facilitate visualizing greater detail of the object(s).

36. The method of claim 35, the third imaging component comprises a camera having higher resolution than the first and second imaging components.

39. A computer readable medium having stored thereon computer- executable method of claim 22.

37. A data packet adapted to be transmitted between two or more computer processes facilitating gesture-based interaction with an input device, the data packet comprising:

information associated with acquiring at least two input images from at least two cameras, respectively;

warping the input images to a sensing plane or display screen at the same time as removing lens distortion from the input images;

highlighting contours of objects overlapping between the first and second images and combining the first and second images pixel-wise to obtain a touch image comprising objects touching the sensing plane or display screen.

40. A surface UI image processing system that facilitates gesture-based interaction with an input device comprising:

means for acquiring at least first and second input images from first and second imaging components, respectively;

means for remapping the first and second input images with respect to a sensing plane;

means for determining and highlighting contours of the input images to yield highlighted images; and

means for combining the highlighted images to obtain a touch image whereby objects contacting the sensing plane remain and background images seen in the input images are excluded.

41. The system of claim 40, further comprising means for positioning the first and second imaging components to face toward the sensing plane on a side opposite from the user.

42. The system of claim 40, wherein the sensing plane is at least partially transparent.

43. The system of claim 40, remapping the first and second images with respect to the sensing plane comprises:

means for removing lens distortion from the input images; and

means for warping the first input image and the second input image individually to the sensing plane to obtain a one-to-one correspondence of physical points between each input image and the sensing plane.