Surface UI for gesture-based interaction

• US 8,560,972 B2
• Filed: 08/10/2004
• Issued: 10/15/2013
• Est. Priority Date: 08/10/2004
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Wilson, Andrew D
• Primary Examiner(s): Kim, Matt
• Assistant Examiner(s): Pan, Yongjia
• Application Number:
  US10/914,922
• Publication Number:
  US 20060036944A1
• Time in Patent Office: 3,353 Days
• Field of Search: 715/702, 715/863, 882/181, 882/312
• US Class Current: 715/863
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text

Abstract

Disclosed is a unique system and method that facilitates gesture-based interaction with a user interface. The system involves an object sensing configured to include a sensing plane vertically or horizontally located between at least two imaging components on one side and a user on the other. The imaging components can acquire input images taken of a view of and through the sensing plane. The images can include objects which are on the sensing plane and/or in the background scene as well as the user as he interacts with the sensing plane. By processing the input images, one output image can be returned which shows the user objects that are in contact with the plane. Thus, objects located at a particular depth can be readily determined. Any other objects located beyond can be “removed” and not seen in the output image.

Claims

1. A system comprising:

one or more processors;

one or more memories;

at least one sensing plane positioned in space to receive input from one or more users that are interacting with the at least one sensing plane;

a detection component, maintained on the one or more memories and executable by the one or more processors, to detect one or more dimensions of a first input image and a second input image received from a first imaging component and a second imaging component, respectively, and to render a touch image, the touch image comprising a combination of the at least first and second input images, wherein each of the first and second input images include at least part of the received input from the one or more users;

an edge-detection filter that is applied to at least the first and second input images to highlight one or more edge contours of the first and second input images, respectively, to thereby yield a first and a second edge image; and

a pixel-wise comparison component, maintained on the one or more memories and executable by the one or more processors, to perform pixel-wise multiplication of the first and second edge images to render the touch image by identifying where the one or more edge contours of the first and second edge images overlap while excluding background objects that fail to align in the first and second edge images,

wherein;

the detection component is further executable by the one or more processors to identify, using the touch image, the one or more users that are interacting with the at least one sensing plane; and

the first input image or the second input image includes data from one or more other users and the detection component is further executable by the one or more processors to determine that the one or more other users are not interacting with the at least one sensing plane and thereby the one or more other users are not in the touch image.

2. The system of claim 1, further comprising:

a component that removes one or more effects of lens distortion from at least one of the first or second input images; and

a rectification component that resolves one or more first parameters of the at least one sensing plane with one or more second parameters of at least one of the first or second input images.

3. The system of claim 2, wherein the first and second parameters are collected offline and stored.

4. The system of claim 2, wherein the first parameters comprise measurements corresponding to four corners of the at least one sensing plane.

5. The system of claim 2, wherein the second parameters comprise measurements corresponding to four corners of at least one of the first or second input images.

6. The system of claim 2, wherein the rectification component transforms the first input image and the second input image such that physical points therein refer to same physical points on the at least one sensing plane.

7. The system of claim 1, wherein a presence of overlapping edge contours indicates that at least one input object is in contact with the at least one sensing plane.

8. The system of claim 1, wherein the first and second imaging components comprise at least one of a video camera and a still camera.

9. The system of claim 1, wherein a surface of the at least one sensing plane facing the one or more users is not active for interaction to allow calibration offline of the first and second imaging components.

10. The system of claim 1, wherein the of one or more users are positioned opposite of the first and second imaging components on an other side of the at least one sensing plane.

11. The system of claim 1, wherein the at least one sensing plane is located vertically or horizontally with respect to the one or more users and the first and second imaging components.

12. The system of claim 1, further comprising a block matching component that examines patterns of objects to determine whether overlap exists between the first and second input edge images.

13. The system of claim 1, further comprising an IR (infrared) component that employs IR illumination to identify IR reflective objects on or near the at least one sensing plane to determine overlap between the first and second edge images.

14. The system of claim 13, further comprising an IR (infrared) illuminant placed behind the at least one sensing plane and on a same side of the first and second imaging components to illuminate the at least one sensing plane surface evenly, IR pass-filters that can be applied to a lens of each of the first and second imaging components, and an IR cut-filter applied to a projector.

15. The system of claim 1, further comprising a calibration component that manually calibrates parameters of the first and second input images and the at least one sensing plane with respect to current positions of the first and second imaging components for offline storage.

16. The system of claim 1, wherein the detection component identifies the one or more users through application of facial recognition.

17. The system of claim 1, wherein the detection component identifies the one or more users through application of facial tracking.

18. A method comprising:

acquiring first and second input images of one or more users interacting with a sensing plane from first and second imaging components, respectively;

determining edge contours of the first and second input images to yield first and second highlighted edge images based at least in part on features of the one or more users interacting with the sensing plane;

performing pixel-wise multiplication of the first and second highlighted edge images to render a touch image by identifying where the edge contours of the first and second highlighted edge images overlap while excluding background objects that fail to overlap in the first and second highlighted edge images; and

recognizing each of the one or more users interacting with the sensing plane based on the first and second highlighted edge images, wherein the first input image or the second input image includes data from one or more other users not interacting with the sensing plane and thereby the data from the one or more other users is not part of the touch image.

19. The method of claim 18, wherein the sensing plane is a non-diffuse sensing plane positioned in space to receive input from the one or more users;

and

the first and second imaging components are positioned to face toward the non-diffuse sensing plane on a side opposite from the one or more users.

20. The method of claim 18, further comprising:

removing lens distortion from the first and second input images; and

warping the first input image and the second input image individually to the sensing plane to obtain a one-to-one correspondence of physical points between each input image and the sensing plane.

21. The method of claim 20, wherein the removing the lens distortion and the warping are performed at a same time.

22. The method of claim 18, wherein the determining the edge contours of the first and second input images comprises applying an edge detection filter to the first and second input images.

23. The method of claim 18, further comprising projecting the touch image back to the one or more users in real-time.

24. The method of claim 18, further comprising:

calibrating the sensing plane in each current imaging component view to find corner parameters of the sensing plane; and

storing calibration data offline.

25. The method of claim 24, further comprising repeating calibration of the sensing plane when the first imaging component or the second imaging component switches from its current imaging component view.

26. The method of claim 18, wherein the first and second imaging components comprises cameras.

27. The method of claim 18, further comprising analyzing location of objects in the touch image to track objects or cursor control.

28. The method of claim 18, further comprising normalizing the touch image to remove effects due to non-uniformity of illumination in a pixel-wise manner.

29. The method of claim 18, further comprising arbitrarily defining the sensing plane to coincide with a display screen.

30. The method of claim 18, further comprising mounting at least a third imaging component that is triggered to take a high resolution snapshot of at least a portion of object(s) in the sensing plane to visualize greater detail of the object(s).

31. The method of claim 30, wherein the at least the third imaging component comprises a camera having higher resolution than the first and second imaging components.

32. A memory having stored thereon computer-executable instructions to perform the method of claim 18.

33. A system comprising:

a processor;

a memory, coupled to the processor, storing;

a first component, operable by the processor, to acquire at least first and second input images from first and second imaging components, respectively;

a second component, operable by the processor, to remap the first and second input images with respect to a sensing plane;

a third component, operable by the processor, to determine and highlight edge contours of the first and second input images to yield first and second highlighted images; and

a fourth component, operable by the processor, to combine the first and second highlighted images to obtain a touch image that includes first data associated with edge contours that align after the combining and that excludes second data associated with edge contours that do not align after the combining, the first data being associated with one or more users interacting with the sensing plane and the second data being associated with one or more other users not interacting with the sensing plane.

34. The system of claim 33, further comprising a fifth component, operable by the processor, to position the first and second imaging components to face toward the sensing plane on a side opposite from the one or more users.

35. The system of claim 33, wherein the sensing plane is at least partially transparent.

36. The system of claim 33, wherein the second component is further operable by the processor to:

remove lens distortion from the first and second input images; and

warp the first input image and the second input image individually to the sensing plane to obtain a one-to-one correspondence of physical points between each input image and the sensing plane.