TOUCHLESS INPUT

• US 20140104168A1
• Filed: 10/12/2012
• Published: 04/17/2014
• Est. Priority Date: 10/12/2012
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Hegde, Guru
• Granted Patent: US 9,310,895 B2
• Time in Patent Office: Days
• US Class Current: 345/157
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/033 Pointing devices displaced or positioned by the user , e.g. mice, trackballs, pens or joysticks; Accessories therefor digitisers characterised by the transducing means G06F3/041
• CPC Class Codes: G06F 3/0428 by sensing at the edges of the touch surface the interruption of optical paths, e.g. an illumination plane, parallel to the touch surface which may be virtual sensing beam interruptions in a planar beam grid of an optical touch-screen G06F3/0421
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

A series of depth maps image a scene. Each depth map includes a plurality of pixels, where each pixel defines a depth of a surface imaged by that pixel. A set of nonstatic pixels having a shallowest depth within the series of depth maps is identified. The position of this set of nonstatic pixels is mapped to a cursor position.

Claims

1. A method of receiving user input, the method comprising:

receiving a series of depth maps imaging a scene, each depth map including a plurality of pixels, each of the plurality of pixels defining a depth of a surface imaged by that pixel;

identifying a set of nonstatic pixels having a shallowest depth within the series of depth maps; and

mapping a position of the set of nonstatic pixels to a cursor position.

2. The method of claim 1, where the set of nonstatic pixels includes a plurality of contiguous pixels.

3. The method of claim 1, where the set of nonstatic pixels includes one pixel.

4. The method of claim 1, wherein identifying the set of nonstatic pixels includes identifying a set of nonstatic pixels that maintains the shallowest depth for a threshold consecutive number of depth maps in the series of depth maps.

5. The method of claim 1, where identifying the set of nonstatic pixels includes defining a depth difference between a corresponding pixel in different depth maps from the series of depth maps;

for each pixel;

if the difference for that pixel is below a threshold, identifying that pixel as a static pixel;

or

if the difference for that pixel is at or above the threshold, identifying that pixel as a nonstatic pixel.

6. The method of claim 1, where the position of the set of nonstatic pixels is mapped to a cursor position according to a mapping function having a scale that changes as a function of a depth of the set of nonstatic pixels.

7. The method of claim 1, where the position of the set of nonstatic pixels is mapped to a cursor position according to a mapping function, the mapping function having a scale that changes as a function of a size of the set of nonstatic pixels.

8. The method of claim 1, where the position of the set of nonstatic pixels is mapped to a cursor position according to a mapping function, the mapping function having a scale that changes depending on which finger is identified as corresponding to the set of nonstatic pixels having the shallowest depth.

9. The method of claim 1, where the surface imaged by the set of nonstatic pixels is a finger proximate a display device displaying a cursor at the cursor position, and where mapping the position of the set of nonstatic pixels to the cursor position includes shifting the cursor position away from a world space position of the finger.

10. The method of claim 1, where the surface imaged by the set of nonstatic pixels is a finger that is not proximate a display device displaying a cursor at the cursor position.

11. The method of claim 1, further comprising tracking the set of nonstatic pixels throughout a consecutive number of depth maps in the series of depth maps and recognizing a cursor operation from one or more parameters of the set of nonstatic pixels in the consecutive number of depth maps.

12. The method of claim 10, where the cursor operation includes one or more of a click operation, a double-click operation, a triple-click operation, a select operation, a highlight operation, a scroll operation, and a drag operation.

13. The method of claim 1, wherein be one or more parameters include a three-dimensional path of the set of nonstatic pixels.

14. A method, comprising:

receiving from a depth camera a series of depth maps imaging a scene including a human subject, each depth map including a plurality of pixels, each of the plurality of pixels defining a depth of a surface imaged by that pixel;

modeling the human subject with a virtual skeleton including a plurality of virtual finger joints, each virtual finger joint defined with a three-dimensional position derived from one or more depth maps; and

interpreting the three-dimensional behavior of one or more virtual finger joints modeled from one or more depth maps as a finger gesture.

15. A method, comprising:

receiving from a depth camera a series of depth maps imaging a scene including a human subject, each depth map including a plurality of pixels, each of the plurality of pixels defining a depth of a surface imaged by that pixel;

modeling the human subject with a virtual skeleton including a virtual hand joint;

mapping a position of the virtual hand joint to a corresponding portion of one or more depth maps in the series of depth maps, the corresponding portion of the one or more depth maps imaging a hand of the human subject modeled by the virtual hand joint;

modeling one or more fingers from depth map information of the corresponding portion of one or more depth maps; and

interpreting a finger gesture performed by the one or more fingers.

16. The method of claim 15, where the one or more fingers are modeled using a previously trained machine learning algorithm.

17. The method of claim 15, where modeling the one or more fingers further includes:

fitting a substantially linear profile to pixels corresponding to the one or more fingers;

and

calculating a finger angle from the linear profile.

18. The method of claim 15, where the finger gesture is interpreted based on a selection previously made by the human subject.

19. The method of claim 15, where modeling the one or more fingers further includes modeling one or more fingerprints of the human subject.

20. The method of claim 15, where the finger gesture includes one or more of a tap gesture, a double-tap gesture, a press gesture, a scroll gesture, a pan gesture, a flick gesture, a two finger tap gesture, a two finger scroll gesture, a pinch gesture, a spread gesture, and a rotate gesture.