TOUCHLESS INPUT

• US 20160202770A1
• Filed: 03/18/2016
• Published: 07/14/2016
• Est. Priority Date: 10/12/2012
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Hegde, Guru
• Granted Patent: US 10,019,074 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/033 Pointing devices displaced or positioned by the user , e.g. mice, trackballs, pens or joysticks; Accessories therefor digitisers characterised by the transducing means G06F3/041
• CPC Class Codes: G06F 3/0428 by sensing at the edges of the touch surface the interruption of optical paths, e.g. an illumination plane, parallel to the touch surface which may be virtual sensing beam interruptions in a planar beam grid of an optical touch-screen G06F3/0421
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

A series of depth maps image a scene including a human subject. Each depth map includes a plurality of pixels, where each of the plurality of pixels defines a depth of a surface imaged by that pixel. The human subject is modeled with a virtual skeleton including a virtual hand joint. A position of the virtual hand joint is mapped to a corresponding portion of one or more depth maps in the series of depth maps. One or more fingers are modeled from depth map information of the corresponding portion of the one or more depth maps. A finger gesture is interpreted from the modeled fingers.

Claims

1. A method, comprising:

receiving from a depth camera a series of depth maps imaging a scene including a human subject, each depth map including a plurality of pixels, each of the plurality of pixels defining a depth of a surface imaged by that pixel;

modeling the human subject with a virtual skeleton including a virtual hand joint;

mapping a position of the virtual hand joint to a corresponding portion of one or more depth maps in the series of depth maps, the corresponding portion of the one or more depth maps imaging a hand of the human subject modeled by the virtual hand joint;

modeling one or more fingers from depth map information of the corresponding portion of one or more depth maps; and

interpreting a finger gesture performed by the one or more fingers.

2. The method of claim 1, where the one or more fingers are modeled using a previously trained machine learning algorithm.

3. The method of claim 1, where modeling the one or more fingers further includes:

fitting a substantially linear profile to pixels corresponding to the one or more fingers;

and

calculating a finger angle from the linear profile.

4. The method of claim 1, where the finger gesture is interpreted based on a selection previously made by the human subject.

5. The method of claim 1, where modeling the one or more fingers further includes modeling one or more fingerprints of the human subject.

6. The method of claim 1, where the finger gesture includes a tap gesture.

7. The method of claim 1, where the finger gesture includes a double-tap gesture.

8. The method of claim 1, where the finger gesture includes a press gesture.

9. The method of claim 1, where the finger gesture includes a scroll gesture.

10. The method of claim 1, where the finger gesture includes a pan gesture.

11. The method of claim 1, where the finger gesture includes a flick gesture.

12. The method of claim 1, where the finger gesture includes a two finger tap gesture.

13. The method of claim 1, where the finger gesture includes a two finger scroll gesture.

14. The method of claim 1, where the finger gesture includes a pinch gesture.

15. The method of claim 1, where the finger gesture includes a spread gesture.

16. The method of claim 1, where the finger gesture includes a rotate gesture.

17. A computer, comprising:

a logic machine configured to execute instructions; and

a storage machine holding instructions executable by the logic machine, the instructions including;

instructions to receive from a depth camera a series of depth maps imaging a scene including a human subject, each depth map including a plurality of pixels, each of the plurality of pixels defining a depth of a surface imaged by that pixel;

instructions to model the human subject with a virtual skeleton including a virtual hand joint;

instructions to map a position of the virtual hand joint to a corresponding portion of one or more depth maps in the series of depth maps, the corresponding portion of the one or more depth maps imaging a hand of the human subject modeled by the virtual hand joint;

instructions to model one or more fingers from depth map information of the corresponding portion of one or more depth maps; and

instructions to interpret a finger gesture performed by the one or more fingers.

18. The computer of claim 17, where the instructions to model one or more fingers include a previously trained machine learning algorithm.

19. The computer of claim 17, where the instructions to interpret the finger gesture are configured to interpret the finger gesture based on a selection previously made by the human subject.

20. A method, comprising:

receiving from a depth camera a series of depth maps imaging a scene including a human subject, each depth map including a plurality of pixels, each of the plurality of pixels defining a depth of a surface imaged by that pixel;

modeling the human subject with a virtual skeleton including a plurality of virtual finger joints, each virtual finger joint defined with a three-dimensional position derived from one or more depth maps; and

interpreting the three-dimensional behavior of one or more virtual finger joints modeled from one or more depth maps as a finger gesture.