Touchless input

• US 10,019,074 B2
• Filed: 03/18/2016
• Issued: 07/10/2018
• Est. Priority Date: 10/12/2012
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Hegde, Guru
• Primary Examiner(s): Bolotin, Dmitriy
• Application Number:
  US15/073,895
• Publication Number:
  US 20160202770A1
• Time in Patent Office: 844 Days
• Field of Search: 345157
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/033 Pointing devices displaced or positioned by the user , e.g. mice, trackballs, pens or joysticks; Accessories therefor digitisers characterised by the transducing means G06F3/041
• CPC Class Codes: G06F 3/0428 by sensing at the edges of the touch surface the interruption of optical paths, e.g. an illumination plane, parallel to the touch surface which may be virtual sensing beam interruptions in a planar beam grid of an optical touch-screen G06F3/0421
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

A series of depth maps image a scene including a human subject. Each depth map includes a plurality of pixels, where each of the plurality of pixels defines a depth of a surface imaged by that pixel. The human subject is modeled with a virtual skeleton including a virtual hand joint. A position of the virtual hand joint is mapped to a corresponding portion of one or more depth maps in the series of depth maps. One or more fingers are modeled from depth map information of the corresponding portion of the one or more depth maps. A finger gesture is interpreted from the modeled fingers.

Claims

1. A method, comprising:

receiving from a depth camera a series of depth maps imaging a scene including a human subject, each depth map including a plurality of pixels, each of the plurality of pixels defining a depth of a surface imaged by that pixel;

modeling the human subject with a virtual skeleton including a plurality of virtual joints including a virtual hand joint and one or more other virtual joints;

mapping a position of the virtual hand joint to a corresponding portion of one or more depth maps in the series of depth maps, the corresponding portion of the one or more depth maps imaging a hand of the human subject modeled by the virtual hand joint and excluding other portions of the human subject modeled by the one or more other virtual joints;

performing analysis on depth map information of the corresponding portion of one or more depth maps to model one or more fingers of the hand without performing analysis on depth map information of other portions of the one or more depth maps corresponding to the other portions of the human subject modeled by the one or more other virtual joints; and

interpreting a finger gesture performed by the one or more fingers.

2. The method of claim 1, where the one or more fingers are modeled using a previously trained machine learning algorithm.

3. The method of claim 1, where modeling the one or more fingers further includes:

fitting a substantially linear profile to pixels corresponding to the one or more fingers; and

calculating a finger angle from the linear profile.

4. The method of claim 1, where the finger gesture is interpreted based on a selection previously made by the human subject.

5. The method of claim 1, where modeling the one or more fingers further includes modeling one or more fingerprints of the human subject.

6. The method of claim 1, where the finger gesture includes a tap gesture.

7. The method of claim 1, where the finger gesture includes a double-tap gesture.

8. The method of claim 1, where the finger gesture includes a press gesture.

9. The method of claim 1, where the finger gesture includes a scroll gesture.

10. The method of claim 1, where the finger gesture includes a pan gesture.

11. The method of claim 1, where the finger gesture includes a flick gesture.

12. The method of claim 1, where the finger gesture includes a two finger tap gesture.

13. The method of claim 1, where the finger gesture includes a two finger scroll gesture.

14. The method of claim 1, where the finger gesture includes a pinch gesture.

15. The method of claim 1, where the finger gesture includes a rotate gesture.

16. The method of claim 1, wherein the one or more other virtual joints include one or more of a head joint, shoulder center joint, spine joint, hip center joint, right shoulder joint, right elbow joint, right wrist joint, left shoulder joint, left elbow joint, left wrist joint, right hip joint, right knee joint, right ankle joint, right foot joint, left hip joint, left knee joint, left ankle joint, and left foot joint.

17. A computer, comprising:

a logic machine configured to execute instructions; and

a storage machine holding instructions executable by the logic machine, the instructions including;

instructions to receive from a depth camera a series of depth maps imaging a scene including a human subject, each depth map including a plurality of pixels, each of the plurality of pixels defining a depth of a surface imaged by that pixel;

instructions to model the human subject with a virtual skeleton including a plurality of virtual joints including a virtual hand joint and one or more other virtual joints;

instructions to map a position of the virtual hand joint to a corresponding portion of one or more depth maps in the series of depth maps, the corresponding portion of the one or more depth maps imaging a hand of the human subject modeled by the virtual hand joint and excluding other portions of the human subject modeled by the one or more other virtual joints;

instructions to perform analysis on depth map information of the corresponding portion of one or more depth maps to model one or more fingers of the hand without performing analysis on depth map information of other portions of the one or more depth maps corresponding to the other portions of the human subject modeled by the one or more other virtual joints; and

instructions to interpret a finger gesture performed by the one or more fingers.

18. The computer of claim 17, where the instructions to model one or more fingers include a previously trained machine learning algorithm.

19. The computer of claim 17, where the instructions to interpret the finger gesture are configured to interpret the finger gesture based on a selection previously made by the human subject.

20. A method, comprising:

receiving from a depth camera a series of depth maps imaging a scene including a human subject, each depth map including a plurality of pixels, each of the plurality of pixels defining a depth of a surface imaged by that pixel;

modeling the human subject with a virtual skeleton including a plurality of virtual joints including a virtual hand joint and one or more other virtual joints;

after a position of the virtual hand joint is determined, performing analysis on a subset of pixels selected from the plurality of pixels and corresponding to the virtual hand joint to model a plurality of virtual finger joints without performing analysis on pixels not included in the subset; and

interpreting the three-dimensional behavior of one or more virtual finger joints as a finger gesture.