Gesture recognition techniques

• US 10,331,222 B2
• Filed: 05/24/2016
• Issued: 06/25/2019
• Est. Priority Date: 05/31/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Kim, David, Hilliges, Otmar D., Izadi, Shahram, Olivier, Patrick L., Shotton, Jamie Daniel Joseph, Kohli, Pushmeet, Molyneaux, David G., Hodges, Stephen E., Fitzgibbon, Andrew W.
• Primary Examiner(s): Haley, Joseph R
• Assistant Examiner(s): Frank, Emily J
• Application Number:
  US15/162,905
• Publication Number:
  US 20160370867A1
• Time in Patent Office: 1,127 Days
• Field of Search: None
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/75 involving models
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

In one or more implementations, a static geometry model is generated, from one or more images of a physical environment captured using a camera, using one or more static objects to model corresponding one or more objects in the physical environment. Interaction of a dynamic object with at least one of the static objects is identified by analyzing at least one image and a gesture is recognized from the identified interaction of the dynamic object with the at least one of the static objects to initiate an operation of the computing device.

Claims

1. A method implemented by a computing device, the method comprising:

via a sensor, capturing depth data corresponding to a physical environment;

based at least in part on using the depth data corresponding to the physical environment, forming at least one model of a static physical object;

identifying interaction of a dynamic object with the at least one model of the static physical object, the at least one model of the static physical object modeling one or more objects in the physical environment using a geometry model; and

recognizing a gesture from the identified interaction of the dynamic object with the at least one model of the static physical object to initiate an operation of the computing device.

2. A method as described in claim 1, wherein the identified interaction includes identifying contact between the dynamic object and the at least one of the static objects by analyzing the data.

3. A method as described in claim 1, further comprising generating the geometry model using one or more images captured by the sensor, and wherein identifying the interaction of the dynamic object with the at least one model of the static physical object comprises using at least one image captured subsequent to the one or more images that were used to generate the geometry model.

4. A method as described in claim 1, wherein the sensor is a camera and the data includes images captured by the camera.

5. A method as described in claim 1, wherein the operation is to form a communication that identifies the gesture for communicating to another computing device to initiate another operation of the other computing device.

6. A method as described in claim 1, wherein the model of the static physical object comprises a displayed image including the physical object.

7. A method as described in claim 1, wherein the geometry model comprises a static geometry model.

8. A method implemented by a computing device, the method comprising:

via a sensor, capturing depth data corresponding to a physical environment;

based at least in part on using the depth data corresponding to the physical environment, forming at least one model of a static physical object;

identifying contact between a dynamic object and the at least one model of the static physical object, the at least one model of the static physical object modeling one or more objects in the physical environment; and

recognizing a gesture from the identified contact of the dynamic object with the at least one model of a static physical object to initiate an operation of the computing device.

9. A method as described in claim 8, further comprising generating the at least one model of the static physical object using one or more images captured by the sensor, and wherein identifying the contact between the dynamic object and the at least one model of the static physical object comprises using at least one image captured subsequent to the one or more images that were used to generate the at least one model of the static physical object.

10. A method as described in claim 8, wherein the sensor is a camera and the data includes images captured by the camera.

11. A method as described in claim 8, wherein the operation is to form a communication that identifies the gesture for communicating to another computing device to initiate another operation of the other computing device.

12. A method as described in claim 8, wherein the model of the static physical object comprises a displayed image including the physical object.

13. A method as described in claim 8, wherein the model of the static physical object models one or more objects in the physical environment using a static geometry model.

14. An apparatus comprising:

a camera configured to capture depth data of a physical environment of the camera and included within a housing that is configured to be held by a user; and

one or more modules communicatively coupled to the camera and implemented at least partially in hardware, the one or more modules configured to perform operations comprising;

based at least in part on using the depth data corresponding to the physical environment, forming at least one model of a static physical object;

identifying interaction of a dynamic object with the at least one model of the static physical object, the at least one model of the static physical object modeling one or more objects in the physical environment using a geometry model; and

recognizing a gesture from the identified interaction of the dynamic object with the at least one model of the static physical object to initiate an operation of the computing device.

15. An apparatus as described in claim 14, wherein the identified interaction includes identifying contact between the dynamic object and the at least one of the static objects by analyzing the depth data.

16. An apparatus as described in claim 14, wherein the geometry model is generated using one or more images captured by the camera, and wherein identifying the interaction of the dynamic object with the at least one model of the static physical object comprises using at least one image captured subsequent to the one or more images that were used to generate the geometry model.

17. An apparatus as described in claim 14, wherein the depth data includes images captured by the camera.

18. An apparatus as described in claim 14, wherein the operation is to form a communication that identifies the gesture for communicating to another computing device to initiate another operation of the other computing device.

19. An apparatus as described in claim 14, wherein the model of the static physical object comprises a displayed image including the physical object.

20. An apparatus as described in claim 14, wherein the geometry model comprises a static geometry model.