Gestures Beyond Skeletal

• US 20100306715A1
• Filed: 05/29/2009
• Published: 12/02/2010
• Est. Priority Date: 05/29/2009
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Geisner, Kevin, Markovic, Relja, Snook, Gregory N., Latta, Stephen
• Granted Patent: US 9,383,823 B2
• Time in Patent Office: Days
• US Class Current: 715/863
• CPC Class Codes: A63B 2024/0025 Tracking the path or location of one or more users, e.g. players of a game
• CPC Class Codes: A63F 2300/105 using inertial sensors, e.g. accelerometers, gyroscopes
• CPC Class Codes: A63F 2300/1087 comprising photodetecting means, e.g. a camera
• CPC Class Codes: A63F 2300/64 for computing dynamical parameters of game objects, e.g. motion determination or computation of frictional forces for a virtual car
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/014 Hand-worn input/output arrangements, e.g. data gloves
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G09G 2340/12 Overlay of images, i.e. displayed pixel being the result of switching between the corresponding input pixels
• CPC Class Codes: G10H 2220/401 3D sensing, i.e. three-dimensional (x, y, z) position or movement sensing. movement pattern or gesture sensing G10H2220/201; geolocation sensing G10H2220/355, 3D sensing with accelerometer G10H2220/395

Abstract

Systems, methods and computer readable media are disclosed for gesture input beyond skeletal. A user'"'"'s movement or body position is captured by a capture device of a system. Further, non-user-position data is received by the system, such as controller input by the user, an item that the user is wearing, a prop under the control of the user, or a second user'"'"'s movement or body position. The system incorporates both the user-position data and the non-user-position data to determine one or more inputs the user made to the system.

Claims

1. A method for enabling a user to make hybrid-gesture input to an application comprising:

receiving first data concerning movement or position of the user captured by a capture device;

receiving additional data, the additional data not derived from the movement or position of the user captured by a capture device;

analyzing the first data in combination with the additional data and determining therefrom an output corresponding to the likelihood that the user performed a system-recognized input; and

sending the output to the application.

2. The method of claim 1 wherein the additional data comprises one from a set, the set comprising:

a prop, clothing worn by the user, an object in a scene captured by the capture device, a second user or users, a sound made by the user, a controller or remote control input, an amount and/or position of light in the scene, user interaction with a touch-sensitive device, and a facial expression of the user.

3. The method of claim 1 wherein analyzing the first data and the additional data further comprises:

determining a latency between the first data and the additional data.

4. The method of claim 1 wherein the additional data is received from a second user.

5. The method of claim 1 wherein the first data and the additional data are received as a result of the user entering the first data and the additional data at substantially the same time.

6. The method of claim 1 wherein the additional data comprises a prop, and the prop is manipulated by a plurality of users.

7. The method of claim 1 wherein analyzing the first data and the additional data and determining an output corresponding to the likelihood that the user performed a system-recognized input further comprises:

analyzing the first data and the additional data with a gesture filter and at least one corresponding parameter representing the system-recognized input, and determining an output corresponding to the likelihood that the user performed a system-recognized input.

8. A system for enabling a user to make hybrid-gesture input to an application comprising:

a processor;

at least one memory storing computer-readable instructions that when executed on the processor, cause the processor to perform operations comprising;

receiving first data concerning movement or position of the user captured by a capture device;

receiving additional data, the additional data not derived from the movement or position of the user captured by a capture device;

analyzing the first data and the additional data and determining an output corresponding to the likelihood that the user performed a system-recognized input; and

sending the output to the application.

9. The system of claim 8 wherein first data comprises one from a set, the set comprising:

a prop, clothing worn by the user, an object in a scene captured by the capture device, a second user or users, a sound made by the user, a controller or remote control input, an amount and/or position of light in the scene, user interaction with a touch-sensitive device, and a facial expression of the user.

10. The system of claim 8 wherein the component for analyzing the first data and the additional data further comprises:

a component for determining a latency between the first data and the additional data.

11. The system of claim 8 wherein the additional data is received from a second user.

12. The system of claim 8 wherein the first data and the additional data are received as a result of the user entering the first data and the additional data at substantially the same time.

13. The system of claim 8 wherein the additional data comprises a prop, and the prop is manipulated by a plurality of users.

14. The system of claim 8 wherein the component for analyzing the first data and the additional data and determining an output corresponding to the likelihood that the user performed a system-recognized input further comprises:

a component for analyzing the first data and the additional data with a gesture filter and at least one corresponding parameter representing the system-recognized input, and determining an output corresponding to the likelihood that the user performed a system-recognized input.

15. A computer-readable storage medium for enabling a user to make hybrid-gesture input to an application, bearing computer-readable instructions that when executed on a processor cause the processor to perform operations comprising:

receiving first data concerning movement or position of the user captured by a capture device;

receiving additional data, the additional data not derived from the movement or position of the user captured by a capture device;

analyzing the first data and the additional data and determining an output corresponding to the likelihood that the user performed a system-recognized input;

sending the output to the application.

16. The computer-readable storage medium of claim 15 wherein first data comprises one from a set, the set comprising:

a prop, clothing worn by the user, an object in a scene captured by the capture device, a second user or users, a sound made by the user, a controller or remote control input, an amount and/or position of light in the scene, user interaction with a touch-sensitive device, and a facial expression of the user.

17. The computer-readable storage medium of claim 15 wherein analyzing the first data and the additional data further comprises:

determining a latency between the first data and the additional data.

18. The computer-readable storage medium of claim 15 wherein the additional data is received from a second user.

19. The computer-readable storage medium of claim 15 wherein the first data and the additional data are received as a result of the user entering the first data and the additional data at substantially the same time.

20. The computer-readable storage medium of claim 15 wherein the additional data comprises a prop, and the prop is manipulated by a plurality of users.