Translating user motion into multiple object responses

• US 9,075,434 B2
• Filed: 08/20/2010
• Issued: 07/07/2015
• Est. Priority Date: 08/20/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Santos, Oscar Omar Garza, Haigh, Matthew, Vuchetich, Christopher, Hindle, Ben, Bennett, Darren A.
• Primary Examiner(s): Pesin, Boris
• Assistant Examiner(s): HUERTAS TORRES, AYESHA E
• Application Number:
  US12/859,995
• Publication Number:
  US 20120047468A1
• Time in Patent Office: 1,782 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042

Abstract

A system for translating user motion into multiple object responses of an on-screen object based on user interaction of an application executing on a computing device is provided. User motion data is received from a capture device from one or more users. The user motion data corresponds to user interaction with an on-screen object presented in the application. The on-screen object corresponds to an object other than an on-screen representation of a user that is displayed by the computing device. The user motion data is automatically translated into multiple object responses of the on-screen object. The multiple object responses of the on-screen object are simultaneously displayed to the users.

Claims

1. A method for translating user motion into one or more object responses, based on user interaction with an application executing on a computing device, comprising:

displaying a non-interactive scene on a display under control of a computing device being controlled by an executing application, the non-interactive scene including an on-screen object;

receiving user motion data for one or more users from a sensor;

determining if the user motion data matches one or more predefined gestures;

if the user motion data matches one or more predefined gestures, automatically translating the user motion data into one or more object responses controlling how the on-screen object moves but not altering an outcome position of the on-screen object in the non-interactive scene,

the on-screen object corresponding to an object other than an on-screen representation of a user, and the automatically translating the user motion data into one or more object responses controlling how the on-screen object moves but not altering the outcome position of the on-screen object in the non-interactive scene further comprises;

responsive to determining the user motion data matches one or more predefined gestures, accessing one or more object responses of the on-screen object corresponding to the one or more predefined gestures by accessing a data structure correlating the one or more predefined gestures to the one or more object responses, and

implementing the one or more object responses, the implementing comprising mapping a model representation of the user into an object model representation of the on-screen object; and

simultaneously displaying the one or more object responses of the on-screen object based on the translating.

2. The method of claim 1, wherein:

at least one of the one or more object responses includes an audio response of the on-screen object and a visual response of the on-screen object.

3. The method of claim 1, wherein:

automatically translating the user motion data into one or more object responses controlling how the on-screen object moves is based on at least one of a velocity or an acceleration determined from the user motion data.

4. The method of claim 1, wherein receiving the user motion data from one or more users further comprises:

receiving a first user motion data from a first user and second user motion data from a second user.

5. The method of claim 4 further comprising:

automatically translating the first user motion data into a first object response of the on-screen object;

automatically translating the second user motion data into a second object response of the on-screen object; and

simultaneously displaying the first object response and the second object response of the on-screen object to the first user and the second user.

6. The method of claim 5, wherein:

the second object response is different from the first object response when the second user motion data is different from the first user motion data.

7. The method of claim 5, wherein:

the second object response is an amplification of the first object response, when the second user motion data is identical to the first user motion data.

8. The method of claim 1, wherein:

the model representation of the user is a skeletal model representation of the user.

9. The method of claim 1, further comprising:

in response to receiving the user motion data, displaying an animation of an object performing the user motion in addition to displaying the one or more object responses of the on-screen object.

10. The method of claim 1, wherein:

the on-screen object comprises an in-animate object.

11. An apparatus for trigging object response based on user interaction of an application, comprising:

a sensor operable to capture user motion data;

one or more computing devices operable to be controlled by an application during execution of the application and being communicatively connected to the sensor to receive user motion data for one or more users;

a display operable to be controlled by the one or more computing devices for displaying a non-interactive scene, the non-interactive scene including an on-screen object, the on-screen object corresponding to an object other than an on-screen representation of a user;

the one or more computing devices operable to determine if the user motion data matches one or more predefined gestures and if the user motion data matches one or more predefined gestures, the one or more computing devices operable to automatically translate the user motion data into one or more object responses for controlling how the on-screen object moves but not for altering an outcome position of the on-screen object in the non-interactive scene,

the one or more computing devices operable to automatically translate the user motion data into one or more object responses for controlling how the on-screen object moves but not for altering the outcome position of the on-screen object in the non-interactive scene further comprises;

responsive to determining the user motion data matches one or more predefined gestures, the one or more computing devices are operable to access one or more object responses of the on-screen object corresponding to the one or more predefined gestures through access to a data structure correlating the one or more predefined gestures to the one or more object responses, and

the one or more computing devices are operable to implement the one or more object responses, the one or more computing devices are operable to implement the one or more object responses further comprises the one or more computing devices are operable to map a model representation of the user into an object model representation of the on-screen object; and

the one or more computing devices being operable to control the display to display the one or more object responses of the on-screen object based at least in part on the translating.

12. One or more processor readable storage devices having processor readable code embodied on the one or more processor readable storage devices, the processor readable code for programming one or more processors to perform a method comprising:

displaying a non-interactive scene on a display under control of a computing device being controlled by an executing application, the non-interactive scene including an on-screen object;

receiving user motion data for one or more users from a sensor;

determining if the user motion data matches one or more predefined gestures;

if the user motion data matches one or more predefined gestures, automatically translating the user motion data into one or more object responses controlling how the on-screen object moves but not altering an outcome position of the on-screen object in the non-interactive scene,

the on-screen object corresponding to an object other than an on-screen representation of a user, and the automatically translating the user motion data into one or more object responses controlling how the on-screen object moves but not altering the outcome position of the on-screen object in the non-interactive scene further comprises;

responsive to determining the user motion data matches one or more predefined gestures, accessing one or more object responses of the on-screen object corresponding to the one or more predefined gestures by accessing a data structure correlating the one or more predefined gestures to the one or more object responses, and

implementing the one or more object responses, the implementing comprising mapping a model representation of the user into an object model representation of the on-screen object; and

simultaneously displaying the one or more object responses of the on-screen object based on the translating.

13. The one or more processor readable storage devices of claim 12 wherein receiving user motion data for one or more users from a sensor further comprises:

receiving first user motion data for a first user from one or more sensors, the first user motion data corresponding to a first user interaction with the on-screen object;

receiving a second user motion data for a second user from the sensor, the second user motion data corresponding to a second user interaction with the on-screen object;

wherein automatically translating the user motion data into one or more objects controlling how the on-screen object moves further comprises translating the first user motion data into a first motion response of the on-screen object and translating the second user motion data into a second motion response of the on-screen object; and

wherein simultaneously displaying the one or more object responses of the on-screen object based on the automatically translating further comprises simultaneously displaying the first motion response and the second motion response to the first user and the second user.

14. One or more processor readable storage devices of claim 13, wherein:

the second motion response is different from the first object response when the second user motion data is different from the first user motion data.

15. One or more processor readable storage devices of claim 13, wherein:

the second motion response is an amplification of the first motion response, when the second user motion data is identical to the first user motion data.

16. One or more processor readable storage devices of claim 13, wherein:

the sensor includes a depth camera; and

the first user motion data and the second user motion data include depth information from the depth camera.