Determine intended motions

• US 8,451,278 B2
• Filed: 08/03/2012
• Issued: 05/28/2013
• Est. Priority Date: 05/01/2009
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Geisner, Kevin, Markovic, Relja, Latta, Stephen Gilchrist, Snook, Gregory Nelson, Tsunoda, Kudo, Bennett, Darren Alexander
• Primary Examiner(s): Brier, Jeffery A
• Application Number:
  US13/566,939
• Publication Number:
  US 20120293518A1
• Time in Patent Office: 298 Days
• Field of Search: 345/474, 715/863
• US Class Current: 345/474
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/42 by mapping the input signals into game commands, e.g. mapping the displacement of a stylus on a touch screen to the steering angle of a virtual vehicle
• CPC Class Codes: A63F 13/428 involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes
• CPC Class Codes: A63F 13/57 Simulating properties, behaviour or motion of objects in the game world, e.g. computing tyre load in a car race game A63F13/56 takes precedence
• CPC Class Codes: A63F 13/63 by the player, e.g. authoring using a level editor
• CPC Class Codes: A63F 13/65 automatically by game devices or servers from real world data, e.g. measurement in live racing competition
• CPC Class Codes: A63F 13/833 Hand-to-hand fighting, e.g. martial arts competition A63F13/837 takes precedence
• CPC Class Codes: A63F 2300/1093 using visible light
• CPC Class Codes: A63F 2300/6045 for mapping control signals received from the input arrangement into game commands
• CPC Class Codes: A63F 2300/64 for computing dynamical parameters of game objects, e.g. motion determination or computation of frictional forces for a virtual car
• CPC Class Codes: A63F 2300/6615 using models with different levels of detail [LOD]
• CPC Class Codes: A63F 2300/69 Involving elements of the real world in the game world, e.g. measurement in live races, real video
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06T 13/40 of characters, e.g. humans, animals or virtual beings

Abstract

It may be desirable to apply corrective data to aspects of captured image or the user-performed gesture for display of a visual representation that corresponds to the corrective data. The captured motion may be any motion in the physical space that is captured by the capture device, such as a camera. Aspects of a skeletal or mesh model of a person, that is generated based on the image data captured by the capture device, may be modified prior to animation. The modification may be made to the model generated from image data that represents a target or a target'"'"'s motion, including user gestures, in the physical space. For example, certain joints of a skeletal model may be readjusted or realigned. A model of a target may be modified by applying differential correction, magnetism principles, binary snapping, confining virtual movement to defined spaces, or the like.

Claims

1. A system for modifying data representative of captured motion, the system comprising:

a processor; and

a memory communicatively coupled to the processor when the system is operational, the memory bearing processor-executable instructions that, when executed on the processor, cause the system to at least;

receive image data of a scene, the image data including data representative of captured motion, the image data having been captured with a camera;

generate a model of the captured motion based on the image data;

modify at least a portion of a size of the model to constrain the model to an avatar; and

render the avatar using the modified model.

2. The system of claim 1, wherein the captured motion corresponds to a first user and the image data includes data representative of a second captured motion of a second user, and wherein the memory further bears processor-executable instructions that, when executed on the processor, cause the system to at least:

generate a second model based of the second captured motion based on the image data;

modify at least a portion of a size of the second model to constrain the second model to a second avatar, a size of the avatar and a size of the second avatar being the same; and

render the second avatar based on the modified second model.

3. The system of claim 1, wherein the memory further bears processor-executable instructions that, when executed on the processor, cause the system to at least:

detect an asymmetrical balance of the model representative of the captured motion, the rendered avatar being a visual representation that is more symmetrically balanced than the model representative of the captured motion.

4. The system of claim 1, wherein the memory further bears processor-executable instructions that, when executed on the processor, cause the system to at least:

compare the data representative of the captured motion to a virtual object and detecting a misalignment between the model of the captured motion and the virtual object, the rendered avatar being a visual representation that is aligned with the virtual object.

5. The system of claim 1, wherein the model is a flood model, skeletal model, a depth model, or a mesh human model.

6. The system of claim 1, wherein the instructions that, when executed on the processor, cause the system to at least apply corrective data further cause the system to at least:

apply an attraction from at least the a portion of the model to a virtual magnet.

7. The system of claim 1, wherein the instructions that, when executed on the processor, cause the system to at least apply corrective data further cause the system to at least:

apply differential correction to modify the model to correspond to a virtually defined space.

8. The system of claim 1, wherein the instructions that, when executed on the processor, cause the system to at least apply corrective data further cause the system to at least:

apply binary snapping the at least a portion of the model to a desired position.

9. The system of claim 1, wherein the memory further bears processor-executable instructions that, when executed on the processor, cause the system to at least:

scan a target to generate the model, wherein the model is a skeletal model; and

wherein the instructions that, when executed on the processor, cause the system at least apply corrective data to the at least a portion of the model further cause the system to at least apply corrective data to the skeletal model.

10. The system of claim 1, wherein the image data comprises data for a series of images that represent the captured motion, the corrective data being applied to the model of the target in each of the series of images.

11. The system of claim 1, wherein the memory further bears processor-executable instructions that, when executed on the processor, cause the system to at least:

detect a similarity between the captured motion and an ideal motion, the rendered avatar being a pre-authored model that represents the ideal motion.

12. A method for rendering a visual representation of a gesture, the method comprising:

receiving image data of a scene, the image data comprising data representative of a user-performed gesture in the physical space, the image data having been captured by a capture device;

mapping the image data to a model representative of the user;

modifying at least a portion of a size of the model to constrain the model to an avatar;

mapping the user-performed gesture to modified model; and

rendering the avatar corresponding to the modified model and the gesture mapped to the modified model on a display device.

13. The method of claim 12, wherein the application of corrective data is triggered by at least one of:

comparing a variation between the data representative of the target and an error band and detecting that the variation is within the error band;

detecting an asymmetrical balance of the model representative of the user; and

comparing the data representative of the user to a virtual object and detecting a misalignment between the model of the user and the virtual object.

14. The method of claim 13, wherein the error band is defined by at least one parameter of a gesture filter that corresponds to the user-performed gesture.

15. The method of claim 12, further comprising:

applying a filter corresponding to the gesture, prior to applying the corrective data, to the received data and identifying the gesture.

16. The method of claim 12, further comprising:

identifying an intended or expected control from the data representative of the gesture, wherein the corrective data applied to the model corresponds to the identified or expected control.

17. The method of claim 12, wherein the rendered avatar corresponds to a user accuracy or a user skill setting.

18. The method of claim 12, further comprising:

identifying the user-performed gesture by comparing the digital representation of the model to a gesture filter.

19. The method of claim 12, further comprising:

providing a filter representing the gesture, the filter comprising base information about the gesture;

applying the filter to the image data and determining an output from the base information about the gesture.

20. A computer-readable storage device for adjusting a joint in a model of a target, bearing computer-readable instructions that, when executed on a computer, cause the computer to perform operations comprising:

receiving the image data of a physical space, wherein the image data includes data representative of a target in the physical space;

generating a skeletal model of the target having at least one joint, wherein the joint position corresponds to a virtual 3-dimensional coordinate system that corresponds to the physical space;

identifying a deviation of a length of a bone of the skeletal model that is too long or too short to fit in a corresponding avatar;

adjusting the length of the bone of the model to fit within the avatar; and

rendering an avatar for the target based on the adjusted length of the bone in the model.