Determine intended motions

• US 8,253,746 B2
• Filed: 05/01/2009
• Issued: 08/28/2012
• Est. Priority Date: 05/01/2009
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Geisner, Kevin, Markovic, Relja, Latta, Stephen Gilchrist, Snook, Gregory Nelson, Tsunoda, Kudo, Bennett, Darren Alexander
• Primary Examiner(s): Brier, Jeffery A
• Application Number:
  US12/434,532
• Publication Number:
  US 20100277489A1
• Time in Patent Office: 1,215 Days
• Field of Search: 345/474, 715/863
• US Class Current: 345/474
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/42 by mapping the input signals into game commands, e.g. mapping the displacement of a stylus on a touch screen to the steering angle of a virtual vehicle
• CPC Class Codes: A63F 13/428 involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes
• CPC Class Codes: A63F 13/57 Simulating properties, behaviour or motion of objects in the game world, e.g. computing tyre load in a car race game A63F13/56 takes precedence
• CPC Class Codes: A63F 13/63 by the player, e.g. authoring using a level editor
• CPC Class Codes: A63F 13/65 automatically by game devices or servers from real world data, e.g. measurement in live racing competition
• CPC Class Codes: A63F 13/833 Hand-to-hand fighting, e.g. martial arts competition A63F13/837 takes precedence
• CPC Class Codes: A63F 2300/1093 using visible light
• CPC Class Codes: A63F 2300/6045 for mapping control signals received from the input arrangement into game commands
• CPC Class Codes: A63F 2300/64 for computing dynamical parameters of game objects, e.g. motion determination or computation of frictional forces for a virtual car
• CPC Class Codes: A63F 2300/6615 using models with different levels of detail [LOD]
• CPC Class Codes: A63F 2300/69 Involving elements of the real world in the game world, e.g. measurement in live races, real video
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06T 13/40 of characters, e.g. humans, animals or virtual beings

Abstract

It may be desirable to apply corrective data to aspects of captured image or the user-performed gesture for display of a visual representation that corresponds to the corrective data. The captured motion may be any motion in the physical space that is captured by the capture device, such as a camera. Aspects of a skeletal or mesh model of a person, that is generated based on the image data captured by the capture device, may be modified prior to animation. The modification may be made to the model generated from image data that represents a target or a target'"'"'s motion, including user gestures, in the physical space. For example, certain joints of a skeletal model may be readjusted or realigned. A model of a target may be modified by applying differential correction, magnetism principles, binary snapping, confining virtual movement to defined spaces, or the like.

Claims

1. A system for modifying data representative of captured motion, the method comprising:

a processor; and

a memory communicatively coupled to the processor when the system is operational, the memory bearing processor-executable instructions that, when executed on the processor, cause the system to at least;

receive image data of a scene, the image data including data representative of captured motion, the image data having been captured with a camera;

generate a model of the captured motion based on the image data;

apply corrective data to at least a portion of the model to correspond to a desired visual representation of the model; and

render a visual representation of the model including the corrected portion.

2. The system of claim 1, wherein the memory further bears processor-executable instructions that, when executed on the processor, cause the system to at least:

compare a variation between the data representative of the captured motion and an error band and, if it is determined from the comparison that the variation is within the error band, triggering the application of corrective data.

3. The system of claim 1, wherein the memory further bears processor-executable instructions that, when executed on the processor, cause the system to at least:

detect an asymmetrical balance of the model representative of the captured motion, the desired visual representation of the model being a visual representation that is more symmetrically balanced than the model representative of the captured motion.

4. The system of claim 1, wherein the memory further bears processor-executable instructions that, when executed on the processor, cause the system to at least:

compare the data representative of the captured motion to a virtual object and detecting a misalignment between the model of the captured motion and the virtual object, the desired visual representation of the model being a visual representation that is aligned with the virtual object.

5. The system of claim 1, wherein the model is a flood model, skeletal model, a depth model, or a mesh human model.

6. The system of claim 1, wherein the instructions that, when executed on the processor, cause the system to at least apply corrective data further cause the system to at least:

apply an attraction from at least the a portion of the model to a virtual magnet.

7. The system of claim 1, wherein the instructions that, when executed on the processor, cause the system to at least apply corrective data further cause the system to at least:

apply differential correction to modify the model to correspond to a virtually defined space.

8. The system of claim 1, wherein the instructions that, when executed on the processor, cause the system to at least apply corrective data further cause the system to at least:

apply binary snapping the at least a portion of the model to a desired position.

9. The system of claim 1, wherein the desired visual representation of the model corresponds to a user accuracy or a user skill setting.

10. The system of claim 1, wherein the memory further bears processor-executable instructions that, when executed on the processor, cause the system to at least:

scan a target to generate the model, wherein the model is a skeletal model; and

wherein the instructions that, when executed on the processor, cause the system at least apply corrective data to the at least a portion of the model further cause the system to at least apply corrective data to the skeletal model.

11. The system of claim 1, wherein the image data comprises data for a series of images that represent the captured motion, the corrective data being applied to the model of the target in each of the series of images.

12. The system of claim 1, wherein the memory further bears processor-executable instructions that, when executed on the processor, cause the system to at least:

detect a similarity between the captured motion and an ideal motion, the desired visual representation being a pre-authored model that represents the ideal motion.

13. A method for rendering a visual representation of a gesture, the method comprising:

receiving image data of a scene, the image data comprising data representative of a user-performed gesture in the physical space, the image data having been captured by a capture device;

mapping the image data to a model representative of the user;

applying corrective data to the model to correspond to a desired visual representation of the model;

mapping the user-performed gesture to the corrected model; and

rendering a visual representation of the corrected model and the gesture mapped to the corrected model on a display device.

14. The method of claim 13, wherein the application of corrective data is triggered by at least one of:

comparing a variation between the data representative of the target and an error band and detecting that the variation is within the error band;

detecting an asymmetrical balance of the model representative of the user;

comparing the data representative of the user to a virtual object and detecting a misalignment between the model of the user and the virtual object.

15. The method of claim 14, wherein the error band is defined by at least one parameter of a gesture filter that corresponds to the user-performed gesture.

16. The method of claim 13, further comprising applying a filter corresponding to the gesture, prior to applying the corrective data, to the received data and identifying the gesture.

17. The method of claim 13, further comprising identifying an intended or expected control from the data representative of the gesture, wherein the corrective data applied to the model corresponds to the identified or expected control.

18. The method of claim 13, further comprising interpreting a series of images that represent the user'"'"'s motion, the corrective data being applied to the model in each of the series of images.

19. The method of claim 13, wherein the model is at least one of a flood model, skeletal model, a depth model, or a mesh human model.

20. The method of claim 13, wherein applying corrective data comprises applying an attraction from at least a portion of the model to a virtual magnet.

21. The method of claim 16, wherein applying corrective data comprises applying differential correction to modify the model to correspond to a virtually defined space.

22. The method of claim 13, wherein applying corrective data comprises binary snapping at least a portion of the model to a desired position.

23. The method of claim 13, wherein the desired visual representation of the model corresponds to a user accuracy or a user skill setting.

24. The method of claim 13, further comprising identifying the user-performed gesture by comparing the corrected model to a gesture filter.

25. The method of claim 13, further comprising:

providing a filter representing the gesture, the filter comprising base information about the gesture;

applying the filter to the image data and determining an output from the base information about the gesture.

26. A method for modifying data representative of captured motion, the method comprising:

receiving image data of a scene, wherein the image data includes data representative of captured motion;

generating a model of the captured motion based on the image data;

identifying a variation between the captured motion and a gesture filter, wherein the gesture filter comprises information for a gesture;

applying corrective data to at least a portion of the model to correspond to the gesture;

identifying the gesture from the corrected model by applying the gesture filter to the corrected model; and

processing an input command invoked by the gesture on a computer system.

27. The method of claim 26, further comprising displaying the corrected model.

28. The method of claim 26, further comprising displaying the model prior to applying the corrective data.

29. The method of claim 26, wherein the gesture controls an aspect of a virtual space.

30. The method of claim 26, wherein identifying a variation between the captured motion and the gesture filters comprises identifying a variation between the data representative of the captured motion and an error band.

31. The method of claim 26, wherein applying corrective data comprises applying an attraction from at least a portion of the model to a virtual magnet.

32. The method of claim 26, wherein applying corrective data comprises applying differential correction to modify the model to correspond to a virtually defined space.

33. The method of claim 26, wherein applying corrective data comprises binary snapping at least a portion of the model to a desired position.

34. The method of claim 26, further comprising:

scanning a target to generate the model, wherein the model is a skeletal model;

wherein applying corrective data to at least a portion of the model comprises applying corrective data to the skeletal model.

35. A computer-readable storage medium excluding signals for adjusting a joint in a model of a target, bearing computer-readable instructions that, when executed on a computer, cause the computer to perform operations comprising:

receiving the image data of a physical space, wherein the image data includes data representative of a target in the physical space;

generating a skeletal model of the target having at least one joint, wherein the joint position corresponds to a virtual 3-dimensional coordinate system that corresponds to the physical space;

identifying a deviation of a joint position of the at least one joint in the model from a desired position of said joint in the virtual 3-dimensional coordinate system, wherein the joint position and desired position are compared in reference to a 3-dimensional coordinate system;

repositioning the joint in a first direction on a Y axis by a dY value if, based on the comparison, a dY is identified between the joint position of the at least one joint in the model and the desired position of said joint in the virtual 3-dimensional coordinate system;

repositioning the joint in a first direction on a Z axis by a dZ value if, based on the comparison, a dZ is identified between the joint position of the at least one joint in the model and the desired position of said joint in the virtual 3-dimensional coordinate system;

repositioning the joint in a first direction on a X axis by a dX value if, based on the comparison, a dX is identified between the joint position of the at least one joint in the model and the desired position of said joint in the virtual 3-dimensional coordinate system; and

rendering a visual representation of the model of the target with at least one repositioned joint.