Show body position

• US 9,298,263 B2
• Filed: 10/27/2010
• Issued: 03/29/2016
• Est. Priority Date: 05/01/2009
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Geisner, Kevin, Markovic, Relja, Latta, Stephen Gilchrist, Snook, Gregory Nelson, Bennett, Darren
• Primary Examiner(s): ABDUL-ALI, OMAR R
• Application Number:
  US12/913,345
• Publication Number:
  US 20110035666A1
• Time in Patent Office: 1,980 Days
• Field of Search: 715/863
• US Class Current: 1/1
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06V 40/23 Recognition of whole body movements, e.g. for sport training

Abstract

A capture device may capture a user'"'"'s motion and a display device may display a model that maps to the user'"'"'s motion, including gestures that are applicable for control. A user may be unfamiliar with a system that maps the user'"'"'s motions or not know what gestures are applicable for an executing application. A user may not understand or know how to perform gestures that are applicable for the executing application. Providing visual feedback representing instructional gesture data to the user can teach the user how to properly gesture. The visual feedback may be provided in any number of suitable ways. For example, visual feedback may be provided via ghosted images, player avatars, or skeletal representations. The system can process prerecorded or live content for displaying visual feedback representing instructional gesture data. The feedback can portray the deltas between the user'"'"'s actual position and the ideal gesture position.

Claims

1. A method for providing feedback, the method comprising:

capturing a depth image;

identifying data in the depth image representative of a target'"'"'s motion or position in a physical space;

comparing the identified data to a plurality of gesture filters, wherein each of said plurality of gesture filters comprises one or more parameters defining a gesture along with associated thresholds and ranges for said one or more parameters;

identifying a confidence rating output for each of the comparisons to the plurality of gesture filters, wherein each of the plurality of confidence rating outputs indicates a likelihood that the identified data corresponds to a gesture of a respective gesture filter;

in response to determining that no confidence rating output is above a threshold level of acceptance, predicting an intended gesture based on a comparison of the confidence rating outputs for identified data of failed executions of gestures;

determining that a variation between the identified data for a failed execution of a gesture and the one or more filter parameters for the intended gesture is below a threshold level; and

rendering feedback on a display, the feedback comprising a visual representation of the target'"'"'s motion or position and an example of a correct performance of the intended gesture for which the variation is below the threshold level.

2. The method of claim 1 further comprising:

determining that the depth image includes a human target;

scanning the human target;

generating a model of the human target based on the scan; and

rendering a visual representation of the model that maps to the human target.

3. The method in accordance with claim 1, wherein the feedback comprises live content.

4. The method in accordance with claim 1, further comprising:

predicting an intended gesture from the identified data; and

wherein rendering feedback comprises;

rendering feedback applicable to the predicted intended gesture.

5. The method in accordance with claim 1, wherein the feedback comprises visual or auditory feedback.

6. The method in accordance with claim 1, wherein the feedback comprises a visual representation in a form of a skeletal representation, a ghosted image, or a player avatar.

7. The method in accordance with claim 1, wherein rendering feedback comprises rendering:

a visual representation of the feedback superimposed over the captured depth image of the target,

side-by-side images of the captured depth image of the target and a visual representation of the feedback,

a visual representation of the captured depth image of the target and corrected or highlighted feedback that indicates a delta between a position depicted in the captured depth image of the target and an ideal position,

an indication of a position of a body part of the target that differs from a position of a body part of an example of a correct performance of the intended gesture and that is separate from both the visual example of the target'"'"'s motion or position and the example of a correct performance of the intended gesture, or

a visual representation of the example of the correct performance on a portion of a display space, wherein the portion is smaller than the total display space.

8. The method in accordance with claim 1, further comprising:

identifying a trigger that initiates the rendering of the feedback, the trigger comprising an identification of a failure in an execution of an intended gesture, an initiation of an application, or a request for training of an intended gesture.

9. The method in accordance with claim 1, further comprising:

determining a control to perform in an comparison application based on a gesture, the gesture comprising a gesture of a user in a physical space that is mapped to a skeletal model.

10. A system for providing visual feedback, comprising:

a processor; and

a memory communicatively coupled to the processor when the system is operational, the memory bearing processor-executable instructions that, when executed on the processor, cause the system at least to;

identify data representative of a target'"'"'s motion or position in a physical space;

compare the identified data to a plurality of gesture filters, wherein each of said plurality of gesture filters comprises one or more parameters defining a gesture along with associated thresholds and ranges for said one or more parameters;

identify a confidence rating output for each of the comparisons to the plurality of gesture filters, wherein each of the plurality of confidence rating outputs indicates a likelihood that the identified data corresponds to a gesture of a respective gesture filter;

in response to determining that no confidence rating output is above a threshold level of acceptance, predict an intended gesture based on a comparison of the confidence rating outputs for identified data of failed executions of gestures;

determine that a variation between the captured data for a failed execution of a gesture and the filter parameters for the intended gesture is below a threshold level; and

provide feedback comprising a visual representation of the target'"'"'s motion or position and an example of a correct performance of the intended gesture for which the variation is below the threshold level.

11. The system in accordance with claim 10, wherein the memory further comprises processor-executable instructions that, when executed on the processor, cause the system at least to:

receive feedback representing the example of a correct performance of the intended gesture over a network.

12. The system in accordance with claim 10, wherein the memory further comprises processor-executable instructions that, when executed on the processor, cause the system at least to:

determine that the depth image includes a human target, and

in response to determining that the depth image includes a human target, scan the human target, track a skeletal model, and render a visual representation that maps to the skeletal model.

13. The system in accordance with claim 10, wherein the memory further comprises processor-executable instructions that, when executed on the processor, cause the system at least to:

select the visual feedback to render from live content received over a network.

14. The system in accordance with claim 10, wherein the memory further comprises processor-executable instructions that, when executed on the processor, cause the system at least to:

predict an intended gesture from the identified data; and

render visual feedback that represents the example of a correct performance of the predicted intended gesture.

15. The system in accordance with claim 10, wherein the memory further comprises processor-executable instructions that, when executed on the processor, cause the system at least to:

render the visual feedback in slow motion.

16. The system in accordance with claim 10, wherein the memory further comprises processor-executable instructions that, when executed on the processor, cause the system at least to:

render a visual representation that maps to the target'"'"'s motion or position in the physical space, wherein the visual representation is based on a skeletal representation.

17. The system in accordance with claim 10, wherein the visual feedback comprises a visual representation of the example of a correct performance of the intended gesture in a form of a skeletal representation, a ghosted image, or a player avatar.

18. The system in accordance with claim 10, wherein the instructions that, when executed on the processor cause the system to at least render visual feedback, further cause the system at least to:

render a visual representation of the example of a correct performance of the intended gesture superimposed by a visual representation of the target'"'"'s motion or position in the physical space;

render side-by-side images of a visual representation of the target'"'"'s motion or position in the physical space and a visual representation of the example of a correct performance of the intended gesture;

render a visual representation of the target'"'"'s motion or position in the physical space, and corrected or highlighted feedback indicative of an error, such as a delta between an at least one gesture filter actual position and an ideal position;

an indication of a position of a body part of the target that differs from a position of a body part of an example of a correct performance of the intended gesture and that is separate from both the visual example of the target'"'"'s motion or position and the example of a correct performance of the intended gesture;

or

render a visual representation of the example of a correct performance of the intended gesture on a portion of a display space, wherein the portion is smaller than a total display space.

19. The system in accordance with claim 10, wherein the memory further comprises processor-executable instructions that, when executed on the processor, cause the system at least to:

identify a trigger that initiates the rendering of the visual feedback, wherein the trigger is indicative of a failure in an execution of the target'"'"'s motion or position in the physical space, an initiation of an application, or a request for training.

20. A computer-readable storage device bearing computer-executable instructions for determining whether to provide feedback that, when executed upon a computer, cause actions comprising:

receiving image data for a scene, wherein the image data comprises depth data representative of a target'"'"'s motion or position in a physical space;

comparing the depth data representative of the target'"'"'s motion or position in the physical space to a plurality of gesture filters wherein each of said plurality of gesture filters comprises one or more parameters defining a gesture along with associated thresholds and ranges for said one or more parameters;

identifying a confidence rating output for each of the comparisons to the plurality of gesture filters, wherein each of the plurality of confidence rating outputs indicates a likelihood that the depth data representative of the target'"'"'s motion or position corresponds to a gesture of a respective gesture filter;

in response to determining that no confidence rating output is above a threshold level of acceptance, predicting an intended gesture based on a comparison of the confidence rating outputs for identified data of failed executions of gestures;

determining that a variation between the identified data for a failed execution of a gesture and the one or more filter parameters for the intended gesture is below a threshold level; and

rendering feedback on a display, the feedback comprising a visual representation of the target'"'"'s motion or position and an example of a correct performance of the intended gesture for which the variation is below the threshold level.