Interacting with user interface via avatar

• US 8,749,557 B2
• Filed: 06/11/2010
• Issued: 06/10/2014
• Est. Priority Date: 06/11/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Evertt, Jeffrey, Deaguero, Joel, Bennett, Darren, Vance, Dylan, Galloway, David, Markovic, Relja, Latta, Stephen, Garza Santos, Oscar Omar, Geisner, Kevin
• Primary Examiner(s): Tung, Kee M
• Assistant Examiner(s): Du, Haixia
• Application Number:
  US12/814,237
• Publication Number:
  US 20110304632A1
• Time in Patent Office: 1,460 Days
• Field of Search: None
• US Class Current: 345/473
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06T 13/40 of characters, e.g. humans, animals or virtual beings

Abstract

Embodiments are disclosed that relate to interacting with a user interface via feedback provided by an avatar. One embodiment provides a method comprising receiving depth data, locating a person in the depth data, and mapping a physical space in front of the person to a screen space of a display device. The method further comprises forming an image of an avatar representing the person, outputting to a display an image of a user interface comprising an interactive user interface control, and outputting to the display device the image of the avatar such that the avatar faces the user interface control. The method further comprises detecting a motion of the person via the depth data, forming an animated representation of the avatar interacting with the user interface control based upon the motion of the person, and outputting the animated representation of the avatar interacting with the control.

Claims

1. In a computing device, a method of presenting a user interface, the method comprising:

receiving depth data from a depth-sensing camera;

locating a plurality of persons in the depth data;

determining a selected person of the plurality of persons to be in control of the user interface relative to one or more other persons of the plurality of persons from one or more of a posture and a gesture of the selected person in the depth data relative to the one or more other persons of the plurality of persons, the one or more of the posture and the gesture comprising one or more characteristics indicative of an intent of the selected person to assume control of the user interface;

mapping a physical space in front of the selected person to a screen space of a display device;

forming an image of an avatar representing the selected person;

outputting to the display device an image of a user interface, the user interface comprising an interactive user interface control;

outputting to the display device the image of the avatar such that the avatar appears to face the user interface control;

detecting a motion of the selected person via the depth data;

forming an animated representation of the avatar interacting with the user interface control based upon the motion of the selected person;

biasing a movement of the animated representation of the avatar in a direction toward the user interface control compared to the motion of the selected person as the avatar becomes closer to the user interface control; and

outputting to the display device the animated representation of the avatar interacting with the user interface control.

2. The method of claim 1, wherein outputting the image of the avatar comprises outputting the image of the avatar such that the avatar appears to face the selected person.

3. The method of claim 1, wherein detecting the motion of the selected person comprises detecting a motion of a limb of the selected person, and wherein forming an animated representation of the avatar comprises forming an animated representation of a limb of the avatar moving relative to the user interface control.

4. The method of claim 3, wherein the biasing the movement of the animated representation of the avatar comprises biasing a motion of the limb of the avatar in a direction toward the user interface control compared to the motion of the limb of the selected person as the limb of the avatar becomes closer to the user interface control.

5. The method of claim 1, wherein mapping the physical space in front of the selected person comprises performing skeletal tracking of the selected person, determining an orientation and size of the selected person from the skeletal tracking of the selected person, and then determining an estimated extent of limb motion of the selected person from a perspective of the depth-sensing camera.

6. The method of claim 1, wherein outputting to the display device the image of the avatar comprises outputting images of a plurality of avatars such that a selected avatar representing the selected person is visually distinguished from other avatars representing other persons.

7. The method of claim 6, wherein the selected avatar is visually distinguished by one or more of size, location in the image, and posture relative to the other avatars.

8. The method of claim 6, further comprising detecting an input configured to cause a change of person in control of the user interface, and in response, changing which avatar is visually distinguished from the other avatars, wherein the input includes the gesture.

9. The method of claim 8, wherein detecting an input configured to cause a change of person in control of the user interface further comprises detecting the input from speech data.

10. The method of claim 1, wherein the image of the avatar is configured to be displayed as appearing to be located behind the user interface control and facing the selected person.

11. A computing device, comprising:

a processor; and

memory comprising instructions executable by the processor to;

receive depth data from a depth-sensing camera;

locate a person in the depth data;

map a physical space in front of the person to a screen space of a display device to form a mapping of the physical space in front of the person;

form an image of an avatar representing the person;

output to the display device an image of a user interface, the user interface comprising an interactive user interface control;

output to the display device the image of the avatar such that the avatar appears to face outwardly from the display device toward the person;

detect a motion of a limb of the person via the depth data;

form an animated representation of a limb of the avatar moving based upon the motion of the limb of the person and the mapping of the physical space in front of the person;

form an animated representation of the limb of the avatar interacting with the user interface control;

bias a movement of the animated representation of the limb of the avatar in a direction toward the user interface control and away from a motion of the limb of the person as the limb of the avatar becomes closer to the user interface control; and

output to the display device the animated representation of the avatar interacting with the user interface control, the animated representation comprising a hand of the avatar closing over the user interface control to form a representation of a hand that is closed over the user interface control and grips the user interface control.

12. The computing device of claim 11, wherein the person is a selected person of a plurality of persons located in the depth data, and wherein the image of the avatar representing the selected person also comprises images of a plurality of other avatars representing other persons located in the depth data.

13. The computing device of claim 12, wherein the instructions are executable to visually distinguish the image of the avatar representing the selected person from the other avatars by one or more of size, location in the image, and posture relative to the other avatars.

14. The computing device of claim 12, wherein the instructions are executable to detect an input configured to cause a change of person in control of the user interface, and in response, to change which avatar is visually distinguished from the other avatars.

15. The computing device of claim 11, wherein the instructions are executable to map the physical space in front of the person by performing skeletal tracking of the person, determining an orientation and size of the person from the skeletal tracking of the person, and then determining an estimated extent of limb motion of the person from a perspective of the depth-sensing camera.

16. A computer-readable storage device comprising instructions stored thereon that are executable by a computing device to:

receive depth data from a depth-sensing camera;

locate a plurality of persons in the depth data;

determine a selected person of the plurality of persons to be in control of a user interface;

map a physical space in front of the selected person to a space on a screen of a display device to form a mapping of the physical space in front of the selected person;

form an image of a selected avatar representing the selected person and an image of an additional avatar representing a different person of the plurality of persons;

output to the display device an image of a user interface comprising an interactive user interface control;

output to the display device the image of the selected avatar such that the selected avatar appears to face outwardly from the display device toward the selected person, the selected avatar being displayed with an interactive pose indicating an intent of the user to provide control input to the user interface;

output to the display device the image of the additional avatar representing the different person, the additional avatar being displayed with a non-interactive pose;

detect a motion of a limb of the selected person via the depth data;

form an animated representation of an arm of the avatar moving based upon the motion of the limb of the selected person and the mapping of the physical space in front of the selected person;

as the limb of the avatar moves closer toward the user interface control, bias a movement of the limb of the avatar in a direction toward the user interface control compared to the motion of the limb of the selected person; and

output to the display device an animated representation of a hand of the avatar interacting with the user interface control, the animated representation comprising the hand of the avatar closing over the user interface control to form a representation of a hand that is closed over the user interface control and grips the user interface control.

17. The computer-readable storage device of claim 16, wherein the instructions are further executable to form images of additional avatars for each of the plurality of persons and output to the display device the images of additional avatars for each of the plurality of persons in non-interactive poses.

18. The computer-readable storage device of claim 17, wherein the instructions are executable to visually distinguish the image of the avatar representing the selected person from other avatars by one or more of size compared to the other avatars and location in the image relative to the other avatars.

19. The computer-readable storage device of claim 17, wherein the instructions are executable to detect an input configured to cause a change of person in control of the user interface, and, in response, to change which avatar is visually distinguished from other avatars.

20. The computer-readable storage device of claim 16, wherein the instructions are executable to map the physical space in front of the selected person by performing skeletal tracking of the selected person, determining an orientation and size of the selected person from the skeletal tracking of the selected person, and then determining an estimated extent of limb motion of the selected person from a perspective of the depth-sensing camera.