AUTOMATICALLY TRACKING USER MOVEMENT IN A VIDEO CHAT APPLICATION

• US 20120092445A1
• Filed: 10/14/2010
• Published: 04/19/2012
• Est. Priority Date: 10/14/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: McDowell, Brian, Apfel, Darren
• Granted Patent: US 9,628,755 B2
• Time in Patent Office: Days
• US Class Current: 348/14.16
• CPC Class Codes: A63F 2300/572 Communication between players during game play of non game information, e.g. e-mail, chat, file transfer, streaming of audio and streaming of video
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/248 involving reference images or patches
• CPC Class Codes: G06V 10/145 Illumination specially adapted for pattern recognition, e.g. using gratings
• CPC Class Codes: G06V 20/64 Three-dimensional objects
• CPC Class Codes: G06V 40/167 using comparisons between temporally consecutive images
• CPC Class Codes: H04N 13/204 using stereoscopic image cameras stereoscopic photography G03B35/00
• CPC Class Codes: H04N 7/147 Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals selecting H04Q

Abstract

A system for automatically tracking movement of a user participating in a video chat application executing in a computing device is disclosed. A capture device connected to the computing device captures a user in a field of view of the capture device and identifies a sub-frame of pixels identifying a position of the head, neck and shoulders of the user in a capture frame of a capture area. The sub-frame of pixels is displayed to a remote user at a remote computing system who is participating in the video chat application with the user. The capture device automatically tracks the position of the head, neck and shoulders of the user as the user moves to a next location within the capture area. A next sub-frame of pixels identifying a position of the head, neck and shoulders of the user in the next location is identified and displayed to the remote user at the remote computing device.

Claims

1. A method for automatically tracking movement of a user participating in a video chat application executing in a computing device, the method comprising:

receiving a capture frame comprising one or more depth images of a capture area from a depth camera connected to a computing device;

determining if the capture frame includes a user in a first location in the capture area;

identifying a sub-frame of pixels in the capture frame, the sub-frame of pixels identifying a position of the head, neck and shoulders of the user in the capture frame;

displaying the sub-frame of pixels to a remote user at a remote computing system;

automatically tracking the position of the head, neck and shoulders of the user to a next location within the capture area;

identifying a next sub-frame of pixels, the next sub-frame of pixels identifying a position of the head, neck and shoulders of the user in the next location, wherein the next sub-frame of pixels is included in a next capture frame of the capture area; and

displaying the next sub-frame of pixels to the remote user in the remote computing system.

2. The method of claim 1, wherein displaying the sub-frame of pixels and the next sub-frame of pixels to the remote user further comprises:

encoding the sub-frame of pixels and the next sub-frame of pixels into a video communication data stream to the remote user at the remote computing device.

3. The method of claim 1, wherein displaying the sub-frame of pixels and the next sub-frame of pixels to the remote user further comprises:

automatically centering the position of the head, neck and shoulders of the user in the sub-frame of pixels and the next sub-frame of pixels prior to the displaying.

4. The method of claim 1, wherein displaying the sub-frame of pixels and the next sub-frame of pixels to the remote user further comprises:

automatically altering the size of the head, neck and shoulders of the user in the sub-frame of pixels and the next sub-frame of pixels so that the size of the head, neck and shoulders of the user in the sub-frame of pixels and the next sub-frame of pixels is approximately constant prior to the displaying.

5. The method of claim 1, wherein:

the size of the sub-frame of pixels in the capture frame is different from the size of the next sub-frame of pixels in the next capture frame.

6. The method of claim 1, wherein automatically tracking the position of the head, neck and shoulders of the user to a next location within the capture area further comprises:

automatically tracking lateral movement of the user to the next location in the capture area.

7. The method of claim 1, wherein automatically tracking the position of the head, neck and shoulders of the user to a next location within the capture area further comprises:

automatically tracking movement of the user to different distances from the capture device.

8. The method of claim 1, further comprising:

generating a skeletal model of the user based on the one or more depth images, wherein the head, neck and shoulders of the user is identified using the skeletal model.

9. The method of claim 1, further comprising:

detecting and identifying more than one user in the capture area, determining if at least one of the sub-frame of pixels or the next sub-frame of pixels can include each of the users, automatically adjusting the sub-frame of pixels or the next sub-frame of pixels to include the head, neck and shoulders of each of the users and displaying the sub-frame of pixels or the next sub-frame of pixels to the remote user.

10. The method of claim 1, further comprising:

detecting and identifying more than one user in the capture area and identifying individual sub-frames of pixels to include each of the users.

12. The method of claim 9, further comprising:

identifying each of the users as a group of users in the sub-frame of pixels or the next sub-frame of pixels when the users are sufficiently close together, for a minimum period of time.

13. The method of claim 12, further comprising:

identifying the users in at least two individual sub-frames of pixels when the users are no longer sufficiently close together, for a minimum period of time.

14. The method of claim 1, further comprising:

identifying a sub-frame of pixels containing the head, neck and shoulders of more than one user, determining if a voice input is detected from at least one user, automatically adjusting the sub-frame of pixels to include the one or more users providing the voice input and displaying the sub-frame of pixels containing the one or more users providing the voice input to the remote user at the remote computing device.

11. The method of claim 11, further comprising:

displaying the individual sub-frames of pixels to the remote user.

15. One or more processor readable storage devices having processor readable code embodied on said one or more processor readable storage devices, the processor readable code for programming one or more processors to perform a method comprising:

receiving a capture frame comprising one or more depth images of a capture area from a depth camera connected to a computing device;

determining if the capture frame includes a user in a first location in the capture area;

identifying a sub-frame of pixels in the capture frame, the sub-frame of pixels identifying a position of the head, neck and shoulders of the user in the capture frame;

displaying the sub-frame of pixels to a remote user at a remote computing system;

receiving a next capture frame comprising one or more depth images of the capture area from a depth camera;

automatically tracking movement of one or more users within the capture area;

determining if the next capture frame includes the one or more users in a next location in the capture area based on the tracking;

identifying a next sub-frame of pixels containing the head, neck and shoulders of the one or more users in the next capture frame; and

displaying the next sub-frame of pixels to the remote user at the remote computing system.

16. One or more processor readable storage devices of claim 15, further comprising:

automatically centering the position of the head, neck and shoulders of the one or more users in the next sub-frame of pixels prior to the displaying.

17. One or more processor readable storage devices of claim 15, further comprising:

identifying the one or more users as a group of users in the next sub-frame of pixels when the one or more users are sufficiently close together, for a minimum period of time.

18. One or more processor readable storage devices of claim 17, further comprising:

automatically tracking movement of the group of users to a next location within the capture area by determining the position of the head, neck and shoulders of the group of users in the capture area.

19. An apparatus for automatically tracking movement of one or more users participating in a video chat application, comprising:

a depth camera; and

a computing device connected to the depth camera to receive a capture frame comprising one or more depth images of a capture area from a depth camera connected to a computing device;

determine if the capture frame includes more than one user in a first location in the capture area;

identify a sub-frame of pixels in the capture frame, the sub-frame of pixels identifying a position of the head, neck and shoulders of each of the users in the capture frame;

display the sub-frame of pixels to a remote user at a remote computing system;

determine if a voice input is received from at least one user in the first sub-frame of pixels, identify one or more of the users providing the voice input;

automatically adjust the sub-frame of pixels to include the one or more users providing the voice input and display the sub-frame of pixels containing the one or more users providing the voice input to a remote user at a remote computing device.

20. The apparatus of claim 19, wherein:

the computing device displays the one or more users providing the voice input in one or more individual sub-frames of pixels to the remote user at the remote computing device.