Automatically tracking user movement in a video chat application

• US 9,628,755 B2
• Filed: 10/14/2010
• Issued: 04/18/2017
• Est. Priority Date: 10/14/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: McDowell, Brian, Apfel, Darren
• Primary Examiner(s): Nguyen, Duc
• Assistant Examiner(s): PATEL, YOGESHKUMAR G
• Application Number:
  US12/904,980
• Publication Number:
  US 20120092445A1
• Time in Patent Office: 2,378 Days
• Field of Search: 348 1401, 348 1404, 348 1408, 348 1416
• CPC Class Codes: A63F 2300/572 Communication between players during game play of non game information, e.g. e-mail, chat, file transfer, streaming of audio and streaming of video
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/248 involving reference images or patches
• CPC Class Codes: G06V 10/145 Illumination specially adapted for pattern recognition, e.g. using gratings
• CPC Class Codes: G06V 20/64 Three-dimensional objects
• CPC Class Codes: G06V 40/167 using comparisons between temporally consecutive images
• CPC Class Codes: H04N 13/204 using stereoscopic image cameras stereoscopic photography G03B35/00
• CPC Class Codes: H04N 7/147 Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals selecting H04Q

Abstract

A system for automatically tracking movement of a user participating in a video chat application executing in a computing device is disclosed. A capture device connected to the computing device captures a user in a field of view of the capture device and identifies a sub-frame of pixels identifying a position of the head, neck and shoulders of the user in a capture frame of a capture area. The sub-frame of pixels is displayed to a remote user at a remote computing system who is participating in the video chat application with the user. The capture device automatically tracks the position of the head, neck and shoulders of the user as the user moves to a next location within the capture area. A next sub-frame of pixels identifying a position of the head, neck and shoulders of the user in the next location is identified and displayed to the remote user at the remote computing device.

Claims

1. A method for automatically tracking movement of a user participating in a video chat application executing in a computing device, the method comprising:

receiving a capture frame comprising one or more depth images of a capture area from a depth camera connected to a computing device, the one or more depth images comprising pixels for one or more objects where each pixel has a depth value representing a distance to the one or more objects in the capture area from the depth camera;

determining if the capture frame includes a user in a first location in the capture area by analyzing edges of the capture frame by comparing the depth values associated with adjacent pixels of the capture frame;

identifying a sub-frame of pixels in the capture frame, the sub-frame of pixels including a position of a head, neck and shoulders of the user in the capture frame;

displaying the sub-frame of pixels including only the position of the head, neck and shoulders of the user to a remote user at a remote computing system;

automatically tracking the position of the head, neck and shoulders of the user to a next location within the capture area;

identifying a next sub-frame of pixels, the next sub-frame of pixels including a position of the head, neck and shoulders of the user in the next location, wherein the next sub-frame of pixels is included in a next capture frame of the capture area;

displaying the next sub-frame of pixels to the remote user in the remote computing system;

detecting and identifying more than one user in the capture area;

identifying individual sub-frames and next sub-frames of pixels for each of the more than one user, each of the individual sub-frames and next sub-frames including only the position of the head, neck and shoulders; and

automatically adjusting the individual sub-frames of pixels or the individual next sub-frame of pixels to include the head, neck and shoulders of each of the users and displaying the individual sub-frame of pixels or the individual next sub-frame of pixels to the remote user, wherein each of the more than one user in the capture area is at a different location within the capture area.

2. The method of claim 1, wherein displaying the sub-frame of pixels and the next sub-frame of pixels to the remote user further comprises:

encoding the sub-frame of pixels and the next sub-frame of pixels into a video communication data stream to the remote user at the remote computing device.

3. The method of claim 1, wherein:

the size of the sub-frame of pixels in the capture frame is different from the size of the next sub-frame of pixels in the next capture frame.

4. The method of claim 1, wherein automatically tracking the position of the head, neck and shoulders of the user to a next location within the capture area further comprises:

automatically tracking lateral movement of the user to the next location in the capture area.

5. The method of claim 1, wherein automatically tracking the position of the head, neck and shoulders of the user to a next location within the capture area further comprises:

automatically tracking movement of the user to different distances from the capture device.

6. The method of claim 1, further comprising:

generating a skeletal model of the user based on the one or more depth images, wherein the head, neck and shoulders of the user is identified using the skeletal model.

7. The method of claim 1, further comprising:

determining if at least one of the sub-frame of pixels or the next sub-frame of pixels can include each of the users.

8. The method of claim 1, further comprising:

detecting and identifying more than one user in the capture area and identifying individual sub-frames of pixels to include each of the users such that each of the sub-frame of pixels or the next sub-frame of pixels are displayed as separately captured images.

9. The method of claim 7, further comprising:

identifying each of the users as a group of users in the sub-frame of pixels or the next sub-frame of pixels when the users are sufficiently close together, for a minimum period of time.

10. The method of claim 9, further comprising:

identifying the users in at least two individual sub-frames of pixels when the users are no longer sufficiently close together, for a minimum period of time.

18. The method of claim 6, wherein a width of a bitmask of the head, neck and shoulders of the two or more users is compared to a value of a predetermined width associated with a head, neck and shoulders of a body model of a human.

19. The method of claim 1, further comprising

automatically centering the position of the head, neck and shoulders of the user in the next sub-frame of pixels;

and

automatically altering the size of the head, neck and shoulders of the user in the next sub-frame of pixels so that the size of the head, neck and shoulders of the user in the sub-frame of pixels and the next sub-frame of pixels is approximately constant.

11. One or more processor readable storage devices having processor readable code embodied on said one or more processor readable storage devices, the processor readable code for programming one or more processors to perform a method comprising:

receiving a capture frame comprising one or more depth images of a capture area from a depth camera connected to a computing device, the one or more depth images comprising pixels for one or more objects where each pixel has a depth value representing a distance to the one or more objects in the capture area from the depth camera, and where each pixel within determined edges is associated with each other to define the one or more objects in the capture area;

determining if the capture frame includes a user in a first location in the capture area by analyzing the edges of the capture frame by comparing the depth values associated with adjacent pixels of the capture frame, and when the depth values are greater than a predetermined edge tolerance, the pixels define the edge;

identifying a sub-frame of pixels in the capture frame, the sub-frame of pixels including a position of a head, neck and shoulders of the user in the capture frame;

displaying the sub-frame of pixels including the position of the head, neck and shoulders of the user to a remote user at a remote computing system;

receiving a next capture frame comprising one or more depth images of the capture area from a depth camera;

automatically tracking movement of one or more users within the capture area;

determining if the next capture frame includes the one or more users in a next location in the capture area based on the tracking;

identifying a next sub-frame of pixels containing the head, neck and shoulders of the one or more users in the next capture frame;

displaying the next sub-frame of pixels to the remote user at the remote computing system; and

detecting and identifying two or more users in the capture area and identifying individual sub-frames of pixels to include each of the users such that each of the sub-frame of pixels or the next sub-frame of pixels are displayed as separately captured images.

12. One or more processor readable storage devices of claim 11, further comprising:

automatically centering the position of the head, neck and shoulders of the one or more users in the next sub-frame of pixels prior to the displaying.

13. One or more processor readable storage devices of claim 11, further comprising:

identifying the one or more users as a group of users in the next sub-frame of pixels when the one or more users are sufficiently close together, for a minimum period of time.

14. One or more processor readable storage devices of claim 13, further comprising:

automatically tracking movement of the group of users to a next location within the capture area by determining the position of the head, neck and shoulders of the group of users in the capture area.

15. One or more processor readable storage devices of claim 11, further comprising:

detecting and identifying two or more users in the capture area, determining if at least one of the sub-frame of pixels or the next sub-frame of pixels can include each of the users, automatically adjusting the sub-frame of pixels or the next sub-frame of pixels to include the head, neck and shoulders of each of the users and displaying the sub-frame of pixels or the next sub-frame of pixels to the remote user, wherein each of the two or more users in the capture area is at a different location within the capture area.

16. The method of claim 11, further comprising:

organizing the depth information including the depth image into layers that are perpendicular to a Z-axis extending from the depth camera along a line of sight to the user.

17. An apparatus for automatically tracking movement of one or more users participating in a video chat application, comprising:

a depth camera; and

a computing device connected to the depth camera to receive a capture frame comprising one or more depth images of a capture area, the one or more depth images comprising pixels for one or more objects where each pixel has a depth value representing a distance to the one or more objects in the capture area from the depth camera, and where each pixel within determined edges is associated with each other to define the one or more objects in the capture area, the computing device storing instructions, that when executed, perform;

determining if the capture frame includes a user in a first location in the capture area by analyzing the edges of the capture frame by comparing the depth values associated with adjacent pixels of the capture frame, and when the depth values are greater than a predetermined edge tolerance, the pixels define the edge;

identifying a sub-frame of pixels in the capture frame, the sub-frame of pixels including a position of a head, neck and shoulders of the user in the capture frame;

displaying the sub-frame of pixels including the position of the head, neck and shoulders of the user to a remote user at a remote computing system;

receiving a next capture frame comprising one or more depth images of the capture area from a depth camera;

automatically tracking movement of two or more users within the capture area;

determining if the next capture frame includes the two or more users in a next location in the capture area based on the tracking;

identifying a next sub-frame of pixels containing the head, neck and shoulders of the two or more users in the next capture frame;

and

displaying the next sub-frame of pixels including the two or more users to the remote user at the remote computing system.

20. The apparatus of claim 17, the computing device storing instructions, that when executed, further perform:

automatically centering the position of the head, neck and shoulders of the two or more users in the next sub-frame of pixels; and

automatically altering the size of the head, neck and shoulders of the two or more users in the next sub-frame of pixels so that the size of the head, neck and shoulders of the two or more users in the sub-frame of pixels and the next sub-frame of pixels is approximately constant.