Methods and systems for classifying pixels as foreground using both short-range depth data and long-range depth data

• US 10,244,224 B2
• Filed: 05/26/2015
• Issued: 03/26/2019
• Est. Priority Date: 05/26/2015
• Status: Active Grant

• Current Assignee: Personify Incorporated (Wilson Human Capital Group, Inc.)
• Original Assignee: Personify Incorporated (Wilson Human Capital Group, Inc.)
• Inventors: Nguyen, Quang, Dang, Long, Nguyen, Cong, Lin, Dennis, Venshtain, Simion, Do, Minh
• Primary Examiner(s): Lefkowitz, Sumati
• Assistant Examiner(s): Perlman, David
• Application Number:
  US14/721,428
• Publication Number:
  US 20160353080A1
• Time in Patent Office: 1,400 Days
• Field of Search: None
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/20076 Probabilistic image processing
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/174 involving the use of two or more images
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G06V 20/64 Three-dimensional objects
• CPC Class Codes: H04N 13/15 for colour aspects of image signals
• CPC Class Codes: H04N 13/158 Switching image signals
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 2013/0085 Motion estimation from stereoscopic image signals
• CPC Class Codes: H04N 2013/0092 Image segmentation from stereoscopic image signals

Abstract

Disclosed herein are methods and systems for classifying pixels as foreground using both short-range depth data and long-range depth data. One embodiment takes the form of a process that includes obtaining video data depicting at least a portion of a user. The process also includes obtaining short-range depth data associated with the video data. The process also includes obtaining long-range depth data associated with the video data. The video data, short-range depth data, and long-range depth data may be obtained via a single 3-D video camera. The process also includes classifying pixels of the video data as foreground based at least in part on both the short-range depth data and the long-range depth data. In some embodiments, classifying pixels of the video data as foreground comprises employing an alpha mask. The alpha mask may comprise binary foreground (hard) indicators. The alpha mask may comprise foreground-likelihood (soft) indicators.

Claims

1. A method comprising:

obtaining, from a three-dimensional (3-D) video camera, video data depicting at least a portion of a user;

obtaining, from the 3-D video camera operating in a short-range mode, short-range depth data associated with the video data;

obtaining, from the 3-D video camera operating in a long-range mode, long-range depth data associated with the video data;

detecting less than a threshold amount of motion in at least one of the obtained short-range depth data and the obtained video data during a motion-detection period;

classifying pixels of the video data as foreground based at least in part on a comparison between the short-range depth data and the long-range depth data; and

extracting a user-persona from the video data based at least in part on the pixels of the video data classified as foreground.

2. The method of claim 1, wherein detecting less than a threshold amount of motion in at least one of the obtained short-range depth data and the obtained video data during a motion-detection period comprises detecting a mode-switching trigger, and responsively switching from obtaining the short-range depth data from the 3-D video camera operating in the short-range mode to obtaining the long-range depth data from the 3-D video camera operating in the long-range mode.

3. The method of claim 2, wherein the mode-switching trigger is at least one of a periodic mode-switching trigger and an on-demand mode-switching trigger.

4. The method of claim 1, wherein the motion-detection period is a periodic motion-detection period.

5. The method of claim 1, further comprising:

identifying a short-range foreground region at least in part by using the short-range depth data; and

identifying a long-range foreground region at least in part by using the long-range depth data.

6. The method of claim 5, wherein classifying pixels of the video data as foreground based at least in part on a comparison between the short-range depth data and the long-range depth data comprises classifying pixels of the video data as foreground based at least in part on a comparison between the identified short-range foreground region and the identified long-range foreground region.

7. The method of claim 5, wherein:

identifying the short-range foreground region at least in part by using the short-range depth data comprises employing a threshold depth value; and

identifying the long-range foreground region at least in part by using the long-range depth data comprises employing the threshold depth value.

8. The method of claim 5, further comprising determining a user-hair region of the video data at least in part by using both the short-range foreground region and the long-range foreground region.

9. The method of claim 8, further comprising identifying a foreground-region delta at least in part by subtracting the short-range foreground region from the long-range foreground region, wherein determining the user-hair region of the video data comprises including the identified foreground-region delta in the user-hair region.

10. The method of claim 9, wherein classifying pixels of the video data as foreground comprises classifying pixels in the identified foreground-region delta as foreground.

11. The method of claim 9, further comprising updating a user-hair-color model using respective colors of pixels in the identified foreground-region delta, wherein classifying pixels of the video data as foreground comprises classifying pixels of the video data as foreground at least in part by using the updated user-hair-color model.

12. The method of claim 11, wherein classifying pixels of the video data as foreground at least in part by using the updated user-hair-color model comprises performing a flood fill using the updated user-hair-color model.

13. The method of claim 1, wherein classifying pixels of the video data as foreground comprises employing an alpha mask.

14. A system comprising:

a communication interface;

a processor; and

data storage containing instructions executable by the processor for causing the system to carry out a set of functions, the set of functions including;

obtaining, from a three-dimensional (3-D) video camera, video data depicting at least a portion of a user;

obtaining, from the 3-D video camera operating in a short-range mode, short-range depth data associated with the video data;

obtaining, from the 3-D video camera operating in a long-range mode, long-range depth data associated with the video data;

detecting less than a threshold amount of motion in at least one of the obtained short-range depth data and the obtained video data during a motion-detection period;

classifying pixels of the video data as foreground based at least in part on a comparison between the short-range depth data and the long-range depth data; and

extracting a user-persona from the video data based at least in part on the pixels of the video data classified as foreground.

15. A method comprising:

obtaining, from a three-dimensional (3-D) video camera, video data depicting at least a portion of a user;

obtaining, from the 3-D video camera operating in a short-range mode, short-range depth data associated with the video data;

obtaining, from the 3-D video camera operating in a long-range mode, long-range depth data associated with the video data;

classifying pixels of the video data as foreground based at least in part on a comparison between the short-range depth data and the long-range depth data;

extracting a user-persona from the video data based at least in part on the pixels of the video data classified as foreground;

identifying a short-range foreground region at least in part by using the short-range depth data;

identifying a long-range foreground region at least in part by using the long-range depth data; and

determining a user-hair region of the video data at least in part by using both the short-range foreground region and the long-range foreground region.

16. The method of claim 15, wherein classifying pixels of the video data as foreground based at least in part on a comparison between the short-range depth data and the long-range depth data comprises classifying pixels of the video data as foreground based at least in part on a comparison between the identified short-range foreground region and the identified long-range foreground region.

17. The method of claim 15, wherein:

identifying the short-range foreground region at least in part by using the short-range depth data comprises employing a threshold depth value; and

identifying the long-range foreground region at least in part by using the long-range depth data comprises employing the threshold depth value.

18. The method of claim 15, further comprising identifying a foreground-region delta at least in part by subtracting the short-range foreground region from the long-range foreground region, wherein determining the user-hair region of the video data comprises including the identified foreground-region delta in the user-hair region.

19. The method of claim 18, wherein classifying pixels of the video data as foreground comprises classifying pixels in the identified foreground-region delta as foreground.

20. The method of claim 18, further comprising updating a user-hair-color model using respective colors of pixels in the identified foreground-region delta, wherein classifying pixels of the video data as foreground comprises classifying pixels of the video data as foreground at least in part by using the updated user-hair-color model.