Methods and systems for assigning pixels distance-cost values using a flood fill technique

• US 9,953,223 B2
• Filed: 02/06/2017
• Issued: 04/24/2018
• Est. Priority Date: 05/19/2015
• Status: Active Grant

• Current Assignee: Personify Incorporated (Wilson Human Capital Group, Inc.)
• Original Assignee: Personify Incorporated (Wilson Human Capital Group, Inc.)
• Inventors: Lin, Dennis, Nguyen, Quang, Dang, Gia, Zhang, Yi, Venshtain, Simion, Nguyen, Cong
• Primary Examiner(s): Allison, Andrae S
• Application Number:
  US15/425,331
• Publication Number:
  US 20170147882A1
• Time in Patent Office: 442 Days
• Field of Search: None
• CPC Class Codes: G06F 18/2413 based on distances to training or reference patterns
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/20156 Automatic seed setting
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 7/0002 Inspection of images, e.g. flaw detection
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/12 Edge-based segmentation
• CPC Class Codes: G06T 7/181 involving edge growing; involving edge linking
• CPC Class Codes: G06T 7/187 involving region growing; involving region merging; involving connected component labelling
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G06T 7/50 Depth or shape recovery
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G06V 20/48 Matching video sequences
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships

Abstract

Disclosed herein are methods and systems for assigning pixels distance-cost values using a flood fill technique. One embodiment takes the form of a process that includes obtaining video data depicting a head of a user, obtaining depth data associated with the video data, and selecting seed pixels for a flood fill at least in part by using the depth information. The process also includes performing the flood fill from the selected seed pixels. The flood fill assigns respective distance-cost values to pixels of the video data based on position-space cost values and color-space cost values. In some embodiments, the process also includes classifying pixels of the video data as foreground based at least in part on the assigned distance-cost values. In some other embodiments, the process also includes assigning pixels of the video data foreground-likelihood values based at least in part on the assigned distance-cost values.

Claims

1. A method comprising:

obtaining video data depicting a head of a user;

obtaining depth data associated with the video data;

selecting seed pixels for a hair-identification flood fill for identifying pixels depicting hair of the head of the user, the seed pixels selected at least in part by using the obtained depth data;

performing the hair-identification flood fill from the selected seed pixels, the hair-identification flood fill assigning respective distance-cost values to pixels of the video data based on respective position-space-cost values and respective color-space-cost values, wherein performing the hair-identification flood fill comprises;

identifying a plurality of neighbor pixels of a current pixel;

determining respective step-cost values from the current pixel to each pixel in the plurality of neighbor pixels; and

assigning each pixel in the plurality of neighbor pixels a respective distance-cost value based on a distance-cost value of the current pixel and the respective step-cost values; and

identifying, using a foreground identification process, a persona of the user from the video data, wherein the respective distance-cost values assigned to pixels by the hair-identification flood fill is one of a plurality of inputs of the foreground identification process.

2. The method of claim 1, wherein selecting seed pixels for the hair-identification flood fill further comprises selecting seed pixels for the hair-identification flood fill at least in part by using the video data.

3. The method of claim 1, further comprising:

obtaining a head contour that estimates an outline of the depicted head of the user, the head contour being based at least in part on the depth data associated with the video data,

wherein the selected seed pixels are on an upper contour, wherein the upper contour is an upper portion of the head contour.

4. The method of claim 3, wherein the selected seed pixels are equally distributed along the upper contour.

5. The method of claim 3, wherein the selected seed pixels are of colors that are found in a user-hair-color model.

6. The method of claim 1, wherein selecting seed pixels for the hair-identification flood fill comprises:

identifying pixels having noisy depth values over a series of frames; and

selecting the identified noisy depth-pixels as seed pixels for the hair-identification flood fill.

7. The method of claim 6, wherein the selected seed pixels are located within an extended head box.

8. The method of claim 6, wherein the selected seed pixels have intermittent depth values that are within a threshold tolerance of a depth value corresponding to the head of the user.

9. The method of claim 6, wherein the selected seed pixels are of colors that are found in a user-hair-color model.

10. The method of claim 1, wherein a first set of the selected seed pixels are on an upper contour and a second set of the selected seed pixels have noisy depth values over a series of frames, wherein the upper contour is an upper portion of a head contour that estimates an outline of the depicted head of the user, the method further comprising:

initializing the distance-cost values of the seed pixels in the first set to be zero; and

initializing the distance-cost values of the seed pixels in the second set to be non-zero.

11. The method of claim 1, wherein performing the hair-identification flood fill comprises:

determining a minimum distance-cost value from at least one of the selected seed pixels to a current pixel; and

assigning the current pixel a distance-cost value that is the determined minimum distance-cost value.

12. The method of claim 11, wherein determining a minimum distance-cost value comprises:

comparing a current distance-cost value corresponding with a current flood-fill path to a prior distance-cost value corresponding with a prior flood-fill path.

13. The method of claim 12, wherein the current flood-fill path and the prior flood-fill path originate from a common seed pixel.

14. The method of claim 12, wherein the current flood-fill path and the prior flood-fill path originate from different seed pixels.

15. The method of claim 1, wherein performing the hair-identification flood fill comprises performing the hair-identification flood fill along a plurality of flood-fill paths, the method further comprising:

terminating the hair-identification flood fill along a current flood-fill path in response to at least one termination criteria, the termination criteria comprising;

a current pixel not being a user-hair color according to a user-hair-color model;

the current pixel being a background color according to a background-color model;

a distance-cost value to the current pixel being greater than a distance-cost threshold; and

a step-cost value to the current pixel being greater than a step-cost threshold.

16. A system comprising:

a communication interface;

a processor; and

data storage containing instructions executable by the processor for causing the system to carry out a set of functions, the set of functions including;

obtaining video data depicting a head of a user;

obtaining depth data associated with the video data;

selecting seed pixels for a hair-identification flood fill for identifying pixels depicting hair of the head of the user, the seed pixels selected at least in part by using the depth data;

performing the hair-identification flood fill from the selected seed pixels, the hair-identification flood fill assigning respective distance-cost values to pixels of the video data based on respective position-space-cost values and respective color-space-cost values, wherein performing the hair-identification flood fill comprises;

identifying a plurality of neighbor pixels of a current pixel;

determining respective step-cost values from the current pixel to each pixel in the plurality of neighbor pixels; and

assigning each pixel in the plurality of neighbor pixels a respective distance-cost value based on a distance-cost value of the current pixel and the respective step-cost values; and

identifying, using a foreground identification process, a persona of the user from the video data, wherein the respective distance-cost values assigned to pixels by the hair-identification flood fill is one of a plurality of inputs of the foreground identification process.