Methods and systems for real-time user extraction using deep learning networks

• US 9,881,207 B1
• Filed: 10/25/2016
• Issued: 01/30/2018
• Est. Priority Date: 10/25/2016
• Status: Active Grant

• Current Assignee: Personify Incorporated (Wilson Human Capital Group, Inc.)
• Original Assignee: Personify Incorporated (Wilson Human Capital Group, Inc.)
• Inventors: Nguyen, Quang, Nguyen, Cong, Dang, Long, Dang, Gia, Venshtain, Simion
• Primary Examiner(s): Allison, Andrae S
• Application Number:
  US15/333,623
• Time in Patent Office: 462 Days
• Field of Search: None
• CPC Class Codes: G06F 18/213 Feature extraction, e.g. by transforming the feature space; Summarisation; Mappings, e.g. subspace methods
• CPC Class Codes: G06N 3/044 Recurrent networks, e.g. Hopfield networks
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition

Abstract

Methods and systems for real-time user extraction using deep learning networks. In one embodiment, user extraction comprises obtaining a given frame of color pixel data, checking whether a reset flag is cleared or set, and generating a trimap for the given frame. If the reset flag is cleared, generating the trimap comprises: obtaining a user-extraction contour based on a preceding frame; and generating the trimap based on the obtained user-extraction contour. If the reset flag is set, generating the trimap comprises: detecting at least one persona feature in the given frame; generating an alpha mask by aligning an intermediate contour with the detected persona feature(s), wherein the intermediate contour is based on a color-based flood-fill operation performed on a previous frame which was segmented by a machine-learning-segmentation process; and generating the trimap based on the generated alpha mask. The generated trimap is output for extracting a user persona.

Claims

1. A method comprising:

obtaining a first frame of color pixel data;

checking whether a reset flag is cleared or set at a first time;

generating a trimap for the first frame, wherein;

if the reset flag is cleared at the first time, then generating the trimap for the first frame comprises;

obtaining a user-extraction contour that is based on an immediately preceding frame; and

generating the trimap for the first frame based on the obtained user-extraction contour;

if the reset flag is set at the first time, then generating the trimap for the first frame comprises;

detecting at least one persona feature in the first frame;

generating an alpha mask at least in part by aligning an intermediate persona contour with the detected at least one persona feature, wherein the intermediate persona contour is based on a result of a color-based flood-fill operation having been performed on a previous frame of color pixel data that had been segmented by a machine-learning-segmentation (MLS) process; and

generating the trimap for the first frame based on the generated alpha mask; and

outputting the generated trimap for use in extracting a user persona from the first frame.

2. The method of claim 1, wherein if the reset flag is cleared at the first time, then generating the trimap for the first frame further comprises performing an erosion operation inward from the obtained user-extraction contour and a dilation operation outward from the obtained user-extraction contour.

3. The method of claim 1, wherein if the reset flag is set at the first time, generating the trimap for the first frame further comprises performing an erosion operation inward from the generated alpha mask and a dilation operation outward from the generated alpha mask.

4. The method of claim 1, wherein if the reset flag is set at the first time, aligning the intermediate persona contour with the detected at least one persona feature comprises at least one of scaling and translating the intermediate persona contour to fit the detected at least one persona feature.

5. The method of claim 1, further comprising:

evaluating at least one reset condition;

if at least a predefined subset of the at least one reset condition is true, responsively setting the reset flag for a following frame; and

if each of the at least one reset condition is true, responsively communicating an MLS request to the MLS process.

6. The method of claim 1, wherein the at least one reset condition includes:

(i) an elapsed time from a previous machine-learning segmentation request being greater than a time threshold;

(ii) a difference between a post-core segmentation user-extraction contour for the first frame and the intermediate persona contour exceeding a difference threshold;

(iii) a level of motion in the first frame exceeding a motion threshold; and

(iv) at least one user being detected in the first frame.

7. The method of claim 6, wherein the predefined subset includes reset conditions (i) and (ii).

8. The method of claim 6, wherein the predefined subset consists of reset conditions (i) and (ii).

9. The method of claim 6, wherein the MLS request specifies the first frame as an input to the MLS process.

10. The method of claim 1, further comprising:

obtaining a second frame of color pixel data;

checking whether the reset flag is cleared or set at a second time;

generating a second-frame trimap for the second frame, wherein;

if the reset flag is cleared at the second time, then generating the second-frame trimap comprises;

obtaining a post-core segmentation user-extraction contour that is based on the first frame; and

generating the second-frame trimap based on the obtained post-core segmentation user-extraction contour;

if the reset flag is set at the second time, then generating the second-frame trimap comprises;

detecting at least one second-frame persona feature in the second frame;

generating a second-frame alpha mask at least in part by aligning the intermediate persona contour with the detected at least one second-frame persona feature; and

generating the second-frame trimap based on the generated second-frame alpha mask; and

outputting the generated second-frame trimap for use in extracting a user persona from the second frame.

11. The method of claim 10 wherein aligning the intermediate persona contour with the detected at least one second-frame persona feature comprises at least one of scaling and translating the intermediate persona contour to fit the detected at least one second-frame persona feature.

12. A method comprising:

receiving a first segmented frame of color pixel data, the first segmented frame comprising an initial-segmentation persona contour that was identified by a machine-learning-segmentation (MLS) process;

defining an eroded MLS persona contour and a dilated MLS persona contour, both based on the initial-segmentation persona contour;

defining an intermediate persona contour at least in part by performing a bidirectional color-based flood-fill operation outward from the eroded MLS persona contour and inward from the dilated MLS persona contour;

obtaining a second frame of color pixel data from an input source;

detecting at least one second-frame persona feature in the second frame;

generating a second-frame alpha mask at least in part by aligning the intermediate persona contour with the detected at least one second-frame persona feature;

generating a second-frame trimap based on the generated second-frame alpha mask; and

outputting the generated second-frame trimap for use in extracting a user persona from the second frame.

13. The method of claim 12, wherein the eroded MLS persona contour is defined at least in part by performing an erosion operation inward from the initial-segmentation persona contour, and the dilated MLS persona contour is defined at least in part by performing a dilation operation outward from the initial-segmentation persona contour.

14. The method of claim 12, wherein generating the second-frame trimap comprises eroding and dilating the alpha mask to a lesser extent than is used when defining the eroded MLS persona contour and the dilated MLS persona contour for the first segmented frame.

15. The method of claim 12, wherein aligning the intermediate persona contour with the detected at least one second-frame persona feature comprises at least one of scaling and translating the intermediate persona contour to fit the detected at least one second-frame persona feature.

16. The method of claim 12, further comprising filtering the first segmented frame to exclude pixels whose confidence level, as defined by a confidence map generated by the MLS process, falls below a confidence threshold.

17. The method of claim 12, further comprising:

evaluating at least one reset condition;

if at least a predefined subset of the at least one reset condition is true, responsively setting a reset flag for a following frame; and

if each of the at least one reset condition is true, responsively communicating an MLS request to the MLS process.

18. The method of claim 17, wherein the at least one reset condition includes:

(i) an elapsed time from a previous MLS request being greater than a time threshold;

(ii) a difference between a post-core segmentation user-extraction contour for the second frame and the intermediate persona contour exceeding a difference threshold;

(iii) a level of motion in the second frame exceeding a motion threshold; and

(iv) at least one user being detected in the second frame.

19. The method of claim 18, wherein the predefined subset includes reset conditions (i) and (ii).

20. The method of claim 18, wherein the predefined subset consists of reset conditions (i) and (ii).

21. The method of claim 12, further comprising:

obtaining a third frame of color pixel data;

checking whether a reset flag is cleared or set at a third time;

generating a third-frame trimap, wherein;

if the reset flag is cleared at the third time, then generating the third-frame trimap comprises;

obtaining a post-core segmentation user-extraction contour that is based on the second frame; and

generating the third-frame trimap based on the obtained post-core segmentation user-extraction contour;

if the reset flag is set at the third time, then generating the third-frame trimap comprises;

detecting at least one third-frame persona feature in the third frame;

generating a third-frame alpha mask at least in part by aligning the intermediate persona contour with the detected at least one third-frame persona feature; and

generating the third-frame trimap based on the generated third-frame alpha mask; and

outputting the generated third-frame trimap for use in extracting a user persona from the third frame.

22. An apparatus comprising:

a communication interface;

a processor; and

non-transitory computer readable data storage containing instructions executable by the processor for causing the system to carry out a set of functions, the set of functions comprising;

obtaining a first frame of color pixel data;

checking whether a reset flag is cleared or set at a first time;

generating a trimap for the first frame, wherein;

if the reset flag is cleared at the first time, then generating the trimap for the first frame comprises;

obtaining a user-extraction contour that is based on an immediately preceding frame; and

generating the trimap for the first frame based on the obtained user-extraction contour;

if the reset flag is set at the first time, then generating the trimap for the first frame comprises;

detecting at least one persona feature in the first frame;

generating an alpha mask at least in part by aligning an intermediate persona contour with the detected at least one persona feature, wherein the intermediate persona contour is based on a result of a color-based flood-fill operation having been performed on a previous frame of color pixel data that had been segmented by a machine-learning-segmentation (MLS) process; and

generating the trimap for the first frame based on the generated alpha mask; and

outputting the generated trimap for use in extracting a user persona from the first frame.