Automatic detection and tracking of multiple individuals using multiple cues

• US 20050147278A1
• Filed: 01/25/2005
• Published: 07/07/2005
• Est. Priority Date: 12/03/2001
• Status: Active Grant

• Current Assignee: Zhigu Holdings Limited
• Original Assignee: Microsoft Corporation
• Inventors: Chen, Yungqiang, Rui, Yong
• Granted Patent: US 7,151,843 B2
• Time in Patent Office: Days
• US Class Current: 382/103
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 7/251 involving models
• CPC Class Codes: G06V 40/162 using pixel segmentation or colour matching

Abstract

Automatic detection and tracking of multiple individuals includes receiving a frame of video and/or audio content and identifying a candidate area for a new face region in the frame. One or more hierarchical verification levels are used to verify whether a human face is in the candidate area, and an indication made that the candidate area includes a face if the one or more hierarchical verification levels verify that a human face is in the candidate area. A plurality of audio and/or video cues are used to track each verified face in the video content from frame to frame.

Claims

1. A system to track multiple individuals in video content, the system comprising:

an auto-initialization module to detect a candidate region for a new face in a frame of the video content;

a hierarchical verification module to generate a confidence level for the candidate region; and

a multi-cue tracking module to use a plurality of visual cues to track previous candidate regions with confidence levels, generated by the hierarchical verification module, that exceeded a threshold value.

2. A system as recited in claim 1, wherein the hierarchical verification module is further configured to:

check whether the confidence level exceeds the threshold value;

if the confidence level does exceed the threshold value then to pass the candidate region to the multi-cue tracking module; and

if the confidence level does not exceed the threshold value then to discard the candidate region and not pass the candidate region to the multi-cue tracking module.

3. A system as recited in claim 1, wherein the hierarchical verification module is further configured to:

receive, from the multi-cue tracking module, an indication of a region;

verify whether the region is a face; and

return the region to the multi-cue tracking module for continued tracking only if the region is verified as a face.

4. A system as recited in claim 1, wherein the system comprises a video conferencing system.

5. A system as recited in claim 1, wherein the auto-initialization module is further to:

detect whether there is motion in the frame;

if there is motion in the frame, then perform motion-based initialization to identify the candidate region;

detect whether there is audio in the frame;

if there is audio in the frame, then perform audio-based initialization to identify the candidate region; and

if there is neither motion in the frame nor audio in the frame, then use a fast face detector to identify the candidate region.

6. A system as recited in claim 1, wherein the hierarchical verification module is to use one or more hierarchical verification levels that include a coarse level and a fine level, wherein the coarse level can verify whether the new face is in the candidate area faster but with less accuracy than the fine level.