Automatic detection and tracking of multiple individuals using multiple cues

• US 7,151,843 B2
• Filed: 01/25/2005
• Issued: 12/19/2006
• Est. Priority Date: 12/03/2001
• Status: Expired due to Term

• Current Assignee: Zhigu Holdings Limited
• Original Assignee: Microsoft Corporation
• Inventors: Rui, Yong, Chen, Yunqiang
• Primary Examiner(s): Wu, Jingge
• Assistant Examiner(s): GORADIA, SHEFALI DINESH
• Application Number:
  US11/042,015
• Publication Number:
  US 20050147278A1
• Time in Patent Office: 693 Days
• Field of Search: 382/240, 382/226, 382/118, 382190-208, 382/103, 382/107, 382/228, 704/235, 704/275, 704/270.1, 715/863, 348/169, 348/14.1
• US Class Current: 382/103
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 7/251 involving models
• CPC Class Codes: G06V 40/162 using pixel segmentation or colour matching

Abstract

Automatic detection and tracking of multiple individuals includes receiving a frame of video and/or audio content and identifying a candidate area for a new face region in the frame. One or more hierarchical verification levels are used to verify whether a human face is in the candidate area, and an indication made that the candidate area includes a face if the one or more hierarchical verification levels verify that a human face is in the candidate area. A plurality of audio and/or video cues are used to track each verified face in the video content from frame to frame.

Claims

1. A system to track multiple individuals in video content, the system comprising:

an auto-initialization module to detect a candidate region for a new face in a frame of the video content;

a hierarchical verification module to generate a confidence level for the candidate region;

a multi-cue tracking module to use a plurality of visual cues to track previous candidate regions with confidence levels, generated by the hierarchical verification module, that exceeded a threshold value, and

wherein the hierarchical verification module is further configured to;

check whether the confidence level exceeds the threshold value;

if the confidence level does exceed the threshold value then to pass the candidate region to the multi-cue tracking module; and

if the confidence level does not exceed the threshold value then to discard the candidate region and not pass the candidate region to the multi-cue tracking module.

2. A system to track multiple individuals in video content, the system comprising:

an auto-initialization module to detect a candidate region for a new face in a frame of the video content;

a hierarchical verification module to generate a confidence level for the candidate region;

a multi-cue tracking module to use a plurality of visual cues to track previous candidate regions with confidence levels, generated by the hierarchical verification module, that exceeded a threshold value, and

wherein the auto-initialization module is further configured to;

detect whether there is motion in the frame

if there is motion in the frame, then perform motion-based initialization to identify the candidate region;

detect whether there is audio in the frame;

if there is audio in the frame, then perform audio-based initialization to identify the candidate region; and

if there is neither motion in the frame nor audio in the frame, then use a fast face detector to identify the candidate region.