Automatic detection and tracking of multiple individuals using multiple cues

• US 20050129278A1
• Filed: 01/25/2005
• Published: 06/16/2005
• Est. Priority Date: 12/03/2001
• Status: Active Grant

• Current Assignee: Zhigu Holdings Limited
• Original Assignee: Microsoft Corporation
• Inventors: Rui, Yong, Chen, Yunqiang
• Granted Patent: US 7,171,025 B2
• Time in Patent Office: Days
• US Class Current: 382/103
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 7/251 involving models
• CPC Class Codes: G06V 40/162 using pixel segmentation or colour matching

Abstract

Automatic detection and tracking of multiple individuals includes receiving a frame of video and/or audio content and identifying a candidate area for a new face region in the frame. One or more hierarchical verification levels are used to verify whether a human face is in the candidate area, and an indication made that the candidate area includes a face if the one or more hierarchical verification levels verify that a human face is in the candidate area. A plurality of audio and/or video cues are used to track each verified face in the video content from frame to frame.

Claims

1. One or more computer readable media having stored thereon a plurality of instructions to track faces from frame to frame of content, wherein the plurality of instructions, when executed by one or more processors, causes the one or more processors to:

predict, using a plurality of cues, where a contour of a face will be in a frame;

encode a smoothness constraint that penalizes roughness;

apply the smoothness constraint to a plurality of possible contour locations; and

select the contour location having the smoothest contour as the location of the face in the frame.

2. One or more computer readable media as recited in claim 1, wherein the plurality of cues include foreground color, background color, edge intensity, and motion.

3. One or more computer readable media as recited in claim 1, wherein the plurality of cues include audio.

4. One or more computer readable media as recited in claim 1, wherein the smoothness constraint includes contour smoothness.

5. One or more computer readable media as recited in claim 1, wherein the smoothness constraint includes both contour smoothness and region smoothness.

6. One or more computer readable media as recited in claim 1, wherein the plurality of instructions to encode the smoothness constraint comprises instructions that cause the one or more processors to generate Hidden Markov Model (HMM) state transition probabilities.

7. One or more computer readable media as recited in claim 1, wherein the plurality of instructions to encode the smoothness constraint comprises instructions that cause the one or more processors to generate Joint Probability Data Association Filter (JPDAF) state transition probabilities.

8. One or more computer readable media as recited in claim 1, wherein the plurality of instructions further comprise instructions that cause the one or more processors to:

adapt the predicting for the face in subsequent frames to account for changing color distributions.

9. One or more computer readable media as recited in claim 1, wherein the plurality of instructions further comprise instructions that cause the one or more processors to:

adapt the predicting for the face in subsequent frames based on one or more cues observed in the frame.

10. One or more computer readable media as recited in claim 1, the plurality of instructions further comprise instructions that cause the one or more processors to concurrently track multiple possible locations for the face from frame to frame.

11. A method for tracking an object along frames of content, the method comprising:

predicting where the object will be in a frame;

encoding a smoothness constraint that penalizes roughness;

applying the smoothness constraint to a plurality of possible object locations; and

selecting the object location having the smoothest contour as the location of the object in the frame.

12. A method as recited in claim 11, wherein the predicting uses a plurality of cues that include foreground color, background color, edge intensity, motion, and audio.

13. A method as recited in claim 11, wherein the smoothness constraint includes both contour smoothness and region smoothness.

14. A method as recited in claim 11, wherein encoding the smoothness constraint comprises generating Hidden Markov Model (HMM) state transition probabilities.

15. A method as recited in claim 11, wherein using the plurality of cues to track each verified face further comprises, for each face:

adapting the predicting for the face in subsequent frames based on one or more cues observed in the frame.

16. A method as recited in claim 11, wherein predicting where the object will be comprises:

accessing a set of one or more feature points of the face; and

analyzing the frame to identify an area that includes the set of one or more feature points.

17. A method as recited in claim 11, wherein using the plurality of cues to track each verified face comprises concurrently tracking multiple possible locations for the face from frame to frame.

18. A method as recited in claim 17, further comprising using a multiple-hypothesis tracking technique to concurrently track the multiple possible locations.

19. A method as recited in claim 11, wherein the object comprises a face in video content.

20. A method as recited in claim 11, wherein the object comprises a sound source location in audio content.