Method for detecting talking heads in a compressed video

• US 6,847,680 B2
• Filed: 12/17/2001
• Issued: 01/25/2005
• Est. Priority Date: 12/17/2001
• Status: Expired due to Fees

• Current Assignee: Mitsubishi Electric Research Laboratories, Inc. (Mitsubishi Electric Corporation)
• Original Assignee: Mitsubishi Electric Research Laboratories, Inc. (Mitsubishi Electric Corporation)
• Inventors: Radhakrishnan, Regunathan, Divakaran, Ajay
• Primary Examiner(s): Kelley, Chris
• Assistant Examiner(s): REKSTAD, ERICK J
• Application Number:
  US10/022,789
• Publication Number:
  US 20030112865A1
• Time in Patent Office: 1,135 Days
• Field of Search: 375/240.01, 375/240.08, 382/159, 382/224, 382/118, 348/169, 725/38
• US Class Current: 375/240.01
• CPC Class Codes: G06V 20/46 Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
• CPC Class Codes: G06V 40/167 using comparisons between temporally consecutive images
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: H04N 19/142 Detection of scene cut or scene change
• CPC Class Codes: H04N 19/179 the unit being a scene or a shot
• CPC Class Codes: H04N 19/46 Embedding additional information in the video signal during the compression process H04N19/517, H04N19/68, H04N19/70 take precedence
• CPC Class Codes: H04N 19/503 involving temporal prediction adaptive coding with adaptive selection between spatial and temporal predictive coding H04N19/107; adaptive coding with adaptive selection among a plurality of temporal predictive coding modes H04N19/109
• CPC Class Codes: H04N 19/517 by encoding
• CPC Class Codes: H04N 19/87 involving scene cut or scene change detection in combination with video compression

Abstract

A method identifies a talking head or principal cast in a compressed video by first segmenting the video into shots. Motion activity descriptors are extracted from each of the shots, and combined into a shot motion activity descriptor. A distance between the shot motion activity descriptor and a template motion activity descriptor is measured. The template motion activity descriptor is obtained from a training video. If the measured distance is less than a predetermined threshold, then the shot is identified as including a talking head.

Claims

1. A method for identifying talking heads in a compressed video, comprising:

extracting motion activity descriptors from each of a plurality of shots;

combining the plurality of motion activity descriptors of each shot, into a shot motion activity descriptor;

measuring a distance between the shot motion activity descriptor and a template motion activity descriptor; and

identifying a particular shot as a talking head if the measured distance is less than a predetermined threshold.

2. The method of claim 1 further comprising:

extracting a plurality of training motion activity descriptors from a training video including a plurality of training shots, each training shot including a training talking head; and

combining the plurality of training motion activity descriptors into the template motion activity descriptor.

3. The method of claim 2 wherein the combining is a median of the plurality of training motion activity descriptors.

4. The method of claim 2 wherein the combining is a mean of the plurality of training motion activity descriptors.

5. The method of claim 1 further comprising:

normalizing the measured distance.

6. The method of claim 1 wherein the threshold is a standard deviation σ

of the temple motion activity descriptor.

7. The method of claim 1 wherein each motion activity descriptor is of the form C_mv^avg,N_sr,N_mr,N_lr,σ

_fr, where C_mv^avg is an average motion vector, and N_sr,N_mr,N_lr are short, medium and long run zero-length motion vectors, respectively.

8. The method of claim 7 wherein the distance is measured according to:

$\begin{array}{c}D<br><br>\left(S,T\right)=<br><br>\frac{W_{\mathrm{tot}}}{C_{\mathrm{avg}}<br><br>\left(T\right)}<br><br><br><br>C_{\mathrm{avg}}<br><br>\left(T\right)-C_{\mathrm{avg}}<br><br>\left(S\right)<br><br>+\frac{W_{\mathrm{tot}}}{N_{\mathrm{sr}}<br><br>\left(T\right)}<br><br><br><br>N_{\mathrm{sr}}<br><br>\left(T\right)-N_{\mathrm{sr}}<br><br>\left(S\right)<br><br>+\\ <br><br>\frac{W_{\mathrm{tot}}}{N_{\mathrm{mr}}<br><br>\left(T\right)}<br><br><br><br>N_{\mathrm{mr}}<br><br>\left(T\right)-N_{\mathrm{mr}}<br><br>\left(S\right)<br><br>+\frac{W_{\mathrm{tot}}}{N_{\mathrm{lr}}<br><br>\left(T\right)}<br><br><br><br>N_{\mathrm{lr}}<br><br>\left(T\right)-N_{\mathrm{lr}}<br><br>\left(S\right)<br><br>\end{array}$

where W_tot is a normalizing weight, T is the template motion activity descriptor, and S is the shot motion activity descriptor.

9. The method of claim 1 further comprising:

measuring a distance between the shot motion activity descriptor and a set of template motion activity descriptors.

10. The method of claim 1 wherein the distance is a semi-Hausdorff distance.

11. The method of claim 1 wherein the template motion activity is modeled by a discrete function.

12. The method of claim 1 wherein the template motion activity is modeled by a continuous function.

13. The method of claim 12 wherein the continuous function is a mixture of Gaussian distributions.

14. The method of claim 1 further comprising:

extracting a plurality of training motion activity descriptors from sampled frames of a training video including a plurality of training shots, each training shot including a training talking head; and

combining the plurality of training motion activity descriptors into the template motion activity descriptor.

15. The method of claim 1 further comprising:

segmenting the video into the plurality of shots using the motion activity descriptors.

16. The method of claim 1 further comprising:

retaining only talking head shots.