SYSTEM AND METHOD FOR JOINT SPEAKER AND SCENE RECOGNITION IN A VIDEO/AUDIO PROCESSING ENVIRONMENT

• US 20130300939A1
• Filed: 05/11/2012
• Published: 11/14/2013
• Est. Priority Date: 05/11/2012
• Status: Abandoned Application

• Current Assignee: Cisco Technology, Inc. (Cisco Systems, Inc.)
• Original Assignee: Cisco Technology, Inc. (Cisco Systems, Inc.)
• Inventors: Chou, Jim Chen, Kajarekar, Sachin, Sankar, Ananth, Catchpole, Jason J.
• Application Number:
  US13/469,886
• Publication Number:
  US 20130300939A1
• Time in Patent Office: Days
• US Class Current: 348/700
• CPC Class Codes: G06F 18/295 Markov models or related models, e.g. semi-Markov models; Markov random fields; Networks embedding Markov models
• CPC Class Codes: G06V 10/85 Markov-related models; Markov random fields
• CPC Class Codes: G06V 20/49 Segmenting video sequences, i.e. computational techniques such as parsing or cutting the sequence, low-level clustering or determining units such as shots or scenes
• CPC Class Codes: H04N 7/147 Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals selecting H04Q

Abstract

An example method is provided and includes receiving a media file that includes video data and audio data; determining an initial scene sequence in the media file; determining an initial speaker sequence in the media file; and updating a selected one of the initial scene sequence and the initial speaker sequence in order to generate an updated scene sequence and an updated speaker sequence respectively. The initial scene sequence is updated based on the initial speaker sequence, and wherein the initial speaker sequence is updated based on the initial scene sequence.

Claims

1. A method, comprising:

receiving a media file that includes video data and audio data;

determining an initial scene sequence in the media file;

determining an initial speaker sequence in the media file; and

updating a selected one of the initial scene sequence and the initial speaker sequence in order to generate an updated scene sequence and an updated speaker sequence respectively, wherein the initial scene sequence is updated based on the initial speaker sequence, and wherein the initial speaker sequence is updated based on the initial scene sequence.

2. The method of claim 1, further comprising:

detecting a plurality of scenes and a plurality of speakers in the media file.

3. The method of claim 1, further comprising:

modeling the video data as a hidden Markov Model (HMM) with hidden states corresponding to different scenes of the media file; and

modeling the audio data as another HMM with hidden states corresponding to different speakers of the media file.

4. The method of claim 1, wherein updating the initial scene sequence comprises:

computing a conditional probability of the initial scene sequence given the initial speaker sequence;

estimating the updated scene sequence based on at least the conditional probability of the initial scene sequence given the initial speaker sequence;

comparing the updated scene sequence with the initial scene sequence; and

updating the initial determined scene sequence to the updated scene sequence if there is a difference between the updated scene sequence and the initial scene sequence.

5. The method of claim 1, further comprising:

estimating an initial conditional probability of the initial scene sequence given the initial speaker sequence through off-line training sequences using supervised learning algorithms.

6. The method of claim 1, further comprising:

estimating an initial conditional probability of the initial scene sequence given the initial speaker sequence through off-line training sequences using unsupervised learning algorithms.

7. The method of claim 1, wherein updating the initial speaker sequence comprises:

computing a conditional probability of the initial speaker sequence given the initial scene sequence;

estimating the updated speaker sequence based on at least the conditional probability of the initial speaker sequence given the initial scene sequence;

comparing the updated speaker sequence with the initial speaker sequence; and

updating the initial determined speaker sequence to the updated speaker sequence if there is a difference between the updated speaker sequence and the initial speaker sequence.

8. The method of claim 1, further comprising:

estimating an initial conditional probability of the initial speaker sequence given the initial scene sequence through off-line training sequences using supervised learning algorithms.

9. The method of claim 1, further comprising:

estimating an initial conditional probability of the initial speaker sequence given the initial scene sequence through off-line training sequences using unsupervised learning algorithms.

10. An apparatus, comprising:

a memory configured to store data; and

a processor that executes instructions associated with the data, wherein the processor and the memory cooperate such that the apparatus is configured for;

receiving a media file that includes video data and audio data;

determining an initial scene sequence in the media file;

determining an initial speaker sequence in the media file; and

updating a selected one of the initial scene sequence and the initial speaker sequence in order to generate an updated scene sequence and an updated speaker sequence respectively, wherein the initial scene sequence is updated based on the initial speaker sequence, and wherein the initial speaker sequence is updated based on the initial scene sequence.

11. The apparatus of claim 10, wherein the apparatus is further configured for:

modeling the video data as a HMM with hidden states corresponding to different scenes of the media file; and

modeling the audio data as another HMM with hidden states corresponding to different speakers of the media file.

12. The apparatus of claim 10, wherein updating the scene sequence comprises:

computing a conditional probability of the initial scene sequence given the initial speaker sequence;

estimating the updated scene sequence based on at least the conditional probability of the initial scene sequence given the initial speaker sequence;

comparing the updated scene sequence with the initial scene sequence; and

updating the initial determined scene sequence to the updated scene sequence if there is a difference between the updated scene sequence and the initial scene sequence.

13. The apparatus of claim 10, wherein the apparatus is further configured for:

estimating an initial conditional probability of the initial scene sequence given the initial speaker sequence through off-line training sequences using supervised learning algorithms.

14. The apparatus of claim 10, wherein updating the speaker sequence comprises:

computing a conditional probability of the initial speaker sequence given the initial scene sequence;

estimating the updated speaker sequence based on at least the conditional probability of the initial speaker sequence given the initial scene sequence;

comparing the updated speaker sequence with the initial speaker sequence; and

updating the initial determined speaker sequence to the updated speaker sequence if there is a difference between the updated speaker sequence and the initial speaker sequence.

15. The apparatus of claim 10, wherein the apparatus is further configured for:

estimating an initial conditional probability of the initial speaker sequence given the initial scene sequence through off-line training sequences using supervised learning algorithms.

16. Logic encoded in non-transitory media that includes code for execution and when executed by a processor is operable to perform operations comprising:

receiving a media file that includes video data and audio data;

determining an initial scene sequence in the media file;

determining an initial speaker sequence in the media file; and

updating a selected one of the initial scene sequence and the initial speaker sequence in order to generate an updated scene sequence and an updated speaker sequence respectively, wherein the initial scene sequence is updated based on the initial speaker sequence, and wherein the initial speaker sequence is updated based on the initial scene sequence.

17. The logic of claim 16, wherein the updating the scene sequence comprises:

computing a conditional probability of the initial scene sequence given the initial speaker sequence;

estimating the updated scene sequence based on at least the conditional probability of the initial scene sequence given the initial speaker sequence;

comparing the updated scene sequence with the initial scene sequence; and

updating the initial determined scene sequence to the updated scene sequence if there is a difference between the updated scene sequence and the initial scene sequence.

18. The logic of claim 16, the operations further comprising:

estimating an initial conditional probability of the initial scene sequence given the initial speaker sequence through off-line training sequences using supervised learning algorithms.

19. The logic of claim 16, wherein updating the speaker sequence comprises:

computing a conditional probability of the initial speaker sequence given the initial scene sequence;

estimating the updated speaker sequence based on at least the conditional probability of the initial speaker sequence given the initial scene sequence;

comparing the updated speaker sequence with the initial speaker sequence; and

updating the initial determined speaker sequence to the updated speaker sequence if there is a difference between the updated speaker sequence and the initial speaker sequence.

20. The logic of claim 16, the operations further comprising:

estimating an initial conditional probability of the initial speaker sequence given the initial scene sequence through off-line training sequences using supervised learning algorithms.