Annotating Video Segments Using Feature Rhythm Models

• US 20080310709A1
• Filed: 06/18/2007
• Published: 12/18/2008
• Est. Priority Date: 06/18/2007
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Kender, John R.
• Granted Patent: US 8,126,262 B2
• Time in Patent Office: Days
• US Class Current: 382/156
• CPC Class Codes: G06F 16/70 of video data
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: H04N 5/147 Scene change detection

Abstract

Each video segment in a plurality of video segments is annotated with an indicator of the likelihood that the respective video segment shows a particular feature. The plurality of video segments forms an episode of interest from a given video domain. Initial feature probabilities are calculated for respective ones of the plurality of video segments using a machine learning algorithm. Each initial feature probability indicates the likelihood that its respective video segment shows the particular feature. Refined feature probabilities are determined for respective ones of the plurality of video segments by finding the most probable state sequence in a finite state machine. This is accomplished at least in part using the determined initial feature probabilities. Finally, each of the video segments in the plurality of vides segments is annotated with its respective refined feature probability.

Claims

1. A method of annotating each video segment in a plurality of video segments with an indicator of the likelihood that the respective video segment shows a particular feature, the plurality of video segments forming an episode of interest from a given video domain, the method comprising the steps of:

determining initial feature probabilities for respective ones of the plurality of video segments using a machine learning algorithm, each initial feature probability indicating the likelihood that its respective video segment shows the particular feature;

determining refined feature probabilities for respective ones of the plurality of video segments, the refined feature probabilities determined by finding the most probable state sequence in a finite state machine, the most probable state sequence determined at least in part using the calculated initial feature probabilities; and

annotating each of the video segments in the plurality of video segments with its respective refined feature probability.

2. The method of claim 1, wherein the machine learning algorithm comprises a Neural Network.

3. The method of claim 1, wherein the machine learning algorithm comprises a Bayesian Network.

4. The method of claim 1, wherein the machine learning algorithm comprises a Support Vector Machine.

5. The method of claim 4, wherein the step of determining initial feature probabilities for respective ones of the video segments comprises converting results derived from the one or more machine learning algorithms to probabilities.

6. The method of claim 1, wherein the particular feature belongs to a predetermined ontology of features.

7. The method of claim 1, wherein the finite state machine comprises a plurality of transition probabilities determined by applying an n-th order Markov dependency to the manner in which the particular feature is shown in one or more training episodes, the one or more training episodes from the same video domain as the episode of interest, and n being an integer.

8. The method of claim 7, wherein n is greater than one.

9. The method of claim 7, wherein n is equal to three.

10. The method of claim 1, wherein the step of determining the most probable state sequence in the finite state machine comprises applying a Viterbi Algorithm to the finite state machine.

11. The method of claim 1, wherein the given video domain comprises at least one of news, drama, comedy, sports, and documentary.

12. An article of manufacture comprising a processor-readable storage medium storing one or more programs for annotating each video segment in a plurality of video segments with an indicator of the likelihood that the respective video segment shows a particular feature, the plurality of video segments forming an episode of interest in a given video domain, the one or more programs, when executed by a data processing system comprising a memory and a processor coupled to the memory, operative to cause the data processing system to perform at least the steps of:

determining initial feature probabilities for respective ones of the plurality of video segments using a machine learning algorithm, each initial feature probability indicating the likelihood that its respective video segment shows the particular feature

determining refined feature probabilities for respective ones of the plurality of video segments, the refined feature probabilities determined by finding the most probable state sequence in a finite state machine, the most probable state sequence determined at least in part using the determined initial feature probabilities; and

annotating each of the video segments in the plurality of video segments with its respective refined feature probability.

13. The article of manufacture of claim 12, wherein the finite state machine comprises a plurality of transition probabilities determined by applying an n-th order Markov dependency to the manner in which the particular feature is shown in one or more training episodes, the one or more training episodes from the same video domain as the episode of interest, and n being an integer.

14. The article of manufacture of claim 12, wherein the step of determining the most probable state sequence in the finite state machine comprises applying a Viterbi Algorithm to the finite state machine.

15. A data processing system comprising a memory and a data processor coupled to the memory for annotating each video segment in a plurality of video segments with an indicator of the likelihood that the respective video segment shows a particular feature, the plurality of video segments forming an episode of interest in a given video domain, the data processing system operative to perform the steps of:

determining initial feature probabilities for respective ones of the plurality of video segments using a machine learning algorithm, each initial feature probability indicating the likelihood that its respective video segment shows the particular feature;

determining refined feature probabilities for respective ones of the plurality of video segments, the refined feature probabilities determined by finding the most probable state sequence in a finite state machine, the most probable state sequence determined at least in part using the determined initial feature probabilities; and

annotating each of the video segments in the plurality of video segments with its respective refined feature probability.

16. The data processing system of claim 15, wherein the data processing system receives at least a portion of the finite state machine from hardware external to the data processing system.

17. The data processing system of claim 15, wherein the finite state machine comprises a plurality of transition probabilities determined by applying an n-th order Markov dependency to the manner in which the particular feature is shown in one or more training episodes, the one or more training episodes from the same video domain as the episode of interest, and n being an integer.

18. The data processing system of claim 15, wherein n is greater than one.

19. The data processing system of claim 15, wherein n is equal to three.

20. The data processing system of claim 15, wherein the step of determining the most probable state sequence in the finite state machine comprises applying a Viterbi Algorithm to the finite state machine.