Annotating programs for automatic summary generations

• US 20050160457A1
• Filed: 03/15/2005
• Published: 07/21/2005
• Est. Priority Date: 09/13/1999
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Rui, Yong, Gupta, Anoop, Acero, Alejandro
• Granted Patent: US 7,403,894 B2
• Time in Patent Office: Days
• US Class Current: 725/45
• CPC Class Codes: G06F 16/634 Query by example, e.g. query by humming
• CPC Class Codes: G06F 16/68 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/683 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/739 in form of a video summary, e.g. the video summary being a video sequence, a composite still image or having synthesized frames
• CPC Class Codes: G06F 16/7834 using audio features
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44

Abstract

Audio/video programming content is made available to a receiver from a content provider, and meta data is made available to the receiver from a meta data provider. The meta data corresponds to the programming content, and identifies, for each of multiple portions of the programming content, an indicator of a likelihood that the portion is an exciting portion of the content. In one implementation, the meta data includes probabilities that segments of a baseball program are exciting, and is generated by analyzing the audio data of the baseball program for both excited speech and baseball hits. The meta data can then be used to generate a summary for the baseball program.

Claims

1. One or more computer readable media having stored thereon a plurality of instructions that, when executed by one or more processors, causes the one or more processors to perform acts including:

receiving the audio portion of a sporting event;

classifying a set of segments of the audio portion as excited speech;

classifying a set of frame groupings as including baseball hits;

combining the set of segments and the set of frame groupings to identify probabilities for each segment that the segment is an exciting segment; and

saving an indication of the set of segments and the corresponding probabilities as meta data corresponding to the sporting event.

4. One or more computer readable media as recited in claim 1, wherein the classifying a set of segments as excited speech comprises:

extracting a first set of features from the audio portion;

identifying, based on the first set of features, a plurality of windows of the audio portion that include speech;

extracting a second set of features from the audio portion; and

identifying, based on the second set of features, which of the plurality of windows include excited speech.

5. One or more computer readable media as recited in claim 4, wherein extracting the first set of features comprises, for each of a plurality of windows:

identifying an average waveform amplitude of the audio portion in a first frequency band of the window;

identifying an average waveform amplitude of the audio portion in a second frequency band of the window;

concatenating the identified average waveform amplitudes to generate an energy feature of the first set of features; and

determining, for an MFCC feature of the first set of features, the Mel-frequency Cepstral coefficient of the window.

7. One or more computer readable media as recited in claim 4, wherein extracting the second set of features comprises, for each of a plurality of windows:

identifying, for each of a plurality of frames in the window, an average waveform amplitude of the audio portion in a first frequency band;

concatenating the identified average waveform amplitudes to generate an energy feature;

extracting, as a pitch feature of each frame, the pitch of each frame;

identifying a plurality of statistics regarding each window based on the energy features and pitch features of the plurality of frames.

9. One or more computer readable media as recited in claim 7, wherein the identifying which of the plurality of windows include excited speech comprises identifying a posterior probability that the window corresponds to an excited speech class and identifying a posterior probability that the window corresponds to a non-excited speech class, and classifying the window in the class with the higher posterior probability.

10. One or more computer readable media as recited in claim 4, further comprising instructions that cause the one or more processors to perform acts including outputting an excited speech probability for each of the segments that include excited speech, the excited speech probability for a segment indicating a likelihood that the segment includes excited speech.

14. One or more computer readable media as recited in claim 1, wherein the combining comprises generating, for each of the set of segments, a weighted sum of a probability that the segment includes excited speech and a probability that a frame grouping within the segment includes a baseball hit.

2. (canceled)

3. (canceled)

6. (canceled)

8. (canceled)

11. (canceled)

12. (canceled)

13. (canceled)

15-61. -61. (canceled)

62. A system comprising:

a feature extractor to extract a plurality of audio features from programming content;

an excited speech classification subsystem to identify, based on a sub-set of the audio features, a set of segments of the programming content and corresponding probabilities that the segments include excited speech;

a baseball hit detection subsystem to identify, based on another sub-set of the audio features, a set of frame groupings of the programming content and corresponding probabilities that the frame groupings include baseball hits; and

a probabilistic fusion subsystem to combine the probabilities that the segments include excited speech and the probabilities that the frame groupings include baseball hits, and to generate a probability that portions of the programming content are exciting based on the combination.

63. A system as recited in claim 62, wherein the excited speech classification subsystem is to identify the set of segments by:

identifying, based on a first set of the plurality of audio features, a plurality of windows of the programming content that include speech;

identifying, based on a second set of the plurality of audio features, which of the plurality of windows include excited speech.

64. A system as recited in claim 62, wherein the baseball hit detection subsystem is to identify the set of frame groupings by:

combining, for each of a plurality of multiple-frame groupings, a set of features from the programming content;

comparing the sets of features from the multiple-frame groupings to a set of templates; and

identifying, for each of the multiple-frame groupings, a probability that the grouping includes a baseball hit based on how well the grouping matches the set of templates.

65. (canceled)

66. A method comprising:

analyzing audio data of a program to identify a first plurality of portions of the program each including excited speech;

analyzing the audio data to identify a second plurality of portions of the program each including a potential sport-related event; and

combining the first plurality of portions and the second plurality of portions to identify a set of segments of the program and a likelihood, for each of the segments in the set, that the segment is an exciting part of the program.

67. A method of claim 66 wherein the sport-related event is a baseball hit.

68. A method as recited in claim 66, wherein the analyzing audio data to identify the first plurality of portions comprises:

extracting a first set of features from the audio data;

identifying, based on the first set of features, a plurality of windows of the audio data that include speech;

extracting a second set of features from the audio data; and

identifying, based on the second set of features, which of the plurality of windows include excited speech.

69. A method as recited in claim 68, wherein extracting the second set of features comprises, for each of a plurality of windows:

identifying, for each of a plurality of frames in the window, an average waveform amplitude of the audio data in a first frequency band;

identifying, for each of the plurality of frames in the window, an average waveform amplitude of the audio data in a second frequency band;

concatenating the identified average waveform amplitudes to generate an energy feature;

extracting, as a pitch feature of each frame, the pitch of each frame; and

identifying a plurality of statistics regarding each window based on the energy features and pitch features of the plurality of frames.

70. A method as recited in claim 69, wherein the identifying which of the plurality of windows include excited speech comprises identifying a posterior probability that the window corresponds to an excited speech class and identifying a posterior probability that the window corresponds to a non-excited speech class, and classifying the window in the class with the higher posterior probability.

71. A method as recited in claim 68, further comprising outputting an excited speech probability corresponding to each of a plurality segments that include excited speech, each segment including a plurality of windows, the excited speech probability for a segment indicating a likelihood that the segment includes excited speech.

72. A method as recited in claim 66, wherein the analyzing audio data to identify the second plurality of portions comprises:

extracting, for each frame in a multiple-frame grouping, a set of features from the audio data;

comparing the set of features from the multiple-frame groupings to a set of templates; and

identifying, for each of the multiple-frame groupings, a probability that the grouping includes a sport-related event based on how well the grouping matches the set of templates.

73. A method of claim 72 wherein the sport-related event is a baseball hit.

74. A method as recited in claim 72, wherein the combining comprises adjusting the probability that a portion includes excited speech based on the probability that the portion includes a sport-related event.

75. A method of claim 74 wherein the sport-related event is a baseball hit.