System and method for generating a multimedia summary of multimedia streams

• US 7,738,778 B2
• Filed: 06/28/2004
• Issued: 06/15/2010
• Est. Priority Date: 06/30/2003
• Status: Expired

• Current Assignee: Atreus Labs LLC
• Original Assignee: IPG Electronics 503 Limited (Coller IP Capital Limited)
• Inventors: Agnihotri, Lalitha A., Dimitrova, Nevenka
• Primary Examiner(s): Banks Harold, Marsha D
• Assistant Examiner(s): TEKLE, DANIEL T
• Application Number:
  US10/562,538
• Publication Number:
  US 20060165379A1
• Time in Patent Office: 2,178 Days
• Field of Search: 386/98, 348/700
• US Class Current: 386/231
• CPC Class Codes: G06F 16/735 Filtering based on additional data, e.g. user or group profiles
• CPC Class Codes: G06F 16/739 in form of a video summary, e.g. the video summary being a video sequence, a composite still image or having synthesized frames
• CPC Class Codes: G06F 16/7834 using audio features
• CPC Class Codes: G06F 16/7844 using original textual content or text extracted from visual content or transcript of audio data
• CPC Class Codes: G06F 16/785 using colour or luminescence
• CPC Class Codes: G06F 16/7854 using shape G06F16/7837 takes precedence
• CPC Class Codes: G06F 16/786 using motion, e.g. object motion or camera motion
• CPC Class Codes: G11B 27/28 by using information signals recorded by the same method as the main recording G11B27/22 takes precedence
• CPC Class Codes: H04N 21/2335 involving reformatting operations of audio signals, e.g. by converting from one coding standard to another details of audio signal transcoding G10L19/173
• CPC Class Codes: H04N 21/234354 by altering signal-to-noise ratio parameters, e.g. requantization
• CPC Class Codes: H04N 21/2402 Monitoring of the downstream path of the transmission network, e.g. bandwidth available traffic monitoring in data switching networks H04L43/00; monitoring data switching networks utilization H04L43/0876
• CPC Class Codes: H04N 21/25808 Management of client data terminal profiles in network data switching protocols H04L67/303
• CPC Class Codes: H04N 21/25891 being end-user preferences retrieval of video data in a video database based on user preferences G06F16/739; arrangements for recognizing users' preferences H04H60/46; user profiles in network data switching protocols H04L67/306; processing of user preferences or user profiles in wireless networks H04W8/18
• CPC Class Codes: H04N 21/26216 involving the channel capacity, e.g. network bandwidth admission control, resource allocation in open networks H04L12/5692; flow control in packet networks H04L47/10; establishing a schedule or organising the servicing of application requests taking into account QoS H04L67/61
• CPC Class Codes: H04N 21/2662 Controlling the complexity of the video stream, e.g. by scaling the resolution or bitrate of the video stream based on the client capabilities
• CPC Class Codes: H04N 21/4532 involving end-user characteristics, e.g. viewer profile, preferences monitoring of user activities for profile generation for accessing a video database G06F16/739; user profiles in network data switching protocols H04L67/306; processing of user preferences or user profiles in wireless networks H04W8/18
• CPC Class Codes: H04N 21/47 End-user applications
• CPC Class Codes: H04N 21/4755 for defining user preferences, e.g. favourite actors or genre retrieval personalisation and generation of user profiles for the retrieval of video data G06F16/739; user profiles in network data switching protocols H04L67/306
• CPC Class Codes: H04N 21/6131 involving transmission via a mobile phone network wireless downlink channel access H04W74/006
• CPC Class Codes: H04N 21/6582 Data stored in the client, e.g. viewing habits, hardware capabilities, credit card number arrangements where receivers interact with the broadcast H04H20/38
• CPC Class Codes: H04N 21/84 : Generation or processing of descriptive data, e.g. content descriptors systems specially adapted for using meta-information in broadcast systems H04H60/73
• CPC Class Codes: H04N 21/8453 : by locking or enabling a set of features, e.g. optional functionalities in an executable program
• CPC Class Codes: H04N 21/8549 : Creating video summaries, e.g. movie trailer retrieval in video databases by using presentations in form of a video summary G06F16/739
• CPC Class Codes: H04N 5/147 : Scene change detection
• CPC Class Codes: H04N 7/165 : Centralised control of user terminal subsequent to an upstream request signal H04N7/17345; Registering at central by two-way working H04N7/17309
• CPC Class Codes: H04N 7/17318 : Direct or substantially direct transmission and handling of requests
• CPC Class Codes: H04N 7/17336 : Handling of requests in head-ends

Abstract

A system facilitates and enhances review of one or more multimedia input streams that includes some combination of video, audio and text information, generating a multimedia summary, thereby enabling a user to better browse and/or decide on viewing the multimedia input streams in their entirety. The multimedia summary is constructed automatically, based in part on system specifications, user specifications and network and device constraints. In a particular application of the invention, the input multimedia streams represent news broadcasts (e.g., television news program, video vault footage). In such a particular application, the invention can enable the user to automatically receive a summary of the news stream in accordance with previously provided user preferences and in accordance with prevailing network and user device constraints.

Claims

1. A method for producing a multimedia summary of at least one multimedia stream the summary comprising key elements selected from said at least one multimedia stream, the method comprising:

a.) one of receiving and retrieving said at least one multimedia stream comprising video, audio and text information;

b.) dividing the at least one multimedia stream into a video substream, an audio sub-stream and a text sub-stream;

c.) identifying for potential inclusion in the summary video, audio and text key elements from said video, audio and text sub-streams, respectively;

d.) computing an importance value for the identified video, audio and text key elements identified at said step (c);

e.) first filtering the identified video, audio and text key elements to exclude those key elements whose associated importance value is less than a pre-defined video, audio and text importance threshold, respectively; and

f.) second filtering the remaining key elements from said step (e) in accordance with a user profile;

g.) third filtering the remaining key elements from said step (f) in accordance with network and user device constraints; and

h.) outputting a multimedia summary which comprises key elements remaining from said step (g).

2. The method of claim 1, wherein said at least one multimedia stream is one of an analog and digital multimedia stream.

3. The method of claim 1, wherein the step of dividing the at least one multimedia stream into a video sub-stream further comprises the step of identifying and grouping said at least one multimedia stream into a plurality of news stories where each identified news story is comprised of an anchor portion and a reportage portion.

4. The method of claim 1, wherein the step of dividing the at least one multimedia stream into an audio sub-stream further comprises dividing said at least one multimedia stream into a plurality of equal-sized frames of a fixed time duration.

5. The method of claim 1, wherein the step of dividing the at least one multimedia stream into a text sub-stream further comprises dividing said at least one multimedia stream into a plurality of frames wherein each frame of said plurality of frames is defined on a word boundary.

6. The method of claim 1, wherein the act of identifying video, audio and text key elements from said video, audio and text sub-streams further comprise the acts of:

1.) identifying low, mid and high level features from the plurality of frames which comprise said video, audio and text, substreams;

2.) determining an importance value to each of said extracted low, mid and high level features from said identifying act;

3.) computing a frame importance value for each of said plurality of frames which comprise said video, audio and text sub-streams as a function of the importance values of the feature importance values determined at said determining act;

4.) combining the frames into segments in each of said video, audio and text sub-streams;

5.) computing an importance value per segment for each segment from said combining act;

6.) ranking the segments based on said computed importance value at said computing step; and

7.) identifying key elements based on said ranked segments.

7. The method of claim 6, wherein said act (3). of computing a frame importance value for each of said extracted low, mid and high level features further comprises computing said importance value by one of deterministic, statistical and conditional probability means.

8. The method of claim 7, wherein said conditional probability means comprises computing said frame importance value as one of a Gaussian, Poisson, Rayleigh and Bernoulli distribution.

9. The method of claim 8, further comprising computing the Gaussian distribution according to the equation:

10. The method of claim 7, wherein said deterministic means comprises computing said frame importance value according to the equation:

Σ

w_if_i

where;

f₁ represent low, mid-level and high-level features; and

w₁ represent weighting factors for weighting said features.

11. The method of claim 6, wherein said step (4) of combining the frames into video segments further comprises combining said frames by one of family histogram computation means and shot change detection means.

12. The method of claim 6, wherein said step (4) of combining the frames into audio segments further comprises the steps of:

categorizing each frame from said audio sub-stream as one of a speech frame, a music frame, a silence frame, a noise frame, a speech + speech frame, a speech + noise frame and a speech + music frame; and

grouping consecutive frames having the same categorization.

13. The method of claim 6, wherein act step (4) of combining the frames into text segments further comprises combining said frames based on punctuation included in said text sub-stream.

14. The method of claim 6, wherein said step (5) of computing an importance value per segment further comprises averaging the frame importance values for those frames which comprise said segment.

15. The method of claim 6, wherein said step (5) of computing an importance value per segment further comprises using the highest frame importance value in said segment.

16. The method of claim 6, wherein said step (7) of identifying key elements based on said rankings further comprises identifying key elements whose segment ranking exceeds a predetermined segment ranking threshold.

17. The method of claim 6, wherein said step (7) of identifying key elements based on said rankings further comprises identifying key elements whose segment ranking both exceeds a predetermined segment ranking threshold and constitute a local maxima.

18. The method of claim 6, wherein said step (7) of identifying key elements based on said rankings further comprises identifying key elements whose segment ranking constitutes a local maxima.

19. A system for producing a multimedia summary of at least one multimedia stream, the summary comprising key elements selected from said at last one multimedia stream, the system comprising:

a modality recognition and division (MRAD) module comprising a story segment identifier (SSI) module, an audio identifier (AI) module and a text identifier (TI) module, the MRAD module communicatively coupled to a first external source for receiving said at least one multimedia stream, the MRAD module communicatively coupled to a second external source for receiving said at least one multimedia stream, the MRAD module dividing said at least one multimedia stream into a video, an audio and a text sub-stream and outputting said video, audio and text sub-streams to a key element identifier (KEI) module, the KEI module comprising a feature extraction (FE) module and an importance value (IV) module for identifying key elements from within said video, audio and text sub-streams and assigning importance values thereto, the KEI module communicatively coupled to a key element filter (KEF) for receiving the identified key elements and filtering said key elements that exceed a pre-determined threshold criteria, the KEF module communicatively coupled to a user profile filter (UPF) for receiving filtered key elements and further filtering said filtered key elements in accordance with a user profile, the UPF module communicatively coupled to a network and device constraint (NADC) module, said NADC module receiving said further filtered key elements and further filtering said further filtered key elements in accordance with network and/or user device constraints, the NADC module outputting a multimedia summary of said at least one multimedia stream which comprises key elements remaining after said further filtering in the NADC module.

20. The system of claim 19, further comprising a user preference database communicatively coupled to said UPF module for storing user profiles.

21. The system of claim 19, wherein the first external source is a broadcast channel selector.

22. The system of claim 19, wherein the first external source is a video streaming source.

23. The system of claim 19, wherein said at least one multimedia stream is one of an analog and digital multimedia stream.

24. The system of claim 19, wherein the NADC module is communicatively connected to an external network coupled to a user device.

25. The system of claim 19, wherein the network is the Internet.

26. A non-transitory computer readable medium having computer readable code means embodied thereon, said computer readable program code means comprising:

an act of one of receiving and retrieving said at least one multimedia stream comprising video, audio and text information;

an act of dividing said at least one multimedia stream into a video sub-stream, an audio sub-stream and a text sub-stream an act of identifying for potential inclusion in the summary video, audio and text key elements from said video, audio and text sub-streams, respectively;

an act of computing an importance value for the identified video, audio and text key elements identified at said identification act;

an act of first filtering the identified video, audio and text key elements to exclude those key elements whose associated importance value is less than a pre-defined video, audio and text importance threshold, respectively; and

an act of second filtering the remaining key elements from said first filtering act in accordance with a user profile;

an act of third filtering the remaining key elements from said second filtering act in accordance with network and user device constraints; and

an act of outputting a multimedia summary which comprises key elements remaining from said third filtering act.

27. A non-transitory computer readable medium of claim 26 further wherein the act of identifying video, audio and text key elements from said video, audio and text sub streams, respectively, further comprises:

an act of identifying low, mid and high level features from the plurality of frames which comprise said video, audio and text sub-streams;

an act of determining an importance value to each of said extracted low, mid and high level features from said identifying act;

an act of computing a frame importance value for each of said plurality of frames which comprise said video , audio and text sub-streams as a function of the importance values of the feature importance values determined at said determining step;

an act of combining the frames into segments in each of said video, audio and text sub-streams;

an act of computing an importance value per segment for each segment from said combining act;

an act of ranking the segments based on said computed importance value at said computing act; and

an act of identifying key elements based on said ranked segments.