Intelligent video summaries in information access

• US 8,296,797 B2
• Filed: 10/18/2006
• Issued: 10/23/2012
• Est. Priority Date: 10/19/2005
• Status: Active Grant

• Current Assignee: Microsoft Corporation
• Original Assignee: Microsoft International Holdings BV (Microsoft Corporation)
• Inventors: Olstad, Bjørn, Arentz, Will
• Primary Examiner(s): SHEPARD, JUSTIN E
• Application Number:
  US11/793,389
• Publication Number:
  US 20080097970A1
• Time in Patent Office: 2,197 Days
• Field of Search: 725/120
• US Class Current: 725/41
• CPC Class Codes: G06F 16/738 Presentation of query results
• CPC Class Codes: G06F 16/739 in form of a video summary, e.g. the video summary being a video sequence, a composite still image or having synthesized frames
• CPC Class Codes: G06F 16/78 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/7844 using original textual content or text extracted from visual content or transcript of audio data
• CPC Class Codes: G06F 16/9577 Optimising the visualization of content, e.g. distillation of HTML documents
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/0485 Scrolling or panning
• CPC Class Codes: G10L 15/02 Feature extraction for speech recognition; Selection of recognition unit
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 2015/025 Phonemes, fenemes or fenones being the recognition units
• CPC Class Codes: H04L 67/02 based on web technology, e.g. hypertext transfer protocol [HTTP]
• CPC Class Codes: H04N 21/43 Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware real-time communication protocols in data switching networks H04L65/00
• CPC Class Codes: H04N 21/431 Generation of visual interfaces for content selection or interaction; Content or additional data rendering
• CPC Class Codes: H04N 21/4312 involving specific graphical features, e.g. screen layout, special fonts or colors, blinking icons, highlights or animations
• CPC Class Codes: H04N 21/454 Content or additional data filtering, e.g. blocking advertisements
• CPC Class Codes: H04N 21/462 Content or additional data management, e.g. creating a master electronic program guide from data received from the Internet and a Head-end, controlling the complexity of a video stream by scaling the resolution or bit-rate based on the client capabilities
• CPC Class Codes: H04N 21/47 End-user applications
• CPC Class Codes: H04N 21/472 End-user interface for requesting content, additional data or services; End-user interface for interacting with content, e.g. for content reservation or setting reminders, for requesting event notification, for manipulating displayed content end-user interfaces for retrieving video data from a database G06F16/739; network services for supporting unicast streaming H04L65/612
• CPC Class Codes: H04N 21/47202 for requesting content on demand, e.g. video on demand
• CPC Class Codes: H04N 21/482 : End-user interface for program selection broadcast systems using EPGs H04H60/72
• CPC Class Codes: H04N 21/4828 : for searching program descriptors retrieval of video data G06F16/739

Abstract

In a method for displaying video data within result presentations in information access or search systems, compressed thumbnails are computed for videos in a result page and applied to the result page. An end user is enabled to select a video in the result page and activate streaming of a video summary within the context of the associated result page.

Claims

1. A method for displaying video data within result presentations in information access systems or information search systems, wherein the method comprises:

applying a query to a set of videos;

identifying matching videos in the set of videos;

producing a result page with at least one of the matching videos;

pre-computing compressed thumbnails for the matching videos;

applying one or more of the pre-computed compressed thumbnails to the result page;

enabling a user selection of a video in the result page;

generating a video summary that provides to an end user an abbreviated review of a complete video, the video summary comprising a plurality of frames from the complete video including frames near the beginning of the video and frames near the end of the video but not all of the frames from the complete video, the complete video being one of the matching videos, wherein generating the video summary comprises;

identifying, in the video, a frame sequence whose corresponding audio track contains a query term in the query; and

including the frame sequence in the video summary; and

activating, by a server, streaming of the video summary displayed within the context of the result page in response to the user selection of the video.

2. A method according to claim 1, wherein the set of videos contains at least one of:

video files, webinars, streaming video sources or live video broadcasts.

3. A method according to claim 1, further comprising matching the query based on one of:

textual metadata associated with the video, structured metadata associated with the video, text extracted from an audio track by speech-to-text analysis, or phonetic transcription extracted from the audio track;

or combinations thereof, the structured metadata associated with the video including at least one of;

video size, video length, video quality or video format.

4. A method according to claim 1, further comprising optimizing the pre-computed compressed thumbnails to display capabilities of a client device used to view the result page.

5. A method according to claim 1, further comprising using at least one still frame from a given one of the matching videos in the pre-computed compressed thumbnail for the given one of the matching videos.

6. A method according to claim 5, further comprising selecting the at least one still frame in response to occurrence of query terms inside an audio track of the given one of the videos, the query terms being terms in the query.

7. A method according to claim 1, wherein the end user selects the video by either hovering, mousing over, scrolling, or clicking on the video in the result page.

8. A method according to claim 1, wherein the result page is a web page viewed inside a browser.

9. A method according to claim 1, wherein activating streaming of the video summary comprises streaming the video summary to a client device, the client device being a wireless mobile device.

10. A method according to claim 9, further comprising presenting the result page by scrolling through the videos in the result page individually in full screen mode.

11. A method according to claim 1, further comprising optimizing the video summary to a display capability of a client device used to view the result page.

12. A method according to claim 1, further comprising constructing the video summary by combining multiple time sequences from the video.

13. A method according to claim 12, further comprising selecting the time sequences in response to scene changes in the video, motion analysis in the video, or audio analysis of the video;

or combinations thereof, the query terms being terms in the query.

14. A method according to claim 12, further comprising constructing an audio track of the video summary by combining audio tracks associated with the time sequences.

15. A method according to claim 1, wherein activating streaming of the video summary comprises activating streaming of the video summary to a client device that caches video summaries after streaming, such that subsequent replays do not require additional streaming of video data.

16. A method according to claim 1, further comprising encoding the video summary with flash technology.

17. A method according to claim 1, wherein playback of the video summary is in slow motion.

18. A method according to claim 1,

wherein activating streaming of the video summary comprises:

activating streaming of the video summary to a client device; and

wherein the method further comprises;

constructing the video summary adaptively such that streaming requirements match capabilities of a transfer link to the client device.

19. A method according to claim 1, further comprising integrating advertising information inside the video summary.

20. A method according to claim 19, further comprising integrating the advertising information by mixing advertising video sequences with selected sequences from the video.

21. A method according to claim 20, further comprising including in the video summary one of:

sponsored links from paid web search engines and which can be activated by the end user during viewing/streaming of the video summary, actionable widgets inside a view of the video summary that enable actions relating the end user to an advertiser, or display and means for the end user to buy a multimedia object such as a song or a video;

or combinations thereof.

22. A method for displaying video data within a result page, the method comprising:

receiving a query from a client device;

applying the query to a set of videos containing at least one of;

video files, webinars, streaming video sources and live video broadcasts;

identifying matching videos in the set of videos based on at least one of;

textual metadata associated with the videos, and

structured metadata associated with the videos, the structured metadata including at least one of;

video size, video length, video quality or video format, text extracted from an audio track by speech-to-text analysis, or phonetic transcription extracted from the audio track;

producing a result page with at least one of the matching videos, wherein the result page is a web page viewed inside a browser;

pre-computing compressed thumbnails for the matching videos using at least one still frame from each of the matching videos in the pre-computed compressed thumbnails for the matching videos, wherein the at least one still frame for each of the matching videos is selected in response to occurrences of query terms inside audio tracks of the matching videos;

optimizing the pre-computed compressed thumbnails to display capabilities of the client device;

applying one or more of the pre-computed compressed thumbnails to the result page;

enabling a user selection of a video in the result page, said user selecting the video by either hovering, mousing over, scrolling or clicking on the video in the result page;

selecting at least one time sequence in the video in response to one of;

occurrences of query terms inside an audio track of the video, scene changes in the video, motion analysis in the video, or audio analysis of the video, or combinations thereof;

constructing a video summary by combining the at least one time sequence from an original video of the result page, wherein the video summary is constructed adaptively such that streaming requirements of the video summary match capabilities of a transfer link to the client device;

constructing an audio track of the video summary by combining audio tracks associated with the at least one time sequence;

optimizing the video summary to the display capabilities of the client device;

encoding the video summary with flash technology;

integrating advertising information inside the video summary by mixing advertising video sequences with selected sequences from the video; and

activating, by a server, streaming of the video summary to the client device such that the client device displays the video summary within the context of the result page, wherein playback of the video summary is in slow motion.