Intelligent video summaries in information access

• US 9,122,754 B2
• Filed: 10/22/2012
• Issued: 09/01/2015
• Est. Priority Date: 10/19/2005
• Status: Active Grant

• Current Assignee: Microsoft Corporation
• Original Assignee: Microsoft International Holdings BV (Microsoft Corporation)
• Inventors: Olstad, Bjorn, Arentz, Will
• Primary Examiner(s): SHEPARD, JUSTIN E
• Application Number:
  US13/657,573
• Publication Number:
  US 20130132374A1
• Time in Patent Office: 1,044 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 16/738 Presentation of query results
• CPC Class Codes: G06F 16/739 in form of a video summary, e.g. the video summary being a video sequence, a composite still image or having synthesized frames
• CPC Class Codes: G06F 16/78 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/7844 using original textual content or text extracted from visual content or transcript of audio data
• CPC Class Codes: G06F 16/9577 Optimising the visualization of content, e.g. distillation of HTML documents
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/0485 Scrolling or panning
• CPC Class Codes: G10L 15/02 Feature extraction for speech recognition; Selection of recognition unit
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 2015/025 Phonemes, fenemes or fenones being the recognition units
• CPC Class Codes: H04L 67/02 based on web technology, e.g. hypertext transfer protocol [HTTP]
• CPC Class Codes: H04N 21/43 Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware real-time communication protocols in data switching networks H04L65/00
• CPC Class Codes: H04N 21/431 Generation of visual interfaces for content selection or interaction; Content or additional data rendering
• CPC Class Codes: H04N 21/4312 involving specific graphical features, e.g. screen layout, special fonts or colors, blinking icons, highlights or animations
• CPC Class Codes: H04N 21/454 Content or additional data filtering, e.g. blocking advertisements
• CPC Class Codes: H04N 21/462 Content or additional data management, e.g. creating a master electronic program guide from data received from the Internet and a Head-end, controlling the complexity of a video stream by scaling the resolution or bit-rate based on the client capabilities
• CPC Class Codes: H04N 21/47 End-user applications
• CPC Class Codes: H04N 21/472 End-user interface for requesting content, additional data or services; End-user interface for interacting with content, e.g. for content reservation or setting reminders, for requesting event notification, for manipulating displayed content end-user interfaces for retrieving video data from a database G06F16/739; network services for supporting unicast streaming H04L65/612
• CPC Class Codes: H04N 21/47202 for requesting content on demand, e.g. video on demand
• CPC Class Codes: H04N 21/482 : End-user interface for program selection broadcast systems using EPGs H04H60/72
• CPC Class Codes: H04N 21/4828 : for searching program descriptors retrieval of video data G06F16/739

Abstract

In a method for displaying video data within result presentations in information access or search systems, compressed thumbnails are computed for videos in a result page and applied to the result page. An end user is enabled to select a video in the result page and activate streaming of a video summary within the context of the associated result page.

Claims

1. A system for displaying video data within a result page, comprising:

a processor and a computer-readable storage device; and

a process configured to perform actions using the processor, comprising;

receiving a query from a client device;

applying the query to a set of videos containing at least one of;

video files, webinars, streaming video sources and live video broadcasts;

identifying matching videos in the set of videos based on at least one of;

textual metadata associated with the videos, and

structured metadata associated with the videos, the structured metadata including at least one of;

video size, video length, video quality or video format, text extracted from an audio track by speech-to-text analysis, or phonetic transcription extracted from the audio track;

producing a result page with at least some of the matching videos wherein each of the matching videos of the results page are different from one another and wherein the result page is a web page viewed inside a browser;

computing compressed thumbnails for the matching videos in the result page;

enabling a user selection of each of the matching videos in the result page, said user selecting the video by either hovering, mousing over, scrolling or clicking on the video in the result page;

constructing a video summary in response to each selection from the result page of one of the matching videos that are different from one another that provides to an end user an abbreviated review of the matching video selected by the end user, the video summary comprising a plurality of frames but not all of the frames from the matching video selected by the end user;

optimizing the video summary to the display capabilities of the client device; and

activating, by a server, streaming of the video summary to the client device such that the client device displays the video summary within the context of the result page in response to the user selection of one of the matching videos.

2. A computer-readable storage device storing computer-executable instructions in combination with a processor, the computer-readable storage device and the processor configured to perform actions, comprising:

receiving a query from a client device;

applying the query to a set of videos containing at least one of;

video files, webinars, streaming video sources and live video broadcasts;

identifying matching videos in the set of videos based on at least one of;

textual metadata associated with the videos, and

structured metadata associated with the videos, the structured metadata including at least one of;

video size, video length, video quality or video format, text extracted from an audio track by speech-to-text analysis, or phonetic transcription extracted from the audio track;

producing a result page with at least some of the matching videos wherein each of the matching videos of the results page are different from one another and wherein the result page is a web page viewed inside a browser;

computing compressed thumbnails for the matching videos in the result page;

enabling a user selection of each of the matching videos in the result page, said user selecting the video by either hovering, mousing over, scrolling or clicking on the video in the result page;

constructing a video summary in response to each selection from the result page of one of the matching videos that are different from one another that provides to an end user an abbreviated review of the matching video selected by the end user, the video summary comprising a plurality of frames but not all of the frames from the matching video selected by the end user;

optimizing the video summary to the display capabilities of the client device; and

activating, by a server, streaming of the video summary to the client device such that the client device displays the video summary within the context of the result page in response to the user selection of one of the matching videos.

3. The computer-readable storage device of claim 2, wherein the set of videos contains at least one of:

video files, webinars, streaming video sources or live video broadcasts.

4. The computer-readable storage device of claim 2, further comprising matching the query based on one of:

textual metadata associated with the video, structured metadata associated with the video, text extracted from an audio track by speech-to-text analysis, or phonetic transcription extracted from the audio track;

or combinations thereof, the structured metadata associated with the video including at least one of;

video size, video length, video quality or video format.

5. The computer-readable storage device of claim 2, further comprising optimizing the pre-computed compressed thumbnails to display capabilities of a client device used to view the result page.

6. The computer-readable storage device of claim 2, further comprising using at least one still frame from a given one of the matching videos in the pre-computed compressed thumbnail for the given one of the matching videos.

7. The computer-readable storage device of claim 2, further comprising selecting the at least one still frame in response to occurrence of query terms inside an audio track of the given one of the videos, the query terms being terms in the query.

8. The computer-readable storage device of claim 2, wherein the end user selects the video by either hovering, mousing over, scrolling, or clicking on the video in the result page.

9. The computer-readable storage device of claim 2, wherein the result page is a web page viewed inside a browser.

10. The computer-readable storage device of claim 2, wherein activating streaming of the video summary comprises streaming the video summary to a client device, the client device being a wireless mobile device.

11. The computer-readable storage device of claim 2, further comprising presenting the result page by scrolling through the videos in the result page individually in full screen mode.

12. The computer-readable storage device of claim 2, further comprising optimizing the video summary to a display capability of a client device used to view the result page.

13. The computer-readable storage device of claim 2, further comprising constructing the video summary by combining multiple time sequences from the video.

14. The computer-readable storage device of claim 2, wherein each video summary of the matching videos when selected from the result page is a complete review of the respective matching video.

15. A computer-readable storage device storing computer-executable instructions, comprising:

receiving a query from a client device;

applying the query to a set of videos containing at least one of;

video files, webinars, streaming video sources and live video broadcasts;

identifying matching videos in the set of videos based on at least one of;

textual metadata associated with the videos, and

structured metadata associated with the videos, the structured metadata including at least one of;

video size, video length, video quality or video format, text extracted from an audio track by speech-to-text analysis, or phonetic transcription extracted from the audio track;

producing a result page with at least some of the matching videos wherein each of the matching videos of the results page are different from one another and wherein the result page is a web page viewed inside a browser;

computing compressed thumbnails for the matching videos in the result page;

enabling a user selection of each of the matching videos in the result page, said user selecting the video by either hovering, mousing over, scrolling or clicking on the video in the result page;

constructing a video summary in response to each selection from the result page of one of the matching videos that are different from one another that provides to an end user an abbreviated review of the matching video selected by the end user, the video summary comprising a plurality of frames but not all of the frames from the matching video selected by the end user;

optimizing the video summary to the display capabilities of the client device; and

activating, by a server, streaming of the video summary to the client device such that the client device displays the video summary within the context of the result page in response to the user selection of one of the matching videos.

16. The computer-readable storage device of claim 15, wherein the set of videos contains at least one of:

video files, webinars, streaming video sources or live video broadcasts.

17. The computer-readable storage device of claim 15, further comprising matching the query based on one of:

textual metadata associated with the video, structured metadata associated with the video, text extracted from an audio track by speech-to-text analysis, or phonetic transcription extracted from the audio track;

or combinations thereof, the structured metadata associated with the video including at least one of;

video size, video length, video quality or video format.

18. The computer-readable storage device of claim 15, further comprising optimizing the compressed thumbnails to display capabilities of a client device used to view the result page.

19. The computer-readable storage device of claim 15, further comprising using at least one still frame from a given one of the matching videos in the compressed thumbnail for the given one of the matching videos.

20. The computer-readable storage device of claim 15, further comprising selecting the at least one still frame in response to occurrence of query terms inside an audio track of the given one of the videos, the query terms being terms in the query.

21. The computer-readable storage device of claim 15, wherein the end user selects one of the matching videos by either hovering, mousing over, scrolling, or clicking on one of the matching videos in the result page.

22. The computer-readable storage device of claim 15, wherein the result page is a web page viewed inside a browser.

23. The computer-readable storage device of claim 15, wherein activating streaming of the video summary comprises streaming the video summary to a client device, the client device being a wireless mobile device.

24. The computer-readable storage device of claim 15, further comprising presenting the result page by scrolling through the matching videos in the result page individually in full screen mode.

25. The computer-readable storage device of claim 15, further comprising optimizing the video summary to a display capability of a client device used to view the result page.

26. The computer-readable storage device of claim 15, further comprising constructing the video summary by combining multiple time sequences from the matching video.

27. A system, comprising:

a processor and a computer-readable storage device; and

a process configured to perform actions using the processor, comprising;

receiving a query from a client device;

applying the query to a set of videos containing at least one of;

video files, webinars, streaming video sources and live video broadcasts;

identifying matching videos in the set of videos based on at least one of;

textual metadata associated with the videos, and

structured metadata associated with the videos, the structured metadata including at least one of;

video size, video length, video quality or video format, text extracted from an audio track by speech-to-text analysis, or phonetic transcription extracted from the audio track;

producing a result page with at least some of the matching videos wherein each of the matching videos of the results page are different from one another and wherein the result page is a web page viewed inside a browser;

computing compressed thumbnails for the matching videos in the result page;

enabling a user selection of each of the matching videos in the result page, said user selecting the video by either hovering, mousing over, scrolling or clicking on the video in the result page;

constructing a video summary in response to each selection from the result page of one of the matching videos that are different from one another that provides to an end user an abbreviated review of the matching video selected by the end user, the video summary comprising a plurality of frames but not all of the frames from the matching video selected by the end user;

optimizing the video summary to the display capabilities of the client device; and

activating, by a server, streaming of the video summary to the client device such that the client device displays the video summary within the context of the result page in response to the user selection of one of the matching videos.

28. The system of claim 27, further comprising matching the query based on one of:

textual metadata associated with the video, structured metadata associated with the video, text extracted from an audio track by speech-to-text analysis, or phonetic transcription extracted from the audio track;

or combinations thereof, the structured metadata associated with the video including at least one of;

video size, video length, video quality or video format.

29. The system of claim 27, further comprising:

computing compressed thumbnails for the matching videos;

applying one or more of the computed compressed thumbnails to the result page;

and

optimizing the compressed thumbnails to display capabilities of a client device used to view the result page.

30. The system of claim 27, wherein activating streaming of the video summary comprises streaming the video summary to a client device, the client device being a wireless mobile device.

31. The system of claim 27, further comprising presenting the result page by scrolling through the matching videos in the result page individually in full screen mode.

32. The system of claim 27,wherein each video summary of the matching videos when selected from the result page is a complete review of the respective matching video.

33. A method of displaying video data within a result page, comprising:

receiving a query from a client device;

applying the query to a set of videos containing at least one of;

video files, webinars, streaming video sources and live video broadcasts;

identifying matching videos in the set of videos based on at least one of;

textual metadata associated with the videos, and structured metadata associated with the videos, the structured metadata including at least one of;

video size, video length, video quality or video format, text extracted from an audio track by speech-to-text analysis, or phonetic transcription extracted from the audio track;

producing a result page with at least some of the matching videos wherein each of the matching videos of the results page are different from one another and wherein the result page is a web page viewed inside a browser;

computing compressed thumbnails for the matching videos in the result page;

enabling a user selection of each of the matching videos in the result page, said user selecting the video by either hovering, mousing over, scrolling or clicking on the video in the result page;

constructing a video summary in response to each selection from the result page of one of the matching videos that are different from one another that provides to an end user an abbreviated review of the matching video selected by the end user, the video summary comprising a plurality of frames but not all of the frames from the matching video selected by the end user;

optimizing the video summary to the display capabilities of the client device; and

activating, by a server, streaming of the video summary to the client device such that the client device displays the video summary within the context of the result page in response to the user selection of one of the matching videos.