Method for content-based non-linear control of multimedia playback

• US 8,479,238 B2
• Filed: 05/14/2002
• Issued: 07/02/2013
• Est. Priority Date: 05/14/2001
• Status: Active Grant

• Current Assignee: AT&T Intellectual Property II LP (AT&T, Inc.)
• Original Assignee: AT&T Intellectual Property II LP (AT&T, Inc.)
• Inventors: Chen, Edward Y., Gibbon, David Crawford, Ruedisueli, Laurence W., Shahraray, Behzad
• Primary Examiner(s): HUYNH, AN SON PHI
• Application Number:
  US10/146,300
• Publication Number:
  US 20020170062A1
• Time in Patent Office: 4,067 Days
• Field of Search: 725 39- 41, 725 52- 53, 725 86- 88, 725/91, 725102-104, 715723-725, 715/835, 715/838
• US Class Current: 725/86
• CPC Class Codes: G06F 16/40 of multimedia data, e.g. slideshows comprising image and additional audio data retrieval of still image data G06F16/50; retrieval of audio data G06F16/60; retrieval of video data G06F16/70
• CPC Class Codes: G06F 16/44 Browsing; Visualisation therefor
• CPC Class Codes: G06F 16/48 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/73 Querying
• CPC Class Codes: G06F 16/732 Query formulation
• CPC Class Codes: G06F 16/738 Presentation of query results
• CPC Class Codes: G06F 16/74 Browsing; Visualisation therefor end-user interfaces for requesting or interacting with video content, e.g. video on demand interfaces or electronic program guides, H04N21/472
• CPC Class Codes: H04N 21/23439 for generating different versions
• CPC Class Codes: H04N 21/26603 for automatically generating descriptors from content, e.g. when it is not made available by its provider, using content analysis techniques
• CPC Class Codes: H04N 21/278 Content descriptor database or directory service for end-user access details of content or meta data based information retrieval of video data in video databases G06F16/739
• CPC Class Codes: H04N 21/41407 embedded in a portable device, e.g. video client on a mobile phone, PDA, laptop constructional details of equipment or arrangements specially adapted for portable computer application G06F1/1626; arrangements specially adapted for mobile receivers in broadcast systems H04H20/57
• CPC Class Codes: H04N 21/426 Internal components of the client ; Characteristics thereof H04N5/44 takes precedence
• CPC Class Codes: H04N 21/432 Content retrieval operation from a local storage medium, e.g. hard-disk details of retrieval of video data and associated meta data in video databases G06F16/739
• CPC Class Codes: H04N 21/4345 Extraction or processing of SI, e.g. extracting service information from an MPEG stream
• CPC Class Codes: H04N 21/44008 involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/4622 Retrieving content or additional data from different sources, e.g. from a broadcast channel and the Internet web site content organization and management for information retrieval from the Internet G06F16/958; transmission by internet of broadcast information H04H60/82; stock exchange data over packet-switching network H04L12/1804; push services including data channel over packet-switching network H04L12/1859
• CPC Class Codes: H04N 21/47202 for requesting content on demand, e.g. video on demand
• CPC Class Codes: H04N 21/47217 for controlling playback functions for recorded or on-demand content, e.g. using progress bars, mode or play-point indicators or bookmarks specific graphical features in visual interfaces H04N21/4312
• CPC Class Codes: H04N 21/4782 Web browsing , e.g. WebTV
• CPC Class Codes: H04N 21/4821 using a grid, e.g. sorted out by channel and broadcast time
• CPC Class Codes: H04N 21/4828 : for searching program descriptors retrieval of video data G06F16/739
• CPC Class Codes: H04N 21/6587 : Control parameters, e.g. trick play commands, viewpoint selection
• CPC Class Codes: H04N 21/84 : Generation or processing of descriptive data, e.g. content descriptors systems specially adapted for using meta-information in broadcast systems H04H60/73
• CPC Class Codes: H04N 21/8586 : by using a URL processing chained hypermedia data for information retrieval G06F16/94; information retrieval from the Internet by using URLs G06F16/955; URL in broadcast information H04H20/93; Web-based protocols H04L67/02
• CPC Class Codes: H04N 7/17336 : Handling of requests in head-ends

Abstract

A system and method are provided for content-based non-linear control of video data playback. A multimedia database having multimedia data including multimedia content data is searched based on a user query to determine a first set of multimedia data. The multimedia data includes indexes to and condensed representations of corresponding video data stored in a video database. A portion of the first set of multimedia data is displayed at a control device in response to the user query. A user of the control device selects an element of the first set of multimedia data for video playback and video data corresponding to the element delivered to a video device for playback. A user of the control device selects an element of the first set of multimedia data for additional information and a second set of multimedia data corresponding to the element delivered to the control device.

Claims

1. A mobile system comprising:

a processor;

a graphic display;

a display controller configured to control the graphic display;

a network interface operable for communicating with a plurality of external devices separately coupled to a communications network, the plurality of external devices comprising;

(1) a video server, the video server populated with a plurality of versions of video data of a plurality of programs, each version compressed according to a respective digital video compression format;

(2) a multimedia server, the multimedia server populated with multimedia content data corresponding to the video data, the multimedia content data comprising a condensed representation of the video data, closed captions associated with the video data, and offline transcriptions, the multimedia content data generated by a metadata extraction device, wherein the metadata extraction device performs;

analyzing the video data and identifying video shot boundaries in the video data without user intervention;

extracting content data from the video data and the closed captions including at least one text extract from the closed captions and a frame from the video data comprising a multimedia image scene corresponding to the at least one text extract;

extracting small segments of organized multimedia content based on topic boundaries according to topics within the plurality of programs and the video shot boundaries, each small segment being smaller than a respective program containing the small segments;

extracting the offline transcriptions; and

indexing video data based on keywords referenced in the video data; and

(3) a plurality of video devices, each of the plurality of video devices capable of displaying the video data; and

a non-transitory computer-readable storage medium having stored therein instructions which, when executed by the processor, cause the processor to perform a method comprising;

receiving a user request to display a list of the plurality of video devices;

receiving a first user-specification identifying at least one of the plurality of video devices from the list for displaying of video data to yield an identified video device;

responsive to receiving the first user-specification, displaying, at a to of the graphic display, a plurality of command icons for controlling the identified video device;

receiving a second user-specification of a query with which to search the multimedia content data for determination of a first set of multimedia content data corresponding to the query, the first set of multimedia content data representing a group of programs;

outputting, on the graphic display, the first set of multimedia content data;

receiving a third user-specification of a specified program of the first set of multimedia content data, the third user-specification of the specified program commanding the multimedia server to deliver to the display controller a second listing of a group of visually non-linear extracted small video segments from the specified program, the second listing being selected based on at least one of the closed captions and the offline transcripts; and

receiving a fourth user-specification of one non-linear segment of the visually non-linear extracted small segments from the second listing, wherein the fourth user-specification commands the video server to deliver to the identified video device the specified visually non-linear extracted small video segment for playback at a point corresponding to the one non-linear segment, wherein the plurality of command icons remains at the to of the graphic display regardless of receiving the second user-specification, the third user-specification and the fourth user-specification.

2. The mobile system of claim 1, wherein the multimedia content data comprises any combination of data pertaining to the content of the video data, content specific index, still image data, preview video data, biographical data, audio data, video-audio data, speech data, and text data.

3. The mobile system of claim 1, wherein the multimedia content data further comprises ancillary data.

4. The mobile system of claim 3, wherein the ancillary data comprises any combination of offline transcriptions, automatic speech transcriptions, manual annotations, topic classifications, post-production scripts, metadata, and genre classifications.

5. The mobile system of claim 1, wherein the video server and the multimedia server are a same server.

6. The mobile system of claim 1, wherein the mobile system further comprises at least one of a personal digital assistant, a palm-top computer, a tablet computer, a lap-top computer, and a device capable of displaying the multimedia data and specification of input.

7. The mobile system of claim 1, wherein the network interface utilizes wireless communication with at least one of the video server, the multimedia server, and the plurality of video devices.

8. The mobile system of claim 1, wherein the non-transitory computer-readable storage media further having stored therein instructions which result in the method further comprising controlling the identified video device.

9. The mobile system of claim 1, wherein the third user-specification of the specified program and resulting delivery to the video device of video data corresponding to the specified program is performed in a first state, and wherein the network interface is further operative to, in a second state, upon the user-selection of a second specified program, display a second set of multimedia content data corresponding the second specified program.

10. A method comprising:

receiving, by a mobile control device, user request to display a list of a plurality of video devices;

selecting, from the mobile control device, at least one of the plurality of video devices from the list for display of video data to yield an identified video device, the mobile control device operable to communicate over a communications network with a plurality of external devices, wherein the plurality of external devices comprising a video server, a multimedia server, and the plurality of video devices;

responsive to receiving a selection of at least one of the plurality of video devices from the list, displaying, at a top of a graphic display of the mobile control device, a plurality of command icons for controlling the identified video device;

searching multimedia data stored at the multimedia server operating in the communications network based on a query to determine a first set of multimedia content data comprising a condensed representation of the video data, offline transcriptions, and closed captions associated with the video data, wherein;

the video data is stored on a video server operating in the communications network; and

the multimedia content data is generated by a media analysis/metadata extraction device, and wherein the media analysis/metadata extraction device performs;

(1) analyzing the video data and identifying video shot boundaries in the video data without user intervention;

(2) extracting the multimedia content data from the video data and the closed captions, the multimedia content data including at least one text extract from the closed captions and a frame from the video data comprising a multimedia image scene corresponding to the at least one text extract;

(3) extracting small segments of at least one of organized multimedia content based on the video shot boundaries and topic and story boundaries according to topics within a plurality of programs, each small segment being smaller than a respective program containing the small segments, the video data comprising a plurality of versions of video data of the plurality of programs, each version compressed according to a respective digital video compression format;

(4) extracting the offline transcriptions; and

(5) indexing video data based on keywords referenced in the video data;

displaying at least a portion of the first set of multimedia content data at the mobile control device, the portion comprising at least one image scene, the first set representing a set of small visually non-linear segments within a program and selected based at least in part on at least one of the closed captions and the offline transcriptions;

receiving a selection from the mobile control device of a small segment from the first set of small visually non-linear segments of the first set for video playback to yield a selected small segment; and

delivering to the identified video device the selected small segment from the program in the video at a point corresponding to the small segment in the program,

wherein the plurality of command icons remains at the top of the graphic display of the mobile control device after the identified video device has been selected for control regardless of where a user is in a navigation session.

11. The method of claim 10, wherein the multimedia content data comprises any combination of data pertaining to the content of the video data, content specific index, still image data, preview video data, biographical data, audio data, video-audio data, speech data, and text data.

12. The method of claim 10, wherein the multimedia content data further comprises ancillary data.

13. The method of claim 10, wherein the ancillary data comprises any combination of offline transcriptions, manual annotations, topic classifications, post-production scripts, metadata, and genre classifications.

14. The method of claim 10, wherein the video data and the multimedia content data are stored on a same server.

15. The method of claim 10, wherein the mobile control device is associated with a plurality of video devices, the method further comprising the step of selecting at least one of the plurality of video devices for the delivery of video data.

16. The method of claim 10, wherein the mobile control device comprises at least one of a personal digital assistant, a palm-top computer, a tablet computer, a lap-top computer, and a device capable of displaying the multimedia content data and specification of input.

17. The method of claim 10, wherein the mobile control device utilizes wireless communication.

18. The method of claim 10, further comprising:

populating the multimedia server with the multimedia content data; and

populating the video server with the video data corresponding to the multimedia content data.

19. The method of claim 18, wherein populating the multimedia server comprises:

extracting the multimedia content data from video source material using at least one of automatic media processing techniques, video indexing, and audio indexing.

20. The method of claim 18, wherein populating the video server comprises:

digitizing and compressing video source material.

21. A non-transitory computer-readable storage medium storing instructions, which, when executed by a processor, perform a method comprising:

receiving, by a mobile control device, user request to display a list of a plurality of video devices;

selecting, from the mobile control device, at least one of the plurality of video devices from the list for display of video data to yield an identified video device, the mobile control device operable to communicate over a communications network with a plurality of external devices, wherein the plurality of external devices comprising a video server, a multimedia server, and the plurality of video devices;

responsive to receiving a selection of at least one of the plurality of video devices from the list, displaying, at a top of a graphic display of the mobile control device, a plurality of command icons for controlling the identified video device;

searching multimedia data stored at the multimedia server operating in the communications network based on a query to determine a first set of multimedia content data comprising a condensed representation of the video data, offline transcriptions, and closed captions associated with the video data, wherein;

the video data is stored on a video server operating in the communications network; and

the multimedia content data is generated by a media analysis/metadata extraction device, and wherein the media analysis/metadata extraction device performs;

(1) analyzing the video data and identifying video shot boundaries in the video data without user intervention;

(2) extracting the multimedia content data from the video data and the closed captions, the multimedia content data including at least one text extract from the closed captions and a frame from the video data comprising a multimedia image scene corresponding to the at least one text extract;

(3) extracting small segments of at least one of organized multimedia content based on the video shot boundaries and topic and story boundaries according to topics within a plurality of programs, each small segment being smaller than a respective program containing the small segments, the video data comprising a plurality of versions of video data of the plurality of programs, each version compressed according to a respective digital video compression format;

(4) extracting the offline transcriptions; and

(5) indexing video data based on keywords referenced in the video data;

displaying at least a portion of the first set of multimedia content data at the mobile control device, the portion comprising at least one image scene, the first set representing a set of small visually non-linear segments within a program and selected based at least in part on at least one of the closed captions and the offline transcriptions;

receiving a selection from the mobile control device of a small segment from the first set of small visually non-linear segments of the first set for video playback to yield a selected small segment; and

delivering to the identified video device the selected small segment from the program in the video at a point corresponding to the small segment in the program,

wherein the plurality of command icons remains at the top of the graphic display of the mobile control device after the identified video device has been selected for control regardless of where a user is in a navigation session.