Method for content-based non-linear control of multimedia playback

• US 10,555,043 B2
• Filed: 05/23/2019
• Issued: 02/04/2020
• Est. Priority Date: 05/14/2001
• Status: Expired due to Fees

• Current Assignee: AT&T Corporation (AT&T, Inc.)
• Original Assignee: AT&T Intellectual Property I LP (AT&T, Inc.)
• Inventors: Chen, Edward Y., Gibbon, David Crawford, Ruedisueli, Laurence W., Shahraray, Behzad
• Primary Examiner(s): Huynh, An Son P
• Application Number:
  US16/420,315
• Publication Number:
  US 20190281358A1
• Time in Patent Office: 257 Days
• Field of Search: None
• CPC Class Codes: G06F 16/40 of multimedia data, e.g. slideshows comprising image and additional audio data retrieval of still image data G06F16/50; retrieval of audio data G06F16/60; retrieval of video data G06F16/70
• CPC Class Codes: G06F 16/44 Browsing; Visualisation therefor
• CPC Class Codes: G06F 16/48 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/73 Querying
• CPC Class Codes: G06F 16/732 Query formulation
• CPC Class Codes: G06F 16/738 Presentation of query results
• CPC Class Codes: G06F 16/74 Browsing; Visualisation therefor end-user interfaces for requesting or interacting with video content, e.g. video on demand interfaces or electronic program guides, H04N21/472
• CPC Class Codes: H04N 21/23439 for generating different versions
• CPC Class Codes: H04N 21/26603 for automatically generating descriptors from content, e.g. when it is not made available by its provider, using content analysis techniques
• CPC Class Codes: H04N 21/278 Content descriptor database or directory service for end-user access details of content or meta data based information retrieval of video data in video databases G06F16/739
• CPC Class Codes: H04N 21/41407 embedded in a portable device, e.g. video client on a mobile phone, PDA, laptop constructional details of equipment or arrangements specially adapted for portable computer application G06F1/1626; arrangements specially adapted for mobile receivers in broadcast systems H04H20/57
• CPC Class Codes: H04N 21/426 Internal components of the client ; Characteristics thereof H04N5/44 takes precedence
• CPC Class Codes: H04N 21/432 Content retrieval operation from a local storage medium, e.g. hard-disk details of retrieval of video data and associated meta data in video databases G06F16/739
• CPC Class Codes: H04N 21/4345 Extraction or processing of SI, e.g. extracting service information from an MPEG stream
• CPC Class Codes: H04N 21/44008 involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/4622 Retrieving content or additional data from different sources, e.g. from a broadcast channel and the Internet web site content organization and management for information retrieval from the Internet G06F16/958; transmission by internet of broadcast information H04H60/82; stock exchange data over packet-switching network H04L12/1804; push services including data channel over packet-switching network H04L12/1859
• CPC Class Codes: H04N 21/47202 for requesting content on demand, e.g. video on demand
• CPC Class Codes: H04N 21/47217 for controlling playback functions for recorded or on-demand content, e.g. using progress bars, mode or play-point indicators or bookmarks specific graphical features in visual interfaces H04N21/4312
• CPC Class Codes: H04N 21/4782 Web browsing , e.g. WebTV
• CPC Class Codes: H04N 21/4821 using a grid, e.g. sorted out by channel and broadcast time
• CPC Class Codes: H04N 21/4828 : for searching program descriptors retrieval of video data G06F16/739
• CPC Class Codes: H04N 21/6587 : Control parameters, e.g. trick play commands, viewpoint selection
• CPC Class Codes: H04N 21/84 : Generation or processing of descriptive data, e.g. content descriptors systems specially adapted for using meta-information in broadcast systems H04H60/73
• CPC Class Codes: H04N 21/8586 : by using a URL processing chained hypermedia data for information retrieval G06F16/94; information retrieval from the Internet by using URLs G06F16/955; URL in broadcast information H04H20/93; Web-based protocols H04L67/02
• CPC Class Codes: H04N 7/17336 : Handling of requests in head-ends

Abstract

A system and method are provided for content-based non-linear control of video data playback. A multimedia database having multimedia data including multimedia content data is searched based on a user query to determine a first set of multimedia data. The multimedia data includes indexes to and condensed representations of corresponding video data stored in a video database. A portion of the first set of multimedia data is displayed at a control device in response to the user query. A user of the control device selects an element of the first set of multimedia data for video playback and video data corresponding to the element delivered to a video device for playback. A user of the control device selects an element of the first set of multimedia data for additional information and a second set of multimedia data corresponding to the element delivered to the control device.

Claims

1. A method comprising:

searching by a server, in response to receiving a search term from a user via a user interface of a mobile control device, a plurality of multimedia data stored in a database at a multimedia server to generate matched multimedia data associated with the search term, the matched multimedia data comprising video content and one or more of a condensed representation pertaining to the video content, offline transcriptions, and closed captions associated with the video content, wherein the condensed representation, the offline transcriptions, and the closed captions associated with the video content are stored in the database, wherein the matched multimedia data is generated by;

analyzing, by a media metadata extraction device, the video content and identifying video shot boundaries in the video content;

extracting, by the media metadata extraction device, from the video content, (1) text from the closed captions and (2) a frame corresponding to the text; and

extracting, by the media metadata extraction device, small segments of organized multimedia data based on topic boundaries and the video shot boundaries, each small segment of the small segments being smaller than a respective program containing the small segments, to yield the matched multimedia data;

displaying, with the matched multimedia data and via the user interface, a plurality of command icons for controlling which display device of a plurality of external devices to display a selected video content;

receiving, via the user interface of the mobile control device and from the user, a choice from the matched multimedia data to yield the selected video content; and

receiving, via the user interface of the mobile control device, a user input to display the selected video content on one of a first display device and a second display device of the plurality of external devices, wherein the plurality of command icons remains on the user interface of the mobile control device after one of the first display device and the second display device has been selected for control so that the user can assert control commands regardless of wherein the user is in a navigation session.

2. The method of claim 1, wherein the user interface comprises a graphical user interface.

3. The method of claim 1, wherein the plurality of external devices comprises the multimedia server, a video server, and a plurality of video devices.

4. The method of claim 1, wherein the selected video content is indexed by one of a title, a length, a rating, and a scene.

5. The method of claim 1, wherein the search term is received as an utterance.

6. The method of claim 1, wherein the matched multimedia data comprises a second condensed representation pertaining to the selected video content, an offline transcription, and closed captions associated with the selected video content.

7. The method of claim 1, wherein the database comprises ancillary data used to index the selected video content.

8. The method of claim 7, wherein the ancillary data comprises one of offline transcriptions, manual annotations, topic classifications, post-production scripts, metadata, and genre classifications.

9. A system comprising:

a processor; and

a computer-readable storage medium having instructions stored which, when executed by the processor, cause the processor to perform operations comprising;

searching, in response to receiving a search term from a user via a user interface of the system, a plurality of multimedia data stored in a database at a multimedia server to generate matched multimedia data associated with the search term, the matched multimedia data comprising video content and one or more of a condensed representation pertaining to the video content, offline transcriptions, and closed captions associated with the video content, wherein the condensed representation, the offline transcriptions, and the closed captions associated with the video content are stored in the database, wherein the matched multimedia data is generated by;

analyzing, by a media metadata extraction device, the video content and identifying video shot boundaries in the video content;

extracting, by the media metadata extraction device, from the video content, (1) text from the closed captions and (2) a frame corresponding to the text; and

extracting, by the media metadata extraction device, small segments of organized multimedia data based on topic boundaries and the video shot boundaries, each small segment of the small segments being smaller than a respective program containing the small segments, to yield the matched multimedia data;

displaying, with the matched multimedia data and via the user interface, a plurality of command icons for controlling which display device of a plurality of external devices to display a selected video content;

receiving, via the user interface of the system and from the user, a choice from the matched multimedia data to yield the selected video content; and

receiving, via the user interface of the system, a user input to display the selected video content on one of a first display device and a second display device of the plurality of external devices, wherein the plurality of command icons remains on the user interface of the system after one of the first display device and the second display device has been selected for control so that the user can assert control commands regardless of wherein the user is in a navigation session.

10. The system of claim 9, wherein the user interface comprises a graphical user interface.

11. The system of claim 9, wherein the plurality of external devices comprises the multimedia server, a video server, and a plurality of video devices.

12. The system of claim 9, wherein the selected video content is indexed by one of a title, a length, a rating, and a scene.

13. The system of claim 9, wherein the search term is received as an utterance.

14. The system of claim 9, wherein the matched multimedia data comprises a second condensed representation pertaining to the selected video content, an offline transcription, and closed captions associated with the selected video content.

15. The system of claim 9, wherein the database comprises ancillary data used to index the selected video content.

16. The system of claim 15, wherein the ancillary data comprises one of offline transcriptions, manual annotations, topic classifications, post-production scripts, metadata, and genre classifications.

17. A computer-readable storage device having instructions stored which, when executed by a system, cause the system to perform operations comprising:

searching, in response to receiving a search term from a user via a user interface of the system, a plurality of multimedia data stored in a database at a multimedia server to generate matched multimedia data associated with the search term, the matched multimedia data comprising video content and one or more of a condensed representation pertaining to the video content, offline transcriptions, and closed captions associated with the video content, wherein the condensed representation, the offline transcriptions, and the closed captions associated with the video content are stored in the database, wherein the matched multimedia data is generated by;

analyzing, by a media metadata extraction device, the video content and identifying video shot boundaries in the video content;

extracting, by the media metadata extraction device, from the video content, (1) text from the closed captions and (2) a frame corresponding to the text; and

extracting, by the media metadata extraction device, small segments of organized multimedia data based on topic boundaries and the video shot boundaries, each small segment of the small segments being smaller than a respective program containing the small segments, to yield the matched multimedia data;

displaying, with the matched multimedia data and via the user interface, a plurality of command icons for controlling which display device of a plurality of external devices to display a selected video content;

receiving, via the user interface of the system and from the user, a choice from the matched multimedia data to yield the selected video content; and

receiving, via the user interface of the system, a user input to display the selected video content on one of a first display device and a second display device of the plurality of external devices, wherein the plurality of command icons remains on the user interface of the system after one of the first display device and the second display device has been selected for control so that the user can assert control commands regardless of wherein the user is in a navigation session.

18. The computer-readable storage device of claim 17, wherein the plurality of external devices comprises the multimedia server, a video server, and a plurality of video devices.

19. The computer-readable storage device of claim 17, wherein the selected video content is indexed by one of a title, a length, a rating, and a scene.

20. The computer-readable storage device of claim 17, wherein the search term is received as an utterance.