Browsing and retrieval of full broadcast-quality video

• US 9,171,545 B2
• Filed: 11/30/2010
• Issued: 10/27/2015
• Est. Priority Date: 04/19/1999
• Status: Expired due to Fees

• Current Assignee: AT&T Corporation (AT&T, Inc.)
• Original Assignee: AT&T Intellectual Property II LP (AT&T, Inc.)
• Inventors: Basso, Andrea, Civanlar, Mehmet Reha, Gibbon, David Crawford, Huang, Qian, Levin, Esther, Pieraccini, Roberto, Shahraray, Behzad
• Primary Examiner(s): ROBERTS, SHAUN A
• Application Number:
  US12/956,147
• Publication Number:
  US 20110072466A1
• Time in Patent Office: 1,792 Days
• Field of Search: 725/147, 725/139, 704/231, 704/275
• US Class Current: 1/1
• CPC Class Codes: G06F 16/71 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/73 Querying
• CPC Class Codes: G06F 16/739 in form of a video summary, e.g. the video summary being a video sequence, a composite still image or having synthesized frames
• CPC Class Codes: G06F 16/7844 using original textual content or text extracted from visual content or transcript of audio data
• CPC Class Codes: G06F 40/58 Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G11B 27/10 Indexing; Addressing; Timing or synchronising; Measuring tape travel
• CPC Class Codes: G11B 27/105 of operating discs
• CPC Class Codes: G11B 27/28 by using information signals recorded by the same method as the main recording G11B27/22 takes precedence
• CPC Class Codes: G11B 27/34 Indicating arrangements indicating means incorporated in magazine or cassette G11B23/046 and G11B23/0875; indicating measured values in general G01D
• CPC Class Codes: H04N 21/232 Content retrieval operation locally within server, e.g. reading video streams from disk arrays storage management G06F3/0604; details of querying and searching of video data from a database G06F16/739
• CPC Class Codes: H04N 21/233 Processing of audio elementary streams monitoring, identification or recognition of audio in broadcast systems H04H60/58
• CPC Class Codes: H04N 21/235 Processing of additional data, e.g. scrambling of additional data or processing content descriptors
• CPC Class Codes: H04N 21/25825 involving client display capabilities, e.g. screen resolution of a mobile phone optimising the visualisation of content during browsing in the Internet G06F16/9577; processing of terminal status or physical abilities in wireless networks H04W8/22; authentication in wireless network security H04W12/06
• CPC Class Codes: H04N 21/2662 Controlling the complexity of the video stream, e.g. by scaling the resolution or bitrate of the video stream based on the client capabilities
• CPC Class Codes: H04N 21/431 Generation of visual interfaces for content selection or interaction; Content or additional data rendering
• CPC Class Codes: H04N 21/4316 for displaying supplemental content in a region of the screen, e.g. an advertisement in a separate window
• CPC Class Codes: H04N 21/47 End-user applications
• CPC Class Codes: H04N 21/4828 for searching program descriptors retrieval of video data G06F16/739
• CPC Class Codes: H04N 21/4884 for displaying subtitles
• CPC Class Codes: H04N 21/812 : involving advertisement data advertising per se G06Q30/02
• CPC Class Codes: H04N 21/84 : Generation or processing of descriptive data, e.g. content descriptors systems specially adapted for using meta-information in broadcast systems H04H60/73
• CPC Class Codes: H04N 21/8456 : by decomposing the content in the time domain, e.g. in time segments
• CPC Class Codes: H04N 21/8547 : involving timestamps for synchronizing content
• CPC Class Codes: H04N 5/45 : Picture in picture , e.g. displaying simultaneously another television channel in a region of the screen
• CPC Class Codes: H04N 5/765 : Interface circuits between an apparatus for recording and another apparatus

Abstract

A method includes steps of indexing a media collection, searching an indexed library and browsing a set of candidate program segments. The step of indexing a media collection creates the indexed library based on a content of the media collection. The step of searching the indexed library identifies the set of candidate program segments based on a search criteria. The step of browsing the set of candidate program segments selects a segment for viewing.

Claims

1. A method comprising:

classifying a media collection as program content and commercials, to yield classified program content, wherein the commercials are to be presented chronologically separate from the program content and in a same display screen as the program content;

recognizing speech in the program content, to yield recognized speech comprising recognized speech timestamps;

synchronizing the recognized speech with closed captioning data by aligning the recognized speech timestamps and closed captioning timestamps associated with the closed captioning data, to yield synchronized recognized speech;

identifying program segments within the program content based on the synchronized recognized speech in the program content;

indexing the media collection based on the program segments to create an indexed library, the indexed library comprising an inverted index of the closed captioning data;

receiving a search query to the indexed library from a user;

modifying a semantic representation format of the search query to meet a requirement of the indexed library, to yield a modified search query;

searching the indexed library and a relational database to identify a set of candidate program segments based on the modified search query, wherein the relational database comprises;

entities comprising a network, a show title, a broadcast date, and a broadcast time of the program content; and

a data model specific to the entities within the relational database, wherein the data model comprises attributes belonging to each entity in the entities and relationships between the entities; and

transmitting a low-resolution output of the set of candidate program segments for the user to browse and select for high-resolution playback via a user interface.

2. The method of claim 1, wherein indexing the media collection further comprises:

forming a browseable image for each segment of the set of candidate program segments, each browseable image comprising keywords identified in one of the recognized speech and the closed captioning data for display in the browseable image.

3. The method of claim 2, wherein transmitting the low-resolution output further comprises:

selecting a display segment from the set of candidate program segments; and

transmitting the browseable image with associated keywords.

4. The method of claim 3, wherein each browseable image further comprises key images identified in the indexed library for display in the browseable image, and wherein displaying the browseable image further comprises displaying associated key images.

5. The method of claim 4, wherein the one of the recognized speech and the closed captioning data associated with the display segment comprises a first word having low information content and a second word having high information content, and wherein forming the browseable image further comprises selecting the second word as a keyword and rejecting the first word as the keyword.

6. The method of claim 1, wherein indexing further comprises:

forming a browseable image for each segment of the program segments, each browseable image comprising key images identified in the indexed library for display in the browseable image;

selecting a display segment from the program segments; and

displaying each associated browseable image.

7. The method of claim 6, wherein media associated with the display segment comprises an image of an anchor-person having low information content and a field shot image of an event having high information content, and wherein forming the browseable image further comprises:

selecting the field shot image as a key image; and

rejecting the image of the anchor-person as the key image.

8. A system comprising:

a processor; and

a computer-readable storage device having instructions stored which, when executed by the processor, cause the processor to perform operations comprising;

classifying a media collection as program content and commercials to yield classified program content, wherein the commercials are to be presented chronologically separate from the program content and in a same display screen as the program content;

recognizing speech in the program content, to yield recognized speech comprising recognized speech timestamps;

synchronizing the recognized speech with closed captioning data by aligning the recognized speech timestamps and closed captioning timestamps associated with the closed captioning data, to yield synchronized recognized speech;

identifying program segments within the program content based on the synchronized recognized speech in the program content;

indexing the media collection based on the program segments to create an indexed library, the indexed library comprising an inverted index of the closed captioning data;

receiving a search query to the indexed library from a user;

modifying a semantic representation format of the search query to meet a requirement of the indexed library, to yield a modified search query;

searching the indexed library and a relational database to identify a set of candidate program segments based on the modified search query, wherein the relational database comprises;

entities comprising a network, a show title, a broadcast date, and a broadcast time of the program content; and

a data model specific to the entities within the relational database, wherein the data model comprises attributes belonging to each entity in the entities and relationships between the entities; and

transmitting a low-resolution output of the set of candidate program segments for the user to browse and select for high-resolution playback via a user interface.

9. The system of claim 8, wherein indexing the media collection further comprises:

forming a browseable image for each segment of the set of candidate program segments, each browseable image including keywords identified in one of the recognized speech and the closed captioning data for display in the browseable image.

10. The system of claim 9, wherein transmitting the low-resolution output further comprises:

selecting a display segment from the set of candidate program segments; and

transmitting the browseable image with associated keywords.

11. The system of claim 10, wherein each browseable image further comprises key images identified in the indexed library for display in the browseable image, and wherein displaying the browseable image further comprises displaying associated key images.

12. The system of claim 11, wherein the one of the recognized speech and the closed captioning data associated with the display segment comprises a first word having low information content and a second word having high information content, and wherein forming the browseable image further comprises selecting the second word as a keyword and rejecting the first word as the keyword.

13. The system of claim 8, wherein indexing further comprises:

forming a browseable image for each segment of the program segments, each browseable image comprising key images identified in the indexed library for display in the browseable image;

selecting a display segment from the program segments; and

displaying each associated browseable image.

14. The system of claim 13, wherein the media associated with the display segment comprises an image of an anchor-person having low information content and a field shot image of an event having high information content, and wherein forming the browseable image further comprises:

selecting the field shot image as a key image; and

rejecting the image of the anchor-person as the key image.

15. A computer-readable storage device having instructions stored which, when executed by a computing device, cause the computing device to perform operations comprising:

classifying a media collection as program content and commercials to yield classified program content, wherein the commercials are to be presented chronologically separate from the program content and in a same display screen as the program content;

recognizing speech in the program content, to yield recognized speech comprising recognized speech timestamps;

synchronizing the recognized speech with closed captioning data by aligning the recognized speech timestamps and closed captioning timestamps associated with the closed captioning data, to yield synchronized recognized speech;

identifying program segments within the program content based on the synchronized recognized speech in the program content;

indexing the media collection based on the program segments to create an indexed library, the indexed library comprising an inverted index of the closed captioning data;

receiving a search query to the indexed library from a user;

modifying a semantic representation format of the search query to meet a requirement of the indexed library, to yield a modified search query;

searching the indexed library and a relational database to identify a set of candidate program segments based on the modified search query, wherein the relational database comprises;

entities comprising a network, a show title, a broadcast date, and a broadcast time of the program content; and

a data model specific to the entities within the relational database, wherein the data model comprises attributes belonging to each entity in the entities and relationships between the entities; and

transmitting a low-resolution output of the set of candidate program segments for the user to browse and select for high-resolution playback via a user interface.

16. The computer-readable storage device of claim 15, wherein indexing the media collection further comprises:

forming a browseable image for each segment of the set of candidate program segments, each browseable image comprising keywords identified in one of the recognized speech and the closed captioning data for display in the browseable image.

17. The computer-readable storage device of claim 16, wherein transmitting the low-resolution output further comprises:

selecting a display segment from the set of candidate program segments; and

transmitting the browseable image with associated keywords.

18. The computer-readable storage device of claim 17, wherein each browseable image further comprises key images identified in the indexed library for display in the browseable image, and wherein displaying the browseable image further comprises displaying associated key images.

19. The computer-readable storage device of claim 18, wherein the one of the recognized speech and the closed captioning data associated with the display segment comprises a first word having low information content and a second word having high information content, and wherein forming the browseable image further comprises selecting the second word as a keyword and rejecting the first word as the keyword.

20. The computer-readable storage device of claim 15, wherein indexing further comprises:

forming a browseable image for each segment of the program segments, each browseable image comprising key images identified in the indexed library for display in the browseable image;

selecting a display segment from the program segments; and

displaying each associated browseable image.