Methods and systems for aggregation and organization of multimedia data acquired from a plurality of sources

• US 10,277,946 B2
• Filed: 09/08/2017
• Issued: 04/30/2019
• Est. Priority Date: 10/10/2013
• Status: Active Grant

• Current Assignee: Trustees Of Columbia University In The City Of New York (Columbia University)
• Original Assignee: Trustees Of Columbia University In The City Of New York (Columbia University)
• Inventors: Chang, Shih-Fu, Jou, Brendan, Li, Hongzhi, Ellis, Joseph G., Morozoff-Abezgauz, Daniel, Karaman, Svebor
• Primary Examiner(s): Saltarelli, Dominic D
• Application Number:
  US15/699,446
• Publication Number:
  US 20180109843A1
• Time in Patent Office: 599 Days
• Field of Search: 725 19
• CPC Class Codes: G06F 16/783 using metadata automatically derived from the content
• CPC Class Codes: H04N 21/4332 by placing content in organized collections, e.g. local EPG data repository interfaces, Database management systems or updating for information retrieval G06F16/23; details of retrieval of video data and associated meta data in video database G06F16/739
• CPC Class Codes: H04N 21/4334 Recording operations recording of a television signal H04N5/76; arrangements for recording or accumulating broadcast information or broadcast-related information H04H60/27
• CPC Class Codes: H04N 21/4394 involving operations for analysing the audio stream, e.g. detecting features or characteristics in audio streams arrangements characterised by components specially adapted for monitoring, identification or recognition of audio in broadcast systems H04H60/58
• CPC Class Codes: H04N 21/44008 involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/458 Scheduling content for creating a personalised stream, e.g. by combining a locally stored advertisement with an incoming stream; Updating operations, e.g. for OS modules ; time-related management operations arrangements for replacing or switching information during the broadcast or during the distribution H04H20/10
• CPC Class Codes: H04N 21/4622 Retrieving content or additional data from different sources, e.g. from a broadcast channel and the Internet web site content organization and management for information retrieval from the Internet G06F16/958; transmission by internet of broadcast information H04H60/82; stock exchange data over packet-switching network H04L12/1804; push services including data channel over packet-switching network H04L12/1859
• CPC Class Codes: H04N 21/4821 using a grid, e.g. sorted out by channel and broadcast time
• CPC Class Codes: H04N 21/4826 using recommendation lists, e.g. of programs or channels sorted out according to their score
• CPC Class Codes: H04N 21/4884 for displaying subtitles
• CPC Class Codes: H04N 21/8126 involving additional data, e.g. news, sports, stocks, weather forecasts
• CPC Class Codes: H04N 21/858 Linking data to content, e.g. by linking an URL to a video object, by creating a hotspot

Abstract

A system for aggregation and organization of multimedia data acquired from a plurality of sources can include a data collection element, a multimedia segmentation element, a memory, an extraction element, and a user interface. The data collection element can be configured to receive multimedia data. The multimedia segmentation element can be coupled to the data collection element and can be configured to separate the multimedia data into a plurality of multimedia data segments. The memory can be configured to store hierarchical category information having a plurality of categories. The extraction element can be coupled to the multimedia segmentation element and the memory and can be configured to associate at least one of the plurality of categories with each of the plurality of multimedia data segments.

Claims

1. A system for linking one or more segments of multimedia data to social media, comprising:

a processor configured to extract hierarchical category information by;

identifying at least one text region in each of the one or more segments, if any, containing text and extracting the text from the at least one text region in each of the one or more segments, if any, or

identifying a speaker based on a detected face track, applying a detected name in on-screen text and applying that to the speaker, or applying a detected name in an audio transcript based on temporal proximity to the speaker in each of the one or more segments, if any;

a memory, coupled to the processor, and configured to store the hierarchical category information based on the extracted text or the identified speaker; and

the processor further configured for;

associating the extracted hierarchical category information with the one or more segments of multimedia data, and

linking the one or more associated segments of multimedia data to social media based on a social media identifier.

2. The system of claim 1, wherein the social media identifier is a hashtag.

3. The system of claim 1, wherein a representation of the social media identifier is constructed by aggregating content tagged with the social media identifier.

4. The system of claim 3, wherein a similarity metric is generated that compares the representation of the social media identifier to the one or more associated segments of multimedia data.

5. The system of claim 4, wherein when the similarity metric exceeds a predefined threshold, the social medial identifier is linked the one or more associated segments of multimedia data.

6. A method for linking one or more segments of multimedia data to social media, comprising:

extracting, by a processor, hierarchical category information by;

identifying at least one text region in each of the one or more segments, if any, containing text and extracting the text from the at least one text region in each of the one or more segments, if any, or

identifying a speaker based on a detected face track, applying a detected name in on-screen text and applying that to the speaker, or applying a detected name in an audio transcript based on temporal proximity to the speaker in each of the one or more segments, if any;

storing, by a memory, the hierarchical category information based on the extracted text or the identified speaker;

associating, by the processor, the extracted hierarchical category information with the one or more segments of multimedia data; and

linking the one or more associated segments of multimedia data to social media based on a social media identifier.

7. The method of claim 6, wherein the social media identifier is a hashtag.

8. The method of claim 6, wherein a representation of the social media identifier is constructed by aggregating content tagged with the social media identifier.

9. The method of claim 8, wherein a similarity metric is generated that compares the representation of the social media identifier to the one or more associated segments of multimedia data.

10. The method of claim 9, wherein when the similarity metric exceeds a predefined threshold, the social medial identifier is linked the one or more associated segments of multimedia data.