Method and apparatus for semantic extraction and video remix creation

• US 9,436,875 B2
• Filed: 12/06/2012
• Issued: 09/06/2016
• Est. Priority Date: 12/06/2012
• Status: Active Grant

• Current Assignee: Nokia Technologies Oy (Nokia Corporation)
• Original Assignee: Nokia Technologies Oy (Nokia Corporation)
• Inventors: Curcio, Igor Danilo Diego, Mate, Sujeet Shyamsundar, Cricri, Francesco, Roininen, Mikko Joonas, Sathish, Sailesh
• Primary Examiner(s): Chowdhury, Nigar
• Application Number:
  US13/707,090
• Publication Number:
  US 20140161354A1
• Time in Patent Office: 1,370 Days
• Field of Search: 386223-228, 386/248, 386/278
• US Class Current: 1/1
• CPC Class Codes: G06F 18/253 of extracted features
• CPC Class Codes: G06V 10/806 of extracted features
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G06V 20/46 Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames

Abstract

A method, apparatus and computer program product are provided for extracting semantic information from user-generated media content to create a video remix which is semantically enriched. An exemplary method comprises extracting media content data and sensor data from a plurality of media content, wherein the sensor data comprises a plurality of data modalities. The method may also include classifying the extracted media content data and the sensor data. The method may further include detecting predefined objects or events utilizing the sensor data to create remix video.

Claims

1. A method comprising:

classifying recorded data according to an event type, wherein the recorded data comprises a plurality of modalities;

determining an event template configuration, the event template configuration comprises one or more selected event templates,

wherein determination of the event template configuration comprises selecting each event template based on the event type and a target audience, wherein each event template comprises a set of event-specific image templates;

detecting one or more predefined objects or one or more predefined events in the recorded data in accordance with each event template; and

creating, with a processor, a remix video utilizing the event template and the one or more predefined objects or the one or more predefined events,

wherein each event template, selected based on the event type and the target audience, identifies which one or more predefined objects of a plurality of potential objects or one or more predefined events of a plurality of potential events are to be detected in the recorded data, and

wherein the event template configuration identifies a weighting of each event template'"'"'s inclusion in the remix video.

2. A method of claim 1, wherein the classifying of recorded data comprises:

estimating a quality level of each input data modality;

extracting one or more features from one or more of the plurality of data modalities;

classifying the one or more extracted features;

performing relation analysis for discovering correlative relations, anti-correlative relations, or a combination thereof between the one or more extracted features; and

performing fusion of feature classification using relation analysis results, quality levels of the input modalities, and location information to determine the event type.

3. A method of claim 1 wherein each event template comprises information related to:

assigning roles to one or more recording devices,

selecting a sub-set of editing rules;

selecting a sub-set of potential salient events; and

selecting a sub-set of potential salient objects.

4. A method of claim 1 further comprising:

assigning roles for one or more recording devices, according to the event type template, to record one or more potential salient events or one or more potential salient objects,

wherein the assigning of roles is a function of one or more of a number of recording devices, a capability of the recording devices, preference information of a person associated with the event, preference information of a person associated with the remix video, or a target audience.

5. A method of claim 1 wherein detecting a predefined object in the recorded data comprises:

selecting a sub-set of image templates; and

performing template matching, wherein selecting of the sub-set of image templates further comprises matching the auxiliary sensor data from the recorded data with auxiliary data associated with available templates.

6. A method of according to claim 1, wherein detecting a predefined object in the recorded data comprises analyzing auxiliary sensor data captured in the recorded data;

and

utilizing an offline phase and an online phase, wherein the offline phase comprises collecting a set of multimodal templates, and causing image templates and associated metadata information to be stored,

wherein the associated metadata information is auxiliary sensor data, and

wherein the online phase comprises template matching on the recorded data with a sub-set of the multimodal templates.

7. A method of claim 6, wherein template matching further comprises:

extracting features from the sub-set of image templates;

extracting features from the recorded data; and

matching the extracted features.

8. An apparatus comprising:

a processor and

a memory including software, the memory and the software configured to, with the processor, cause the apparatus to at least;

classify recorded data according to an event type, wherein the recorded data comprises a plurality of modalities;

determine an event template configuration, the event template configuration comprises one or more selected event templates,

wherein determination of the event template configuration comprises software configured to, with the processor, cause the apparatus to at least select each event template based on the event type and a target audience, wherein each event template comprises a set of event-specific image templates;

detect one or more predefined objects or one or more predefined events in the recorded data in accordance with the event template; and

create a remix video utilizing each event template and the one or more predefined objects or the one or more predefined events,

wherein each event template, selected based on the event type and the target audience, identifies which one or more predefined objects of a plurality of potential objects or one or more predefined events of a plurality of potential events are to be detected in the recorded data, and

wherein the event template configuration identifies a weighting of each event template'"'"'s inclusion in the remix video.

9. An apparatus of claim 8 wherein the at least one memory including the computer program code is further configured to, with the at least one processor, cause the apparatus to:

extract one or more features from one or more of the plurality of data modalities;

estimate a quality level of each input data modality;

classify the one or more extracted features;

perform relation analysis for discovering correlative relations, anti-correlative relations, or a combination thereof between the one or more extracted features; and

perform fusion of feature classification using relation analysis results, quality level of each modality and location information to determine the event type.

10. An apparatus of claim 8 wherein each event template comprises information related to:

assigning roles to one or more recording devices,

selecting a sub-set of editing rules;

selecting a sub-set of potential salient events; and

selecting a sub-set of potential salient objects.

11. An apparatus of claim 8 wherein the at least one memory including the computer program code is further configured to, with the at least one processor, cause the apparatus to:

assign roles for one or more recording devices, according to the event type template, to record one or more potential salient events or one or more potential salient objects,

wherein the at least one memory including the computer program code is further configured to, with the at least one processor, cause the apparatus to;

assign roles as a function of one or more of a number of recording devices, a capability of the recording devices, preference information of a person associated with the event, preference information of a person associated with the remix video, or a target audience.

12. An apparatus of claim 8, wherein the at least one memory including the computer program code is further configured to, with the at least one processor, cause the apparatus to select the sub-set of image templates by matching the auxiliary sensor data from the recorded data with auxiliary data associated to available templates.

13. An apparatus of claim 8, wherein the at least one memory including the computer program code is further configured to, with the at least one processor, cause the apparatus to detect a predefined object in the recorded data by:

analyzing auxiliary sensor data captured in the recorded data;

utilizing an offline phase and an online phase,

wherein the offline phase comprises collecting a set of multimodal templates, and storing image templates and associated metadata information,

wherein the associated metadata information is auxiliary sensor data, and

wherein the online phase comprises template matching on the recorded data with a sub-set of the multimodal templates.

14. An apparatus of claim 13, wherein the at least one memory including the computer program code is further configured to, with the at least one processor, cause the apparatus to perform template matching by:

extracting features from the sub-set of image templates;

extracting features from the recorded data; and

matching the extracted features.

15. A computer program product comprising at least one non-transitory computer-readable storage medium having computer-readable program instructions stored therein, the computer-readable program instructions include program instructions configured, when executed by a computer, to:

classify recorded data according to an event type, wherein the recorded data may comprise a plurality of modalities;

determine an event template configuration, the event template configuration comprises one or more selected event templates,

wherein determination of the event template configuration comprises selecting each event template based on the event type and a target audience, wherein each event template comprises a set of event-specific image templates;

detect one or more predefined objects or one or more predefined events in the recorded data in accordance with each event template; and

create a remix video utilizing each event template and the one or more predefined objects or the one or more predefined events,

wherein each event template, selected based on the event type and the target audience, identifies which one or more predefined objects of a plurality of potential objects or one or more predefined events of a plurality of potential events are to be detected in the recorded data, and

wherein the event template configuration identifies a weighting of each event template'"'"'s inclusion in the remix video.

16. A computer program product of claim 15, wherein the program instructions configured to classify recorded data also include program instructions configured, when executed by a computer, to:

estimate a quality level of each input data modality;

extract one or more features from one or more of the plurality of data modalities;

classify the one or more extracted features;

perform relation analysis for discovering correlative relations, anti-correlative relations, or a combination thereof between the one or more extracted features; and

perform fusion of feature classification using relation analysis results, quality level of each input data modality and location information to determine the event type.

17. A computer program product of claim 15 wherein each event template comprises information related to:

assigning roles to one or more recording devices,

selecting a sub-set of editing rules;

selecting a sub-set of potential salient events; and

selecting a sub-set of potential salient objects.

18. A computer program product of claim 15 wherein the program instructions also include program instructions configured, when executed by a computer, to:

assign roles for one or more recording devices, according to the event type template, to record one or more potential salient events or one or more potential salient objects,

wherein the assigning of roles is a function of one or more of a number of recording devices, a capability of the recording devices, preference information of a person associated with the event, preference information of a person associated with the remix video, or a target audience.

19. A computer program product of claim 15 wherein the computer-readable program instructions for detecting the predefined object also include program instructions configured, when executed by a computer, to:

select a sub-set of image templates; and

perform template matching, wherein selecting of the sub-set of image templates further comprises matching the auxiliary sensor data from the recorded data with auxiliary data associated with available templates.

20. A computer program product of according to claim 15, wherein the computer-readable program instructions configured to detect a predefined object in the recorded data also include program instructions configured, when executed by a computer, to analyze auxiliary sensor data captured in the recorded data;

and

utilize an offline phase and an online phase, wherein the offline phase comprises collecting a set of multimodal templates, and causing image templates and associated metadata information to be stored,

wherein the associated metadata information is auxiliary sensor data, and

wherein the online phase comprises template matching on the recorded data with a sub-set of the multimodal templates.

21. A computer program product of claim 20, wherein the computer-readable program instructions configured for template matching also include program instructions configured, when executed by a computer, to:

extract features from the sub-set of image templates;

extract features from the recorded data; and

match the extracted features.