VIDEO ANALYSIS TECHNIQUES FOR IMPROVED EDITING, NAVIGATION, AND SUMMARIZATION

• US 20160092561A1
• Filed: 12/03/2014
• Published: 03/31/2016
• Est. Priority Date: 09/30/2014
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Liu, Shujie, Zhang, Ke, Zhou, Xiaosong, Wu, Hsi-Jung, Chung, Chris Y., Normile, James O., Keinath, Albert E., Price, Douglas S.
• Granted Patent: US 10,452,713 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/739 in form of a video summary, e.g. the video summary being a video sequence, a composite still image or having synthesized frames
• CPC Class Codes: G06F 16/745 the internal structure of a single video sequence
• CPC Class Codes: G06F 16/783 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/7837 using objects detected or recognised in the video content
• CPC Class Codes: G06F 16/7867 using information manually generated, e.g. tags, keywords, comments, title and artist information, manually generated time, location and usage information, user ratings
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G06V 20/47 Detecting features for summarising video content
• CPC Class Codes: G11B 27/031 Electronic editing of digitised analogue information signals, e.g. audio or video signals
• CPC Class Codes: G11B 27/3081 used signal is a video-frame or a video-field (P.I.P)
• CPC Class Codes: G11B 27/34 Indicating arrangements indicating means incorporated in magazine or cassette G11B23/046 and G11B23/0875; indicating measured values in general G01D

Abstract

Systems and processes for improved video editing, summarization and navigation based on generation and analysis of metadata are described. The metadata may be content-based (e.g., differences between neighboring frames, exposure data, key frame identification data, motion data, or face detection data) or non-content-based (e.g., exposure, focus, location, time) and used to prioritize and/or classify portions of video. The metadata may be generated at the time of image capture or during post-processing. Prioritization information, such as a score for various portions of the image data may be based on the metadata and/or image data. Classification information such as the type or quality of a scene may be determined based on the metadata and/or image data. The classification and prioritization information may be metadata and may be used to automatically remove undesirable portions of the video, generate suggestions during editing or automatically generate summary video.

Claims

1. A system, comprising:

one or more processors configured to implement a video analyzer comprising;

a prioritization component configured to receive video data and provide prioritization information for a plurality of segments of the video data, wherein the video data comprises a plurality of frames and corresponding metadata, wherein the prioritization component comprises;

a metadata analyzer configured to analyze the metadata from the video data and determine metadata-based prioritization data for one or more portions of the video data based at least in part on analysis of the metadata;

a content analyzer configured to analyze content of the plurality of frames and determine content-based prioritization data for one or more portions of the video data based at least in part on analysis of the content of the plurality of frames; and

a video prioritization generator configured to generate the prioritization information based at least in part on the metadata-based data and the content-based data, wherein the prioritization information indicates a priority for the plurality of segments of the video data.

2. The system of claim 1 wherein, wherein the video analyzer further comprises a classification component configured to receive video data and provide classification information for a plurality of groups of the video data, wherein the classification component comprises:

a classification metadata analyzer configured to analyze the metadata from the video data and determine metadata-based classification data from one or more portions of the video data based at least in part on analysis of the metadata;

a classification content analyzer configured to analyze content of the plurality of frames and determine content-based classification data from one or more portions of the video data based at least in part on analysis of the content of the plurality of the frames; and

a video classification generator configured to generate the classification information based at least in part on the meta-data-based data and the content-based data, wherein the classification information indicates a class for the plurality of groups of the video data.

3. The system of claim 2, wherein the video analyzer further comprises a summarizer component configured to:

receive the prioritization information and the classification information;

generate a summary video based on the prioritization and classification information, the generation comprising removal of portions of video data and organization of the remaining video data; and

output the generated summary video.

4. The system of claim 1, wherein the metadata comprises non-content-based metadata for at least some of the plurality of frames, wherein the non-content-based metadata comprises one or more of:

motion information for a device capturing the video data, exposure information for the device capturing the video data, focus position information for the device capturing the video data, location information for the device capturing the video data, or time data for when the video data for a plurality of frames was captured.

5. The system of claim 1, wherein the metadata comprises content-based metadata for at least some of the plurality of frames, wherein the content-based metadata is based at least in part on content analysis of the video data, wherein the content-based metadata comprises one or more of:

difference between neighboring frames data, exposure data, key frame identification data, motion data, similarity data (for possible merge), or face detection data.

6. A method, comprising:

receiving video data comprising a plurality of frames and corresponding metadata;

determining a prioritization for a plurality of segments of the video data, wherein said determining the prioritization comprises analyzing the metadata from the video data to determine metadata-based prioritization data for one or more portions of the video data, analyzing content of the plurality of frames to determine content-based prioritization data for one or more portions of the video data, and determining the prioritization based at least in part on combining the metadata-based prioritization data and the content-based prioritization data;

classifying the video data into a classification comprising a plurality of groups of the frames, wherein said classifying comprises analyzing the metadata from the video data to determine metadata-based classification data for one or more portions of the video data, analyzing content of the plurality of frames to determine content-based classification data for one or more portions of the video data, and classifying the frames of the video data into the classification based at least in part on combining the metadata-based classification data and the content-based classification data; and

editing, navigating, or summarizing the video data based at least in part on the prioritization or the classification.

7. The method of claim 6, wherein editing the video data comprises removing undesirable portions of the video data based at least in part on the classification data and the prioritization data of the video data, wherein an amount of the video data removed is a portion of the video data that falls below an image quality threshold or is based on a threshold amount of the video data that is classified as undesirable.

8. The method of claim 6, further comprising:

generating content ranking scores for segments of the video data based at least in part on the prioritization for the plurality of segments of the video data; and

wherein editing the video data comprises removing segments of the video data based on respective ranking scores.

9. The method of claim 8, wherein summarizing the video data comprises:

generating other content ranking scores for remaining segments of the video data based at least in part on an importance of each respective segment;

determining a length for a summarization video based at least in part on a number of important scenes in the video data; and

generating a summarization video based on the other content ranking scores of the remaining segments of the video data and the determined length.

10. The method of claim 9, further comprising:

determining the importance of each respective segment, wherein increased importance is based at least in part on face recognition or speech recognition; and

wherein the determined importance is used to perform the generation of the other content ranking scores.

11. The method of claim 6,

wherein editing the video data comprises:

determining a plurality of scenes in the video data based on analysis of;

location or time metadata associated with the video data,

differences between neighboring frames,

image-based content analysis,

motion analysis, or

audio characteristics; and

generating a story board comprising at least some of the plurality of scenes; and

wherein navigating the video data comprises;

displaying the story board; and

providing navigation tools for navigating among the plurality of scenes of the displayed story board.

12. The method of claim 6, further comprising:

determining a user preference profile for a particular user based on at least one of social data, historical editing selections, or user-configurable user preferences; and

wherein editing the video data comprises generating editing suggestions based at least in part on the user preference profile;

or

wherein summarization of the video data comprises generating a summary video based at least in part on the user preference profile.

13. A method, comprising:

receiving image data for a plurality of frames;

determining non-content-based metadata for at least some of the plurality of frames, wherein the non-content-based metadata comprises one or more of;

motion information for a device capturing the image data, exposure information for the device capturing the image data, focus position information for the device capturing the image data, location information for the device capturing the image data, or time data for when the image data for a plurality of frames was captured;

determining content-based metadata for at least some of the plurality of frames, wherein the content-based metadata is based at least in part on content analysis of the image data, wherein the content-based metadata comprises one or more of;

difference between neighboring frames data, exposure data, key frame identification data, motion data, similarity data (for possible merge), or face detection data; and

outputting video data based at least in part on the image data for a plurality of frames and including at least some of the non-content-based metadata and the content-based metadata in the output video data.

14. The method of claim 13, wherein said determining non-content-based metadata for at least some of the plurality of frames comprises determining the motion information, exposure information, focus position, location information or time data for each of the at least some of the plurality of frames while the respective frame is in an image signal processor.

15. The method of claim 13, wherein said determining content-based metadata for at least some of the plurality of frames comprises determining the difference between neighboring frames data, exposure data, key frame identification data, motion data, similarity data or face detection data for each of the at least some of the plurality of frames while the respective frame is in an image signal processor.

16. The method of claim 13, further comprising:

generating prioritization information for one or more of the plurality of frames based at least in part on the non-content-based metadata and the content-based metadata, wherein the prioritization information indicates a priority among the plurality frames; and

incorporating the generated prioritization information into metadata for respective frames.

17. The method of claim 13, further comprising determining a prioritization for a plurality of segments of the image data, wherein said determining the prioritization comprises analyzing the metadata from the image data to determine metadata-based prioritization data for one or more portions of the image data, analyzing content of the plurality of frames to determine content-based prioritization data for one or more portions of the image data, and determining the prioritization based at least in part on combining the metadata-based prioritization data and the content-based prioritization data.

18. The method of claim 17, wherein the determination of the content-based metadata and the prioritization determination are performed outside of an image signal processor.

19. The method of claim 17, the method further comprising:

generating editing suggestions or a highlights video based at least in part on the prioritization metadata; and

instructing display of the editing suggestions or the highlights video.

20. The method of claim 13, further comprising classifying the image data into a classification comprising a plurality of groups of the plurality of frames, wherein said classifying comprises analyzing the metadata from the image data to determine metadata-based classification data for one or more portions of the image data, analyzing content of the plurality of frames to determine content-based classification data for one or more portions of the image data, and classifying the frames of the image data into the classification based at least in part on combining the metadata-based classification data and the content-based classification data.