AUTOMATIC GENERATION OF VIDEO FROM SPHERICAL CONTENT USING AUDIO/VISUAL ANALYSIS

• US 20150256746A1
• Filed: 03/03/2015
• Published: 09/10/2015
• Est. Priority Date: 03/04/2014
• Status: Active Grant

• Current Assignee: GoPro, Inc.
• Original Assignee: GoPro, Inc.
• Inventors: Newman, David A., Adsumilli, Balineedu Chowdary, Campbell, Scott Patrick, MacMillan, Timothy
• Granted Patent: US 9,652,667 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G03B 37/04 with cameras or projectors providing touching or overlapping fields of view
• CPC Class Codes: G06F 16/71 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06T 3/12 Panospheric to cylindrical image transformations
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: H04L 65/612 for unicast
• CPC Class Codes: H04L 65/762 at the source reformatting of additional data in video distribution servers H04N21/2355
• CPC Class Codes: H04N 13/106 Processing image signals for multi-view video sequence encoding H04N19/597
• CPC Class Codes: H04N 21/233 Processing of audio elementary streams monitoring, identification or recognition of audio in broadcast systems H04H60/58
• CPC Class Codes: H04N 21/23418 involving operations for analysing video streams, e.g. detecting features or characteristics television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture

Abstract

A spherical content capture system captures spherical video content. A spherical video sharing platform enables users to share the captured spherical content and enables users to access spherical content shared by other users. In one embodiment, captured metadata or video/audio processing is used to identify content relevant to a particular user based on time and location information. The platform can then generate an output video from one or more shared spherical content files relevant to the user. The output video may include anon-spherical reduced field of view such as those commonly associated with conventional camera systems. Particularly, relevant sub-frames having a reduced field of view may be extracted from each frame of spherical video o generate an output video that tracks a particular individual or object of interest.

Claims

1. A method for generating an output video from spherical video content, the method comprising:

storing, by a video server, a first spherical video comprising a sequence of spherical video frames, each having a spherical field of view and one or more audio tracks;

processing, by the video server, the first spherical video to identify a target feature of interest meeting one or more audio or visual criteria;

determining, by the video server, a first range of frames of the first spherical video having the target feature of interest;

determining, by the video server, a first sequence of sub-frames from each of the first range of frames, each of the first sequence of sub-frames having a non-spherical field of view, and each of the first sequence of sub-frames including a spatial region around the target feature of interest;

generating, by the video server, a first portion of an output video including the sequence of sub-frames; and

outputting the output video.

2. The method of claim 1, wherein processing the first spherical video to identify the target feature comprises:

performing a facial recognition algorithm on the first spherical video to identify spatial locations of one or more faces depicted in the first spherical video.

3. The method of claim 1, wherein processing the first spherical video to identify the target feature comprises:

performing an object recognition algorithm on the first spherical video to identify spatial locations of one or more objects depicted in the first spherical video.

4. The method of claim 1, wherein processing the first spherical video to identify the target feature comprises:

performing a motion analysis algorithm on the first spherical video to identify spatial locations of an object depicted in the first spherical video meeting predefined motion parameters.

5. The method of claim 1, wherein processing the first spherical video to identify the target feature comprises:

performing a motion analysis algorithm on the first spherical video to identify spatial locations of an object depicted in the first spherical video meeting predefined motion parameters.

6. The method of claim 1, wherein processing the first spherical video to identify the target feature comprises:

performing a gesture recognition algorithm on the first spherical video to identify spatial locations of an individual performing a particular gesture depicted in the first spherical video.

7. The method of claim 1, wherein processing the first spherical video to identify the target feature comprises:

performing an audio analysis on an audio track of first spherical video to identify a direction between an audio source and a camera capturing the first spherical audio; and

identifying a spatial location of the audio source in the first spherical video meeting based on the direction between the audio source and the camera.

8. The method of claim 1, further comprising:

storing, by the video server, a second spherical video comprising a sequence of spherical video frames each having a spherical field of view and one or more audio tracks;

processing, by the video server, the second spherical video to identify the target feature of interest meeting the one or more audio or visual criteria;

determining, by the video server, a second range of frames of the second spherical video having the target feature of interest;

determining, by the video server, a second sequence of sub-frames from each of the second range of frames, each of the second sequence of sub-frames having a non-spherical field of view, and each of the second sequence of sub-frames including a spatial region around the target feature of interest; and

generating, by the video server, a second portion of the output video including the second sequence of sub-frames.

9. A non-transitory computer-readable storage medium storing instructions for generating an output video from spherical video content, the instructions when executed by one or more processors causing the one or more processors to perform steps including:

storing a first spherical video comprising a sequence of spherical video frames each having a spherical field of view and one or more audio tracks;

processing the first spherical video to identify a target feature of interest meeting one or more audio or visual criteria;

determining a first range of frames of the first spherical video having the target feature of interest;

determining a first sequence of sub-frames from each of the first range of frames, each of the first sequence of sub-frames having a non-spherical field of view, and each of the first sequence of sub-frames including a spatial region around the target feature of interest;

generating a first portion of an output video including the sequence of sub-frames; and

outputting the output video.

10. The non-transitory computer-readable storage medium of claim 9, wherein processing the first spherical video to identify the target feature comprises:

performing a facial recognition algorithm on the first spherical video to identify spatial locations of one or more faces depicted in the first spherical video.

11. The non-transitory computer-readable storage medium of claim 9, wherein processing the first spherical video to identify the target feature comprises:

performing an object recognition algorithm on the first spherical video to identify spatial locations of one or more objects depicted in the first spherical video.

12. The non-transitory computer-readable storage medium of claim 9, wherein processing the first spherical video to identify the target feature comprises:

performing a motion analysis algorithm on the first spherical video to identify spatial locations of an object depicted in the first spherical video meeting predefined motion parameters.

13. The non-transitory computer-readable storage medium of claim 9, wherein processing the first spherical video to identify the target feature comprises:

performing a motion analysis algorithm on the first spherical video to identify spatial locations of an object depicted in the first spherical video meeting predefined motion parameters.

14. The non-transitory computer-readable storage medium of claim 9, wherein processing the first spherical video to identify the target feature comprises:

performing a gesture recognition algorithm on the first spherical video to identify spatial locations of an individual performing a particular gesture depicted in the first spherical video.

15. The non-transitory computer-readable storage medium of claim 9, wherein processing the first spherical video to identify the target feature comprises:

performing an audio analysis on an audio track of first spherical video to identify a direction between an audio source and a camera capturing the first spherical audio; and

identifying a spatial location of the audio source in the first spherical video meeting based on the direction between the audio source and the camera.

16. The non-transitory computer-readable storage medium of claim 9, wherein the instructions when executed by the one or more processors further cause the one or more processors to perform steps including:

storing a second spherical video comprising a sequence of spherical video frames each having a spherical field of view and one or more audio tracks;

processing the second spherical video to identify the target feature of interest meeting the one or more audio or visual criteria;

determining a second range of frames of the second spherical video having the target feature of interest;

determining a second sequence of sub-frames from each of the second range of frames, each of the second sequence of sub-frames having a non-spherical field of view, and each of the second sequence of sub-frames including a spatial region around the target feature of interest; and

generating a second portion of the output video including the second sequence of sub-frames.

17. A video server for generating an output video from spherical video content, the video server comprising:

one or more processors; and

a non-transitory computer-readable storage medium storing instructions that when executed by the one or more processors causing the one or more processors to perform steps including;

storing a first spherical video comprising a sequence of spherical video frames each having a spherical field of view and one or more audio tracks;

processing the first spherical video to identify a target feature of interest meeting one or more audio or visual criteria;

determining a first range of frames of the first spherical video having the target feature of interest;

determining a first sequence of sub-frames from each of the first range of frames, each of the first sequence of sub-frames having a non-spherical field of view, and each of the first sequence of sub-frames including a spatial region around the target feature of interest;

generating a first portion of an output video including the sequence of sub-frames; and

outputting the output video.

18. The video server of claim 17, wherein processing the first spherical video to identify the target feature comprises:

performing a facial recognition algorithm on the first spherical video to identify spatial locations of one or more faces depicted in he first spherical video.

19. The video server of claim 17, wherein processing the first spherical video to identify the target feature comprises:

performing an object recognition algorithm on the first spherical video to identify spatial locations of one or more objects depicted in the first spherical video.

20. The video server of claim 17, wherein processing the first spherical video to identify the target feature comprises:

performing an audio analysis on an audio track of first spherical video to identify a direction between an audio source and a camera capturing the first spherical audio; and

identifying a spatial location of the audio source in the first spherical video meeting based on the direction between the audio source and the camera.