Automatic generation of video and directional audio from spherical content

• US 10,056,115 B2
• Filed: 12/30/2016
• Issued: 08/21/2018
• Est. Priority Date: 07/03/2014
• Status: Active Grant

• Current Assignee: GoPro, Inc.
• Original Assignee: GoPro, Inc.
• Inventors: Campbell, Scott Patrick, Jing, Zhinian, Macmillan, Timothy, Newman, David A., Adsumilli, Balineedu Chowdary
• Primary Examiner(s): Dang, Hung
• Application Number:
  US15/396,385
• Publication Number:
  US 20170110155A1
• Time in Patent Office: 599 Days
• Field of Search: 386239-248
• CPC Class Codes: G11B 27/3081 used signal is a video-frame or a video-field (P.I.P)
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 5/77 between a recording apparatus and a television camera
• CPC Class Codes: H04N 9/806 with processing of the sound signal
• CPC Class Codes: H04N 9/8205 involving the multiplexing of an additional signal and the colour video signal
• CPC Class Codes: H04N 9/8211 the additional signal being a sound signal H04N9/835 takes precedence

Abstract

A spherical content capture system captures spherical video and audio content. In one embodiment, captured metadata or video/audio processing is used to identify content relevant to a particular user based on time and location information. The platform can then generate an output video from one or more shared spherical content files relevant to the user. The output video may include a non-spherical reduced field of view such as those commonly associated with conventional camera systems. Particularly, relevant sub-frames having a reduced field of view may be extracted from each frame of spherical video to generate an output video that tracks a particular individual or object of interest. For each sub-frame, a corresponding portion of an audio track is generated that includes a directional audio signal having a directionality based on the selected sub-frame.

Claims

1. A method for generating a video with corresponding audio, the method performing by a computing system including one or more processors, the method comprising:

receiving a video, by the computing system, the video comprising frames including a target, the video having a field of view;

receiving, by the computing system, audio channels representing audio captured concurrently with the video, individual audio channels comprising directional audio corresponding to respective different directions;

defining, by the computing system, respective spatial sub-regions to subdivide the frames of the video;

mapping, by the computing system, the audio channels to the spatial sub-regions of the video such that each sub-region is associated with at least one audio channel;

determining, by the computing system, a time-varying path of the target within the video based on an analysis of content of the video and/or information associated with the video;

extracting, by the computing system, sub-frames from the frames based on the time-varying path of the target, the sub-frames having a reduced field of view relative to the field of view of the video, the sub-frames including the target;

for individual sub-frames, by the computing system;

determining one or more of the spatial sub-regions overlapping a given sub-frame;

determining a composite audio channel, the composite audio channel including one or more of the audio channels mapped to the one or more of the spatial sub-regions overlapping the given sub-frame;

generating a portion of an audio stream from the composite audio channel; and

outputting, by the computing system, the sub-frames and the audio stream.

2. The method of claim 1, wherein determining the composite audio channel further comprises:

weighting the audio channels based on an amount of overlap between the spatial sub-regions and the sub-frames; and

combining the weighted audio channels.

3. The method of claim 1, wherein the individual audio channels corresponds to a direction perpendicular to faces of at least one member of a group consisting of:

a tetrahedron, a cube, or a pyramid.

4. The method of claim 1, wherein determining the time-varying path of the target within the video based on the analysis of the content of the video and/or the information associated with the video includes determining the time-varying path of the target within the video based on analysis of location information generated by a tracking device carried by the target.

5. The method of claim 1, wherein determining the time-varying path of the target within the video based on the analysis of the content of the video and/or the information associated with the video includes determining the time-varying path of the target within the video based on visual recognition of the target within the video.

6. The method of claim 5, wherein the visual recognition of the target includes facial recognition, object recognition, motion recognition, or gesture recognition.

7. The method of claim 1, wherein determining the time-varying path of the target within the video based on the analysis of the content of the video and/or the information associated with the video includes determining the time-varying path of the target within the video based directionality of an audio source of the audio captured concurrently with the video.

8. The method of claim 1, wherein mapping the audio channels to the spatial sub-regions is dependent on an orientation of a camera that was used to capture the video.

9. A non-transitory computer-readable storage medium storing instructions for generating a video with corresponding audio, the instructions when executed by one or more processors causing the one or more processors to perform steps including:

receiving a video, the video comprising frames including a target, the video having a field of view;

receiving audio channels representing audio captured concurrently with the video, individual audio channels comprising directional audio corresponding to respective different directions;

defining respective spatial sub-regions to subdivide the frames of the video;

mapping the audio channels to the spatial sub-regions of the video such that each sub-region is associated with at least one audio channel;

determining a time-varying path of the target within the video based on an analysis of content of the video and/or information associated with the video;

extracting sub-frames from the frames based on the time-varying path of the target, the sub-frames having a reduced field of view relative to the field of view of the video, the sub-frames including the target;

for individual sub-frames;

determining one or more spatial sub-regions overlapping a given sub-frame;

determining a composite audio channel, the composite audio channel including one or more of the audio channels mapped to the one or more of the spatial sub-regions overlapping the given sub-frame;

generating a portion of an audio stream from the composite audio channel; and

outputting the sub-frames and the audio stream.

10. The non-transitory computer-readable storage medium of claim 9, wherein determining the composite audio channel comprises:

weighting the audio channels based on an amount of overlap between the spatial sub-regions and the sub-frames; and

combining the weighted audio channels.

11. The non-transitory computer-readable storage medium of claim 9, wherein the individual audio channels corresponds to a direction perpendicular to the faces of at least one member of a group consisting of:

a tetrahedron, a cube, or a pyramid.

12. The non-transitory computer-readable storage medium of claim 9, wherein mapping the audio channels to the spatial sub-regions is dependent on an orientation of a camera that was used to capture the video.

13. The non-transitory computer-readable storage medium of claim 9, wherein determining the time-varying path of the target within the video based on the analysis of the content of the video and/or the information associated with the video includes determining the time-varying path of the target within the video based on analysis of location information generated by a tracking device carried by the target.

14. The non-transitory computer-readable storage medium of claim 9, wherein determining the time-varying path of the target within the video based on the analysis of the content of the video and/or the information associated with the video includes determining the time-varying path of the target within the video based on visual recognition of the target within the video.

15. A system for generating a video with corresponding audio, the system comprising:

one or more processors; and

a non-transitory computer-readable storage medium storing instructions that when executed by the one or more processors causes the one or more processors to perform steps including;

receiving a video, the video comprising frames including a target, the video having a field of view;

receiving audio channels representing audio captured concurrently with the video, individual audio channels comprising directional audio corresponding to respective different directions;

defining respective spatial sub-regions to subdivide the frames of the video, the spatial sub-regions corresponding to the respective different directions associated with the audio channels;

mapping the audio channels to the spatial sub-regions of the video such that each sub-region is associated with at least one audio channel;

determining a time-varying path of the target within the video based on an analysis of content of the video and/or information associated with the video;

extracting sub-frames from the frames based on the time-varying path of the target, the sub-frames having a reduced field of view relative to the field of view of the video, the sub-frames including the target;

for individual sub-frames;

determining one or more spatial sub-regions overlapping a given sub-frame;

determining a composite audio channel, the composite audio channel including one or more of the audio channels mapped to the one or more spatial sub-regions overlapping the given sub-frame;

generating a portion of an audio stream from the composite audio channel; and

outputting the sub-frames and the audio stream.

16. The system of claim 15, wherein determining the composite audio channel further comprises:

weighting the audio channels based on an amount of overlap between the spatial sub-regions and the sub-frames; and

combining the weighted audio channels.

17. The system of claim 15, wherein the individual audio channels corresponds to a direction perpendicular to faces of at least one member of a group consisting of:

a tetrahedron, a cube, or a pyramid.

18. The system of claim 15, wherein determining the time-varying path of the target within the video based on the analysis of the content of the video and/or the information associated with the video includes determining the time-varying path of the target within the video based on analysis of location information generated by a tracking device carried by the target.

19. The system of claim 15, wherein determining the time-varying path of the target within the video based on the analysis of the content of the video and/or the information associated with the video includes determining the time-varying path of the target within the video based on visual recognition of the target within the video.

20. The system of claim 15, wherein determining the time-varying path of the target within the video based on the analysis of the content of the video and/or the information associated with the video includes determining the time-varying path of the target within the video based directionality of an audio source of the audio captured concurrently with the video.