Automatic Generation of Video and Directional Audio From Spherical Content

• US 20170110155A1
• Filed: 12/30/2016
• Published: 04/20/2017
• Est. Priority Date: 07/03/2014
• Status: Active Grant

• Current Assignee: GoPro, Inc.
• Original Assignee: GoPro, Inc.
• Inventors: Campbell, Scott Patrick, Jing, Zhinian, Macmillan, Timothy, Newman, David A., Adsumilli, Balineedu Chowdary
• Granted Patent: US 10,056,115 B2
• Time in Patent Office: Days
• CPC Class Codes: G11B 27/3081 used signal is a video-frame or a video-field (P.I.P)
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 5/77 between a recording apparatus and a television camera
• CPC Class Codes: H04N 9/806 with processing of the sound signal
• CPC Class Codes: H04N 9/8205 involving the multiplexing of an additional signal and the colour video signal
• CPC Class Codes: H04N 9/8211 the additional signal being a sound signal H04N9/835 takes precedence

Abstract

A spherical content capture system captures spherical video and audio content. In one embodiment, captured metadata or video/audio processing is used to identify content relevant to a particular user based on time and location information. The platform can then generate an output video from one or more shared spherical content files relevant to the user. The output video may include a non-spherical reduced field of view such as those commonly associated with conventional camera systems. Particularly, relevant sub-frames having a reduced field of view may be extracted from each frame of spherical video to generate an output video that tracks a particular individual or object of interest. For each sub-frame, a corresponding portion of an audio track is generated that includes a directional audio signal having a directionality based on the selected sub-frame.

Claims

1. A method for generating a video with corresponding audio, comprising:

receiving a video;

receiving a plurality of audio channels representing audio captured concurrently with the video, each of the plurality of audio channels comprising directional audio corresponding to respective different directions;

defining respective spatial sub-regions to subdivide the frame of the video;

mapping each of the plurality of audio channels to different spatial sub-regions of the video such that each sub-region is associated with at least one audio channel;

for each of a plurality of frames of the video, extracting a sub-frame having a reduced field of view relative to a field of view of the video;

for each of the sub-frames, by a processor,

determining one or more of the spatial sub-regions overlapping the sub-frame;

determining a composite audio channel, the composite audio channel being a combination of the audio channels mapped to the one or more of the spatial sub-regions overlapping the sub-frame;

generating a portion of an audio stream from the composite audio channel; and

outputting the generated video and the audio stream.

2. The method of claim 1, wherein determining the composite audio channel further comprises:

weighting each of the plurality of audio channels based on an amount of overlap between the spatial sub-regions and the extracted sub-frame; and

combining the weighted plurality of audio channels.

3. The method of claim 1, wherein each of the plurality of audio channels corresponds to a direction perpendicular to the faces of at least one member of a group consisting of:

a tetrahedron, a cube, or a pyramid.

4. The method of claim 1, further comprising:

receiving video metadata including location data pertaining to a location of a camera capturing the video and timing data pertaining to a time of capture of the video;

receiving user metadata representing a target path, the target path comprising a sequence of time-stamped locations corresponding to a target; and

determining the sub-frame as having content relevant to the target path.

5. The method of claim 4, wherein determining the sub-frame as having content relevant to the target path comprises:

determining a direction of the target relative to the camera based on the video metadata and the user metadata; and

selecting the sub-frame based on the direction.

6. The method of claim 1, wherein extracting the sub-frame comprises:

performing a recognition algorithm to identify at least one from the group consisting of;

a motion, an object, or a gesture; and

forming the sub-frame with the motion, the object or the gesture within the sub-frame.

7. The method of claim 1, wherein extracting the sub-frame comprises:

querying a user to identify a set of boundaries within the frame of the video; and

forming the sub-frame with the set of boundaries indicated by the user.

8. The method of claim 1, wherein mapping each of the plurality of audio channels to different spatial sub-regions is dependent on the orientation of the camera that was used to capture the frame.

9. A non-transitory computer-readable storage medium storing instructions for generating a video with corresponding audio, the instructions when executed by one or more processors causing the one or more processors to perform steps including:

receiving a video;

receiving a plurality of audio channels representing audio captured concurrently with the video, each of the plurality of audio channels comprising directional audio corresponding to respective different directions;

defining respective spatial sub-regions to subdivide the frame of the video;

mapping each of the plurality of audio channels to different spatial sub-regions of the video such that each sub-region is associated with at least one audio channel;

for each of a plurality of frames of the video, extracting a sub-frame having a reduced field of view relative to a field of view of the video;

for each of the sub-frames, by a processor,

determining one or more spatial sub-regions overlapping the sub-frame;

determining a composite audio channel, the composite audio channel being a combination of the audio channels mapped to the one or more of the spatial sub-regions overlapping the sub-frame;

generating a portion of an audio stream from the composite audio channel; and

outputting the generated video and the audio stream.

10. The non-transitory computer-readable storage medium of claim 9, wherein determining the composite audio channel comprises:

weighting each of the plurality of audio channels based on an amount of overlap between the spatial sub-regions and the extracted sub-frame; and

combining the weighted plurality of audio channels.

11. The non-transitory computer-readable storage medium of claim 9, wherein each of the plurality of audio channels corresponds to a direction perpendicular to the faces of at least one member of a group consisting of:

a tetrahedron, a cube, or a pyramid.

12. The non-transitory computer-readable storage medium of claim 9, wherein mapping each of the plurality of audio channels to different spatial sub-regions is dependent on the orientation of the camera that was used to capture the frame.

13. The non-transitory computer-readable storage medium of claim 9, the instructions when executed further causing the one or more processors to perform steps including:

receiving video metadata including location data pertaining to a location of a camera capturing the video and timing data pertaining to a time of capture of the video;

receiving user metadata representing a target path, the target path comprising a sequence of time-stamped locations corresponding to a target; and

determining the sub-frame as having content relevant to the target path.

14. The non-transitory computer-readable storage medium of claim 13, wherein extracting the sub-frame comprises:

determining a direction of the target relative to the camera based on the video metadata and the user metadata; and

selecting the sub-frame based on the direction.

15. A media server system for generating a video with corresponding audio, the video server system comprising:

one or more processors; and

a non-transitory computer-readable storage medium storing instructions that when executed by the one or more processors causes the one or more processors to perform steps including;

receiving a video;

receiving a plurality of audio channels representing audio captured concurrently with the video, each of the plurality of audio channels comprising directional audio corresponding to respective different directions;

defining respective spatial sub-regions to subdivide the frame of the video, the spatial sub-regions corresponding to the respective different directions associated with each of the plurality of audio channels;

mapping each of the plurality of audio channels to different spatial sub-regions of the video such that each sub-region is associated with at least one audio channel;

for each of a plurality of frames of the video, extracting a sub-frame having a reduced field of view relative to a field of view of the video;

for each of the sub-frames, by a processor,

determining one or more spatial sub-regions overlapping the sub-frame;

determining a composite audio channel, the composite audio channel being a combination of the one or more of the audio channels mapped to the spatial sub-regions overlapping the sub-frame;

generating a portion of an audio stream from the composite audio channel; and

outputting the generated video and the audio stream.

16. The media server system of claim 15, wherein determining the composite audio channel further comprises:

weighting each of the plurality of audio channels based on an amount of overlap between the spatial sub-regions and the extracted sub-frame; and

combining the weighted plurality of audio channels.

17. The media server system of claim 15, wherein each of the plurality of audio channels corresponds to a direction perpendicular to the faces of at least one member of a group consisting of:

a tetrahedron, a cube, or a pyramid.

18. The media server system of claim 15, the instructions when executed further causing the one or more processors to perform steps including:

receiving video metadata including location data pertaining to a location of a camera capturing the video and timing data pertaining to a time of capture of the video;

receiving user metadata representing a target path, the target path comprising a sequence of time-stamped locations corresponding to a target; and

determining the sub-frame as having content relevant to the target path.

19. The media server system of claim 18, wherein extracting the sub-frame comprises:

determining a direction of the target relative to the camera based on the video metadata and the user metadata; and

selecting the sub-frame based on the direction.

20. The media server system of claim 18, wherein mapping each of the plurality of audio channels to different spatial sub-regions is dependent on the orientation of the camera that was used to capture the frame.