Automatic generation of video and directional audio from spherical content
First Claim
1. A method for generating a video with corresponding audio, comprising:
- receiving a spherical video;
receiving a plurality of audio channels representing audio captured concurrently with the spherical video, each of the plurality of audio channels comprising directional audio corresponding to respective different directions;
defining respective spatial sub-regions to subdivide the frame of the spherical video;
mapping each of the plurality of audio channels to different spatial sub-regions of the spherical video such that each sub-region is associated with at least one audio channel;
for each of a plurality of frames of the spherical video, extracting a sub-frame having a reduced field of view relative to a field of view of the spherical video;
for each of the sub-frames, by a processor,determining one or more of the spatial sub-regions overlapping the sub-frame;
determining a composite audio channel, the composite audio channel being a combination of the audio channels mapped to the one or more of the spatial sub-regions overlapping the sub-frame;
generating a portion of an audio stream from the composite audio channel; and
outputting the generated video and the audio stream.
3 Assignments
0 Petitions
Accused Products
Abstract
A spherical content capture system captures spherical video and audio content. In one embodiment, captured metadata or video/audio processing is used to identify content relevant to a particular user based on time and location information. The platform can then generate an output video from one or more shared spherical content files relevant to the user. The output video may include a non-spherical reduced field of view such as those commonly associated with conventional camera systems. Particularly, relevant sub-frames having a reduced field of view may be extracted from each frame of spherical video to generate an output video that tracks a particular individual or object of interest. For each sub-frame, a corresponding portion of an audio track is generated that includes a directional audio signal having a directionality based on the selected sub-frame.
28 Citations
20 Claims
-
1. A method for generating a video with corresponding audio, comprising:
-
receiving a spherical video; receiving a plurality of audio channels representing audio captured concurrently with the spherical video, each of the plurality of audio channels comprising directional audio corresponding to respective different directions; defining respective spatial sub-regions to subdivide the frame of the spherical video; mapping each of the plurality of audio channels to different spatial sub-regions of the spherical video such that each sub-region is associated with at least one audio channel; for each of a plurality of frames of the spherical video, extracting a sub-frame having a reduced field of view relative to a field of view of the spherical video; for each of the sub-frames, by a processor, determining one or more of the spatial sub-regions overlapping the sub-frame; determining a composite audio channel, the composite audio channel being a combination of the audio channels mapped to the one or more of the spatial sub-regions overlapping the sub-frame; generating a portion of an audio stream from the composite audio channel; and outputting the generated video and the audio stream. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8)
-
-
9. A non-transitory computer-readable storage medium storing instructions for generating a video with corresponding audio, the instructions when executed by one or more processors causing the one or more processors to perform steps including:
-
receiving a spherical video; receiving a plurality of audio channels representing audio captured concurrently with the spherical video, each of the plurality of audio channels comprising directional audio corresponding to respective different directions; defining respective spatial sub-regions to subdivide the frame of the spherical video; mapping each of the plurality of audio channels to different spatial sub-regions of the spherical video such that each sub-region is associated with at least one audio channel; for each of a plurality of frames of the spherical video, extracting a sub-frame having a reduced field of view relative to a field of view of the spherical video; for each of the sub-frames, by a processor, determining one or more spatial sub-regions overlapping the sub-frame; determining a composite audio channel, the composite audio channel being a combination of the audio channels mapped to the one or more of the spatial sub-regions overlapping the sub-frame; generating a portion of an audio stream from the composite audio channel; and outputting the generated video and the audio stream. - View Dependent Claims (10, 11, 12, 13, 14)
-
-
15. A media server system for generating a video with corresponding audio, the video server system comprising:
-
one or more processors; and a non-transitory computer-readable storage medium storing instructions that when executed by the one or more processors causes the one or more processors to perform steps including; receiving a spherical video; receiving a plurality of audio channels representing audio captured concurrently with the spherical video, each of the plurality of audio channels comprising directional audio corresponding to respective different directions; defining respective spatial sub-regions to subdivide the frame of the spherical video, the spatial sub-regions corresponding to the respective different directions associated with each of the plurality of audio channels; mapping each of the plurality of audio channels to different spatial sub-regions of the spherical video such that each sub-region is associated with at least one audio channel; for each of a plurality of frames of the spherical video, extracting a sub-frame having a reduced field of view relative to a field of view of the spherical video; for each of the sub-frames, by a processor, determining one or more spatial sub-regions overlapping the sub-frame; determining a composite audio channel, the composite audio channel being a combination of the one or more of the audio channels mapped to the spatial sub-regions overlapping the sub-frame; generating a portion of an audio stream from the composite audio channel; and outputting the generated video and the audio stream. - View Dependent Claims (16, 17, 18, 19, 20)
-
Specification