AUTOMATIC GENERATION OF VIDEO AND DIRECTIONAL AUDIO FROM SPHERICAL CONTENT

• US 20200135238A1
• Filed: 12/19/2019
• Published: 04/30/2020
• Est. Priority Date: 07/03/2014
• Status: Active Grant

• Current Assignee: GoPro, Inc.
• Original Assignee: GoPro, Inc.
• Inventors: Campbell, Scott Patrick, Jing, Zhinian, Macmillan, Timothy, Newman, David A., Adsumilli, Balineedu Chowdary
• Granted Patent: US 10,679,676 B2
• Time in Patent Office: Days
• CPC Class Codes: G11B 27/3081 used signal is a video-frame or a video-field (P.I.P)
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 5/77 between a recording apparatus and a television camera
• CPC Class Codes: H04N 9/806 with processing of the sound signal
• CPC Class Codes: H04N 9/8205 involving the multiplexing of an additional signal and the colour video signal
• CPC Class Codes: H04N 9/8211 the additional signal being a sound signal H04N9/835 takes precedence

Abstract

A spherical content capture system captures spherical video and audio content. In one embodiment, captured metadata or video/audio processing is used to identify content relevant to a particular user based on time and location information. The platform can then generate an output video from one or more shared spherical content files relevant to the user. The output video may include a non-spherical reduced field of view such as those commonly associated with conventional camera systems. Particularly, relevant sub-frames having a reduced field of view may be extracted from each frame of spherical video to generate an output video that tracks a particular individual or object of interest. For each sub-frame, a corresponding portion of an audio track is generated that includes a directional audio signal having a directionality based on the selected sub-frame.

Claims

1. A method for generating a video with corresponding audio, the method performing by a computing system including one or more processors, the method comprising:

receiving, by the computing system, video frames having a field of view;

receiving, by the computing system, directional audio captured with the video frames;

determining, by the computing system, a time-varying path of a target within the video frames based on an analysis of content of the video frames or information associated with the video frames;

identifying, by the computing system, sub-frames within the video frames based on the time-varying path of the target, the sub-frames including the target and having a reduced field of view relative to the field of view;

generating, by the computing system, an audio track based on the time-varying path of the target, the audio track including portions of the directional audio corresponding to a direction of the target; and

generating, by the computing system, an output video including the sub-frames and the audio track.

2. The method of claim 1, wherein generating the audio track comprises selecting the portions based on matching directionalities of the directional audio with the direction of the target.

3. The method of claim 2, wherein signals of the directional audio correspond to directions perpendicular to faces of at least one member of a group consisting of:

a tetrahedron, a cube, or a pyramid.

4. The method of claim 2, wherein the information associated with the video frames includes a directionality of an audio source of the directional audio.

5. The method of claim 2, wherein a scene motion analysis is performed based on the directional audio.

6. The method of claim 1, wherein the information associated with the video frames includes location information generated by a tracking device carried by the target.

7. The method of claim 1, wherein the analysis of the content of the video frames includes visual recognition of the target within the video frames.

8. The method of claim 7, wherein the visual recognition of the target includes facial recognition, object recognition, motion recognition, or gesture recognition.

9. A non-transitory computer-readable storage medium storing instructions for generating a video with corresponding audio, the instructions when executed by one or more processors causing the one or more processors to perform steps including:

receiving video frames having a field of view;

receiving directional audio captured with the video frames;

determining a time-varying path of a target within the video frames based on an analysis of content of the video frames or information associated with the video frames;

identifying sub-frames within the video frames based on the time-varying path of the target, the sub-frames including the target and having a reduced field of view relative to the field of view;

generating an audio track based on the time-varying path of the target, the audio track including portions of the directional audio corresponding to a direction of the target; and

generating an output video including the sub-frames and the audio track.

10. The non-transitory computer-readable storage medium of claim 9, wherein generating the audio track comprises selecting the portions based on matching directionalities of the directional audio with the direction of the target.

11. The non-transitory computer-readable storage medium of claim 10, wherein signals of the directional audio correspond to directions perpendicular to the faces of at least one member of a group consisting of:

a tetrahedron, a cube, or a pyramid.

12. The non-transitory computer-readable storage medium of claim 9, wherein the information associated with the video frames includes location information generated by a tracking device carried by the target.

13. The non-transitory computer-readable storage medium of claim 9, wherein the analysis of the content of the video frames includes visual recognition of the target within the video frames.

14. The non-transitory computer-readable storage medium of claim 13, wherein the visual recognition of the target includes facial recognition, object recognition, motion recognition, or gesture recognition.

15. A system for generating a video with corresponding audio, the system comprising:

one or more processors; and

a non-transitory computer-readable storage medium storing instructions that when executed by the one or more processors causes the one or more processors to perform steps including;

receiving video frames having a field of view;

receiving directional audio captured with the video frames;

determining a time-varying path of a target within the video frames based on an analysis of content of the video frames or information associated with the video frames;

identifying sub-frames within the video frames based on the time-varying path of the target, the sub-frames including the target and having a reduced field of view relative to the field of view;

generating an audio track based on the time-varying path of the target, the audio track including portions of the directional audio corresponding to a direction of the target; and

generating an output video including the sub-frames and the audio track.

16. The system of claim 15, wherein generating the audio tracker comprises selecting the portions based on matching directionalities of the directional audio with the direction of the target.

17. The system of claim 16, wherein signals of the directional audio correspond to directions perpendicular to faces of at least one member of a group consisting of:

a tetrahedron, a cube, or a pyramid.

18. The system of claim 16, wherein the information associated with the video frames includes a directionality of an audio source of the directional audio.

19. The system of claim 15, wherein the information associated with the video frames includes location information generated by a tracking device carried by the target.

20. The system of claim 15, wherein the analysis of the content of the video frames includes visual recognition of the target within the video frames.