Automatic generation of video from spherical content using location-based metadata

• US 9,754,159 B2
• Filed: 03/03/2015
• Issued: 09/05/2017
• Est. Priority Date: 03/04/2014
• Status: Active Grant

• Current Assignee: GoPro, Inc.
• Original Assignee: GoPro, Inc.
• Inventors: MacMillan, Timothy, Newman, David A.
• Primary Examiner(s): Krasnic, Bernard
• Application Number:
  US14/637,173
• Publication Number:
  US 20150254871A1
• Time in Patent Office: 917 Days
• Field of Search: None
• CPC Class Codes: G03B 37/04 with cameras or projectors providing touching or overlapping fields of view
• CPC Class Codes: G06F 16/71 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06T 3/12 Panospheric to cylindrical image transformations
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: H04L 65/612 for unicast
• CPC Class Codes: H04L 65/762 at the source reformatting of additional data in video distribution servers H04N21/2355
• CPC Class Codes: H04N 13/106 Processing image signals for multi-view video sequence encoding H04N19/597
• CPC Class Codes: H04N 21/233 Processing of audio elementary streams monitoring, identification or recognition of audio in broadcast systems H04H60/58
• CPC Class Codes: H04N 21/23418 involving operations for analysing video streams, e.g. detecting features or characteristics television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture

Abstract

A spherical content capture system captures spherical video content. A spherical video sharing platform enables users to share the captured spherical content and enables users to access spherical content shared by other users. In one embodiment, captured metadata or video/audio processing is used to identify content relevant to a particular user based on time and location information. The platform can then generate an output video from one or more shared spherical content files relevant to the user. The output video may include a non-spherical reduced field of view such as those commonly associated with conventional camera systems. Particularly, relevant sub-frames having a reduced field of view may be extracted from each frame of spherical video to generate an output video that tracks a particular individual or object of interest.

Claims

1. A method for generating an output video from spherical video content, the method comprising:

storing, by a video server, a first spherical video having first spherical video content and first video metadata including location data pertaining to a location of a first camera capturing the first spherical video content and timing data pertaining to a time of capture of the first spherical video content;

receiving user metadata representing a target path, the target path comprising a sequence of time-stamped locations corresponding to a target;

determining by the video server, based on the user metadata and the first video metadata, a first matching portion of the first spherical video, the first matching portion captured when the first camera was within a threshold vicinity of the target, wherein determining the first matching portion of the first spherical video comprises;

determining for each of a sequence of corresponding time points, distances between the target and the first camera based on the first video metadata and the user metadata;

determining a time range over which the distances are less than a distance threshold; and

determining the first matching portion based on the time range responsive to the time range exceeding a predefined time threshold;

determining a sequence of sub-frames by selecting, for each of a plurality of frames of the first matching portion of the first spherical video, a sub-frame having content relevant to the target path, each of the sequence of sub-frames comprising a non-spherical field of view;

combining the sequence of sub-frames to generate a first portion of the output video relevant to the target; and

outputting the output video.

2. The method of claim 1, wherein two or more of the selected sub-frames correspond to different spatial regions in different frames of the first spherical video.

3. The method of claim 1, wherein selecting the sub-frame comprises:

determining for a given frame of the first spherical video, a direction of the target relative to the first camera based on the first video metadata and the user metadata; and

selecting the sub-frame based on the direction.

4. The method of claim 1, further comprising:

storing, by the video server, a second spherical video having second spherical video content captured by a second camera and second video metadata;

determining by the video server, based on the user metadata and the second video metadata, a second matching portion of the second spherical video, the second matching portion captured when the second camera was within a threshold vicinity of the target;

for each of a plurality of frames of the second matching portion of the second spherical video, selecting a sub-frame having a non-spherical field of view, the sub-frame having content relevant to the target path;

combining the selected sub-frames to generate a second portion of the output video relevant to the target; and

combining the first portion of the output video with the second portion of the output video.

5. The method of claim 4, wherein combining the first portion of the output video with the second portion of the output video comprises:

identifying a time overlap between the first portion of the output video and the second portion of the output video; and

selecting between the first portion of the output video and the second portion of the output video during the time overlap based on proximity between the first camera and the target and between the second camera and the target.

6. The method of claim 1, further comprising:

receiving the user metadata from a location tracking device tracking the target path of the target.

7. A method for generating an output video from spherical video content, the method comprising:

storing, by a video server, a first spherical video having first spherical video content and first video metadata including location data pertaining to a location of a first camera capturing the first spherical video content and timing data pertaining to a time of capture of the first spherical video content;

receiving user metadata representing a target path, the target path comprising a sequence of time-stamped locations corresponding to a target;

determining by the video server, based on the user metadata and the first video metadata, a first matching portion of the first spherical video, the first matching portion captured when the first camera was within a threshold vicinity of the target;

determining a sequence of sub-frames by selecting, for each of a plurality of frames of the first matching portion of the first spherical video, a sub-frame having content relevant to the target path, each of the sequence of sub-frames comprising a non-spherical field of view;

combining the sequence of sub-frames to generate a first portion of the output video relevant to the target;

storing, by the video server, a second spherical video having second spherical video content captured by a second camera and second video metadata;

determining by the video server, based on the user metadata and the second video metadata, a second matching portion of the second spherical video, the second matching portion captured when the second camera was within a threshold vicinity of the target;

for each of a plurality of frames of the second matching portion of the second spherical video, selecting a sub-frame having a non-spherical field of view, the sub-frame having content relevant to the target path;

combining the selected sub-frames to generate a second portion of the output video relevant to the target;

combining the first portion of the output video with the second portion of the output video, wherein combining the first portion of the output video with the second portion of the output video comprises;

identifying a time overlap between the first portion of the output video and the second portion of the output video; and

selecting between the first portion of the output video and the second portion of the output video during the time overlap based on proximity between the first camera and the target and between the second camera and the target; and

outputting the output video.

8. The method of claim 7, wherein two or more of the selected sub-frames correspond to different spatial regions in different frames of the first spherical video.

9. The method of claim 7, wherein selecting the sub-frame comprises:

determining for a given frame of the first spherical video, a direction of the target relative to the first camera based on the first video metadata and the user metadata; and

selecting the sub-frame based on the direction.

10. The method of claim 7, further comprising:

receiving the user metadata from a location tracking device tracking the target path of the target.

11. A non-transitory computer-readable storage medium storing instructions for generating an output video from spherical video content, the instructions when executed by one or more processors causing the one or more processors to perform steps including:

storing a first spherical video having first spherical video content and first video metadata including location data pertaining to a location of a first camera capturing the first spherical video content and timing data pertaining to a time of capture of the first spherical video content;

receiving user metadata representing a target path, the target path comprising a sequence of time-stamped locations corresponding to a target;

determining, based on the user metadata and the first video metadata, a first matching portion of the first spherical video, the first matching portion captured when the first camera was within a threshold vicinity of the target, wherein determining the first matching portion of the first spherical video comprises;

determining for each of a sequence of corresponding time points, distances between the target and the first camera based on the first video metadata and the user metadata;

determining a time range over which the distances are less than a distance threshold; and

determining the first matching portion based on the time range responsive to the time range exceeding a predefined time threshold;

determining a sequence of sub-frames by selecting, for each of a plurality of frames of the first matching portion of the first spherical video, a sub-frame having content relevant to the target path, each of the sequence of sub-frames comprising a non-spherical field of view;

combining the sequence of sub-frames to generate a first portion of the output video relevant to the target; and

outputting the output video.

12. The non-transitory computer-readable storage medium of claim 11, wherein two or more of the selected sub-frames correspond to different spatial regions in different frames of the first spherical video.

13. The non-transitory computer-readable storage medium of claim 11, wherein selecting the sub-frame comprises:

determining for a given frame of the first spherical video, a direction of the target relative to the first camera based on the first video metadata and the user metadata; and

selecting the sub-frame based on the direction.

14. The non-transitory computer-readable storage medium of claim 11, wherein the instructions when executed by the one or more processors further cause the one or more processors to perform steps including:

storing a second spherical video having second spherical video content captured by a second camera and second video metadata;

determining, based on the user metadata and the second video metadata, a second matching portion of the second spherical video, the second matching portion captured when the second camera was within a threshold vicinity of the target;

for each of a plurality of frames of the second matching portion of the second spherical video, selecting a sub-frame having a non-spherical field of view, the sub-frame having content relevant to the target path;

combining the selected sub-frames to generate a second portion of the output video relevant to the target; and

combining the first portion of the output video with the second portion of the output video.

15. The non-transitory computer-readable storage medium of claim 14, wherein combining the first portion of the output video with the second portion of the output video comprises:

identifying a time overlap between the first portion of the output video and the second portion of the output video; and

selecting between the first portion of the output video and the second portion of the output video during the time overlap based on proximity between the first camera and the target and between the second camera and the target.

16. The non-transitory computer-readable storage medium of claim 11, wherein the instructions when executed by the one or more processors further cause the one or more processors to perform steps including:

receiving the user metadata from a location tracking device tracking the target path of the target.

17. A non-transitory computer-readable storage medium storing instructions for generating an output video from spherical video content, the instructions when executed by one or more processors causing the one or more processors to perform steps including:

storing a first spherical video having first spherical video content and first video metadata including location data pertaining to a location of a first camera capturing the first spherical video content and timing data pertaining to a time of capture of the first spherical video content;

receiving user metadata representing a target path, the target path comprising a sequence of time-stamped locations corresponding to a target;

determining, based on the user metadata and the first video metadata, a first matching portion of the first spherical video, the first matching portion captured when the first camera was within a threshold vicinity of the target;

determining a sequence of sub-frames by selecting, for each of a plurality of frames of the first matching portion of the first spherical video, a sub-frame having content relevant to the target path, each of the sequence of sub-frames comprising a non-spherical field of view;

combining the sequence of sub-frames to generate a first portion of the output video relevant to the target;

storing a second spherical video having second spherical video content captured by a second camera and second video metadata;

determining, based on the user metadata and the second video metadata, a second matching portion of the second spherical video, the second matching portion captured when the second camera was within a threshold vicinity of the target;

for each of a plurality of frames of the second matching portion of the second spherical video, selecting a sub-frame having a non-spherical field of view, the sub-frame having content relevant to the target path;

combining the selected sub-frames to generate a second portion of the output video relevant to the target;

combining the first portion of the output video with the second portion of the output video, wherein combining the first portion of the output video with the second portion of the output video comprises;

identifying a time overlap between the first portion of the output video and the second portion of the output video; and

selecting between the first portion of the output video and the second portion of the output video during the time overlap based on proximity between the first camera and the target and between the second camera and the target; and

outputting the output video.

18. The non-transitory computer-readable storage medium of claim 17, wherein two or more of the selected sub-frames correspond to different spatial regions in different frames of the first spherical video.

19. The non-transitory computer-readable storage medium of claim 17, wherein selecting the sub-frame comprises:

determining for a given frame of the first spherical video, a direction of the target relative to the first camera based on the first video metadata and the user metadata; and

selecting the sub-frame based on the direction.

20. The non-transitory computer-readable storage medium of claim 17, wherein the instructions when executed by the one or more processors further cause the one or more processors to perform steps including:

receiving the user metadata from a location tracking device tracking the target path of the target.