Generation of video from spherical content using edit maps

• US 9,760,768 B2
• Filed: 03/03/2015
• Issued: 09/12/2017
• Est. Priority Date: 03/04/2014
• Status: Active Grant

• Current Assignee: GoPro, Inc.
• Original Assignee: GoPro, Inc.
• Inventors: MacMillan, Timothy, Newman, David A., Adsumilli, Balineedu Chowdary, Campbell, Scott Patrick
• Primary Examiner(s): Findley, Christopher
• Application Number:
  US14/637,180
• Publication Number:
  US 20150256808A1
• Time in Patent Office: 924 Days
• CPC Class Codes: G03B 37/04 with cameras or projectors providing touching or overlapping fields of view
• CPC Class Codes: G06F 16/71 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06T 3/12 Panospheric to cylindrical image transformations
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: H04L 65/612 for unicast
• CPC Class Codes: H04L 65/762 at the source reformatting of additional data in video distribution servers H04N21/2355
• CPC Class Codes: H04N 13/106 Processing image signals for multi-view video sequence encoding H04N19/597
• CPC Class Codes: H04N 21/233 Processing of audio elementary streams monitoring, identification or recognition of audio in broadcast systems H04H60/58
• CPC Class Codes: H04N 21/23418 involving operations for analysing video streams, e.g. detecting features or characteristics television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture

Abstract

A spherical content capture system captures spherical video content. A spherical video sharing platform enables users to share the captured spherical content and enables users to access spherical content shared by other users. In one embodiment, captured metadata or video/audio processing is used to identify content relevant to a particular user based on time and location information. The platform can then generate an output video from one or more shared spherical content files relevant to the user. The output video may include a non-spherical reduced field of view such as those commonly associated with conventional camera systems. Particularly, relevant sub-frames having a reduced field of view may be extracted from each frame of spherical video to generate an output video that tracks a particular individual or object of interest.

Claims

1. A method for managing media content generated from spherical video, the method comprising:

storing, by a video server, a plurality of spherical videos;

receiving an edit map specifying for each frame time of an output video, an identifier of a spherical video of the plurality of spherical videos and a spatial location of a sub-frame in the identified spherical video, the sub-frame having a non-spherical field of view;

generating the output video based on the edit map, the output video including the sub-frame from the identified spherical video specified for each frame time, wherein the spherical video comprises a first hemispherical video and a second hemispherical video captured in synchronization, and wherein generating the output video comprises;

determining if a given sub-frame has a spatial region crossing a boundary between the first hemispherical video and the second hemispherical video;

responsive to the given sub-frame having the spatial region crossing the boundary, stitching corresponding portions of the first hemispherical video and the second hemispherical video contained within the given sub-frame to generate a stitched sub-frame; and

including the stitched sub-frame in the output video; and

outputting the output video.

2. The method of claim 1, further comprising:

receiving a user selection of a selected spherical video from the plurality of spherical videos on the video server;

displaying the selected spherical video in a spherical video viewer,

receiving user inputs indicating selections of sub-frames having a non-spherical field of view for each of a plurality of frames of the selected spherical video; and

generating the edit map based on the selections of the sub-frames.

3. The method of claim 2, wherein receiving the user selection of the selected spherical video comprises:

obtaining user metadata associated with a user, the metadata including at least time and location information;

correlating the user metadata to video metadata associated with each of the plurality of spherical videos to a identify a subset of the plurality of spherical videos captured at a time and location corresponding to the user time and location information;

providing a representation of the subset of the plurality of spherical videos in a user interface accessible to the user; and

receiving the user selection from the subset of the plurality of spherical videos.

4. The method of claim 1, further comprising:

correlating user metadata associated with a user to video data metadata associated with the plurality of spherical videos to automatically identify sub-frames relevant to the user; and

automatically generating the edit map based on the automatically identified sub-frames.

5. The method of claim 1, wherein generating the output video comprises:

performing one or more video processing functions specified by the edit map, the one or more video processing functions including at least one of;

an image stabilization function and a rolling shutter artifact reduction.

6. The method of claim 1, further comprising:

indexing the plurality of spherical videos based on metadata associated with the plurality of spherical videos including at least time and location information representation a time and location of capture.

7. The method of claim 1, wherein the edit map further specifies one or more video processing functions to be performed on the identified spherical video.

8. The method of claim 7, wherein the one or more video processing functions include one or more of stitching, image stabilization, or rolling artifact reduction.

9. A non-transitory computer-readable storage medium storing instructions for managing media content generated from spherical video, the instructions when executed by one or more processors causing the one or more processors to perform steps including:

storing a plurality of spherical videos;

receiving an edit map specifying for each frame time of an output video, an identifier of a spherical video of the plurality of spherical videos and a spatial location of a sub-frame in the identified spherical video, the sub-frame having a non-spherical field of view;

generating the output video based on the edit map, the output video including the sub-frame from the identified spherical video specified for each frame time, wherein the spherical video comprises a first hemispherical video and a second hemispherical video captured in synchronization, and wherein generating the output video comprises;

determining if a given sub-frame has a spatial region crossing a boundary between the first hemispherical video and the second hemispherical video;

responsive to the given sub-frame having the spatial region crossing the boundary, stitching corresponding portions of the first hemispherical video and the second hemispherical video contained within the given sub-frame to generate a stitched sub-frame; and

including the stitched sub-frame in the output video; and

outputting the output video.

10. The non-transitory computer-readable storage medium of claim 9, wherein the instructions when executed by the one or more processors further cause the one or more processors to perform steps including:

receiving a user selection of a selected spherical video from the plurality of spherical videos on the video server;

displaying the selected spherical video in a spherical video viewer, receiving user inputs indicating selections of sub-frames having a non-spherical field of view for each of a plurality of frames of the selected spherical video; and

generating the edit map based on the selections of the sub-frames.

11. The non-transitory computer-readable storage medium of claim 10, wherein receiving the user selection of the selected spherical video comprises:

obtaining user metadata associated with a user, the metadata including at least time and location information;

correlating the user metadata to video metadata associated with each of the plurality of spherical videos to a identify a subset of the plurality of spherical videos captured at a time and location corresponding to the user time and location information;

providing a representation of the subset of the plurality of spherical videos in a user interface accessible to the user; and

receiving the user selection from the subset of the plurality of spherical videos.

12. The non-transitory computer-readable storage medium of claim 9, wherein the instructions when executed by the one or more processors further cause the one or more processors to perform steps including:

correlating user metadata associated with a user to video data metadata associated with the plurality of spherical videos to automatically identify sub-frames relevant to the user; and

automatically generating the edit map based on the automatically identified sub-frames.

13. The non-transitory computer-readable storage medium of claim 9, wherein generating the output video comprises:

performing one or more video processing functions specified by the edit map, the one or more video processing functions including at least one of;

an image stabilization function and a rolling shutter artifact reduction.

14. The non-transitory computer-readable storage medium of claim 9, wherein the instructions when executed by the one or more processors further cause the one or more processors to perform steps including:

indexing the plurality of spherical videos based on metadata associated with the plurality of spherical videos including at least time and location information representation a time and location of capture.

15. A video server for managing media content generated from spherical video, the video server comprising:

one or more processors; and

a non-transitory computer-readable storage medium storing instructions for managing media content generated from spherical video, the instructions when executed by one or more processors causing the one or more processors to perform steps including;

storing a plurality of spherical videos;

receiving an edit map specifying for each frame time of an output video, an identifier of a spherical video of the plurality of spherical videos and a spatial location of a sub-frame in the identified spherical video, the sub-frame having a non-spherical field of view;

generating the output video based on the edit map, the output video including the sub-frame from the identified spherical video specified for each frame time, wherein the spherical video comprises a first hemispherical video and a second hemispherical video captured in synchronization, and wherein generating the output video comprises;

determining if a given sub-frame has a spatial region crossing a boundary between the first hemispherical video and the second hemispherical video;

responsive to the given sub-frame having the spatial region crossing the boundary, stitching corresponding portions of the first hemispherical video and the second hemispherical video contained within the given sub-frame to generate a stitched sub-frame; and

including the stitched sub-frame in the output video; and

outputting the output video.

16. The video server of claim 15, wherein the instructions when executed by the one or more processors further cause the one or more processors to perform steps including:

receiving a user selection of a selected spherical video from the plurality of spherical videos on the video server;

displaying the selected spherical video in a spherical video viewer,

receiving user inputs indicating selections of sub-frames having a non-spherical field of view for each of a plurality of frames of the selected spherical video; and

generating the edit map based on the selections of the sub-frames.

17. The video server of claim 16, wherein receiving the user selection of the selected spherical video comprises:

obtaining user metadata associated with a user, the metadata including at least time and location information;

correlating the user metadata to video metadata associated with each of the plurality of spherical videos to a identify a subset of the plurality of spherical videos captured at a time and location corresponding to the user time and location information;

providing a representation of the subset of the plurality of spherical videos in a user interface accessible to the user; and

receiving the user selection from the subset of the plurality of spherical videos.

18. The video server of claim 15, wherein the instructions when executed by the one or more processors further cause the one or more processors to perform steps including:

correlating user metadata associated with a user to video data metadata associated with the plurality of spherical videos to automatically identify sub-frames relevant to the user; and

automatically generating the edit map based on the automatically identified sub-frames.

19. The video server of claim 15, wherein generating the output video comprises:

performing one or more video processing functions specified by the edit map, the one or more video processing functions including at least one of;

an image stabilization function and a rolling shutter artifact reduction.

20. The video server of claim 15, wherein the instructions when executed by the one or more processors further cause the one or more processors to perform steps including:

indexing the plurality of spherical videos based on metadata associated with the plurality of spherical videos including at least time and location information representation a time and location of capture.