Spherical video editing

• US 10,217,488 B1
• Filed: 12/15/2017
• Issued: 02/26/2019
• Est. Priority Date: 12/15/2017
• Status: Active Grant

• Current Assignee: Snap, Inc.
• Original Assignee: Snap, Inc.
• Inventors: Huang, Justin
• Primary Examiner(s): Chevalier, Robert
• Application Number:
  US15/844,089
• Time in Patent Office: 438 Days
• Field of Search: 386223, 386230, 386278, 386280, 386326, 386323
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2210/22 Cropping
• CPC Class Codes: G06T 3/16 Spatio-temporal transformations, e.g. video cubism
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: G06T 7/292 Multi-camera tracking
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G11B 27/031 Electronic editing of digitised analogue information signals, e.g. audio or video signals
• CPC Class Codes: H04N 23/60 Control of cameras or camera modules
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 5/2628 Alteration of picture size, shape, position or orientation, e.g. zooming, rotation, rolling, perspective, translation
• CPC Class Codes: H04N 5/265 Mixing
• CPC Class Codes: H04N 5/772 the recording apparatus and the television camera being placed in the same enclosure
• CPC Class Codes: H04N 5/9201 involving the multiplexing of an additional signal and the video signal
• CPC Class Codes: H04N 9/8042 involving data reduction
• CPC Class Codes: H04N 9/8205 involving the multiplexing of an additional signal and the colour video signal

Abstract

Systems and methods provide for editing of spherical video data. In one example, a computing device can receive a spherical video (or a video associated with an angular field of view greater than an angular field of view associated with a display screen of the computing device), such as by a built-in spherical video capturing system or acquiring the video data from another device. The computing device can display the spherical video data. While the spherical video data is displayed, the computing device can track the movement of an object (e.g., the computing device, a user, a real or virtual object represented in the spherical video data, etc.) to change the position of the viewport into the spherical video. The computing device can generate a new video from the new positions of the viewport.

Claims

1. A computer-implemented method, comprising:

receiving, using one or more processors of a computing device, first video data including spherical video data;

displaying a portion of a frame of the first video data on a display screen associated with the computing device, wherein the portion of the frame represents a viewport into the first video data and the viewport is associated with an original position for each frame of the first video data;

determining, within a duration between when the computing device detects a first input associated with generating second video data from the first video data and a second input associated with stopping generation of the second video data, movement data associated with an object within a time period when the display screen displays the portion of the frame of the first video data;

determining a plurality of new positions for the viewport using the movement data, wherein a new position corresponds to a movement associated with the object within the time period when the display screen displays the portion of the frame of the first video data; and

generating the second video data from the first video data using the plurality of new positions, wherein pixels of a frame of the second video data correspond to pixels of the frame of the first video data encompassed by the viewport at the new position.

2. The computer-implemented method of claim 1, wherein the object includes the computing device.

3. The computer-implemented method of claim 2, further comprising:

capturing at least one of motion sensor data or position sensor data of the computing device,

wherein the movement data is determined using at least one of the motion sensor data or the position sensor data.

4. The computer-implemented method of claim 3, further comprising:

capturing image data representing a physical environment of the computing device,

wherein the movement data is determined using a combination of the image data and at least one of the motion sensor data or the position sensor data.

5. The computer-implemented method of claim 1, wherein the object is displayed in the first video data.

6. The computer-implemented method of claim 1, wherein the plurality of new positions corresponds to a plurality of centroids of the viewport, and wherein generating the second video data from the first video data includes re-centering the viewport using the plurality of centroids.

7. The computer-implemented method of claim 1, further comprising:

determining at least one of rotation information, translation information, or transformation information for re-centering the viewport from the original position to the new position.

8. The computer-implemented method of claim 1, further comprising:

determining a surface or a volume to extract from the frame of the first video data for the frame of the second video data.

9. The computer-implemented method of claim 1, wherein the second video data includes an equirectangular frame having an angular field of view corresponding to an angular field of view of the display screen.

10. The computer-implemented method of claim 1, wherein the frame of the second video data is associated with an angular field of view along at least one dimension that is greater than an angular field of view along at least one corresponding dimension of a display screen of the computing device but less than 360 degrees.

11. The computer-implemented method of claim 10, wherein the frame of the second video data is associated with an angular field of view equal to 360 degrees along at least one second dimension.

12. The computer-implemented method of claim 1, further comprising:

determining available resources of a second computing device; and

transmitting, based on the available resources, at least one of a cropped version of the second video data, a version of the second video data having a lower video resolution, a version of the second video data have a lower rate of frames per second, or a version of the second video data having regions of varying video resolutions.

13. A computing device, comprising:

one or more processors;

a display screen; and

memory including instructions that, upon execution by the one or more processors, cause the computing device to;

receive first video data associated with an angular field of view greater than an angular field of view associated with the display screen;

display a portion of a frame of the first video data on the display screen, wherein the portion of the frame represents a viewport into the first video data and the viewport is associated with an original position for each frame of the first video data;

receive a first input to the computing device associated with generating second video data from the first video data;

determine movement data associated with an object within a time period when the display screen displays the portion of the frame of the first video data;

determine a plurality of new positions for the viewport using the movement data, wherein a new position corresponds to a movement associated with the object within the time period when the display screen displays the portion of the frame of the first video data; and

generate the second video data from the first video data using the plurality of new positions, wherein pixels of a frame of the second video data correspond to pixels of the frame of the first video data encompassed by the viewport at the new position.

14. The computing device of claim 13, further comprising a plurality of cameras, wherein the instructions upon execution further cause the computing device to:

capture a plurality of videos using the plurality of cameras; and

stitch the plurality of videos together to generate the first video data.

15. The computing device of claim 13, wherein the object includes at least a portion of a user of the computing device.

16. The computing device of claim 15, wherein the instructions upon execution further cause the computing device to:

determine a plurality of coordinates for mapping the pixels of the frame of the first video data to the pixels of the frame of the second video data.

17. A non-transitory computer-readable storage medium storing instructions that, upon execution by one or more processors of a computing device, cause the computing device to:

receive first video data including spherical video data;

receive a first input to the computing device associated with generating second video data from the first video data;

determine movement data associated with an object within a time period when the computing device displays a portion of a frame of the first video data, wherein the portion of the frame represents a viewport into the first video data and the viewport is associated with an original position for each frame of the first video data;

determine a plurality of new positions for the viewport using the movement data, wherein a new position corresponds to a movement associated with the object within the time period when the computing device displays the portion of the frame of the first video data; and

generate the second video data from the first video data using the plurality of new positions, wherein pixels of a frame of the second video data correspond to pixels of the frame of the first video data encompassed by the viewport at the new position.

18. The non-transitory computer-readable storage medium of claim 17, wherein the instructions upon execution further cause the computing device to:

retrieve the pixels of the frame of the first video data from a graphics buffer of the computing device.

19. The non-transitory computer-readable storage medium of claim 17, wherein the frame of the second video data is associated with an angular field of view along at least one first dimension that is greater than an angular field of view along at least one corresponding dimension of a display screen of the computing device but less than 360 degrees, and wherein the frame of the second video data is associated with an angular field of view equal to 360 degrees along at least one second dimension.

20. The non-transitory computer-readable storage medium of claim 17, wherein the instructions upon execution further cause the computing device to:

transmit an ephemeral message including the second video data to a second computing device,

wherein the second video data is deleted after the second video data has been viewed or after a specified time period of the second computing device receiving the second video data.