Spherical video stabilization based on accelerometer data

• US 10,084,962 B2
• Filed: 11/16/2016
• Issued: 09/25/2018
• Est. Priority Date: 11/16/2015
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Kokaram, Anil Christopher, Bengali, Husain, Birkbeck, Neil Aylon Charles, Inguva, Sasi
• Primary Examiner(s): Nazrul, Shahbaz
• Application Number:
  US15/353,488
• Publication Number:
  US 20170142337A1
• Time in Patent Office: 678 Days
• CPC Class Codes: H04N 23/6811 based on the image signal
• CPC Class Codes: H04N 23/6812 based on additional sensors, e.g. acceleration sensors
• CPC Class Codes: H04N 23/6815 by distinguishing pan or tilt from motion
• CPC Class Codes: H04N 23/683 performed by a processor, e.g. controlling the readout of an image memory
• CPC Class Codes: H04N 23/6845 by combination of a plurality of images sequentially taken
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture

Abstract

A method includes identifying a frame of a spherical video as a key frame, storing a compensation component based on position data of a camera in association with the key frame, and compensating for a movement of the camera in a subsequent frame of the spherical video based on the key frame and the compensation component associated with the key frame.

Claims

1. A method comprising:

identifying a frame of a spherical video as a key frame;

storing a compensation component based on position data of a camera in association with the key frame;

compensating for a movement of the camera in a subsequent frame of the spherical video based on the key frame and the compensation component associated with the key frame; and

in response to determining the spherical video was captured as partial spherical data, set a pixel of the movement compensated subsequent frame outside a display region to a default value.

2. The method of claim 1, further comprising:

capturing accelerometer data for the camera with each captured frame of the spherical video;

using the accelerometer data to generate at least one of a center position, a translation position and a rotation position associated with the key frame; and

storing the at least one of the center position, the translation position and the rotation position associated with the key frame, wherein the compensation component is based on the at least one of the center position, the translation position and the rotation position.

3. The method of claim 1, further comprising:

matching a portion of the subsequent frame to a plurality of portions of the key frame;

selecting one of the plurality of portions of the key frame for use in the compensating for the movement of the camera based on a direction of the compensation component; and

compensating for the movement of the camera in the subsequent frame using the selected one of the plurality of portions of the key frame.

4. The method of claim 1, wherein

the key frame is one of a plurality of key frames, and

the compensation component associated with the key frame is further based on a previous key frame of the plurality of key frames.

5. The method of claim 1, wherein the compensating for the camera movement is performed by one of a camera, a server configured to stream the spherical video and a device configured to render the spherical video.

6. The method of claim 1, wherein

the compensation component includes a center position and a translation position associated with the key frame, and

the compensating for the camera movement includes centering the subsequent frame based on the key frame center position and the key frame translation position.

7. The method of claim 1, wherein

the subsequent frame of the spherical video is stored as three dimensional (3D) data,

the compensation component includes a center position and a translation position associated with the key frame, and

the compensating for the camera movement includes rotating the subsequent frame in 3D based on the key frame center position and the key frame translation position.

8. The method of claim 1, wherein

the subsequent frame of the spherical video is stored as two dimensional (2D) data,

the compensation component includes a center position and a translation position associated with the key frame, and

the compensating for the camera movement includes shifting the subsequent frame in 2D based on the key frame center position and the key frame translation position, the method further comprising:

correcting for border image loss.

9. The method of claim 1, wherein

the subsequent frame of the spherical video is stored as two dimensional (2D) data,

the compensation component includes a center position and a translation position associated with the key frame,

the compensating for the camera movement includes shifting the subsequent frame in 2D based on the key frame center position and the key frame translation position;

upon determining the spherical video was not captured as partial spherical data, correcting for border image loss.

10. A device comprising;

a processor;

a memory configured to store instructions that when executed by the processor cause the device to;

identify a frame of a spherical video as a key frame;

store a compensation component based on position data in association with the key frame;

compensate for a camera movement in a subsequent frame of the spherical video based on the key frame and the compensation component associated with the key frame; and

in response to determining the spherical video was captured as partial spherical data, set pixels of the movement compensated subsequent frame outside a display region of the device to a default value.

11. The device of claim 10, wherein the device is one of a camera, a server configured to stream the spherical video and a device configured to render the spherical video.

12. The device of claim 10, wherein the instructions further cause the processor to:

capture accelerometer data for the camera with each captured frame of the spherical video; and

use the accelerometer data to generate at least one of a center position, a translation position and a rotation position associated with the key frame; and

store the at least one of the center position, the translation position and the rotation position associated with the key frame, wherein the compensation component is based on the at least one of the center position, the translation position and the rotation position.

13. The device of claim 10, wherein the instructions further cause the processor to:

match a portion of the subsequent frame to a plurality of portions of the key frame;

select one of the plurality of portions of the key frame for use in the compensating for the movement of the camera based on a direction of the compensation component, and

compensate for the movement of the camera in the subsequent frame using the selected one of the plurality of portions of the key frame.

14. The device of claim 10, wherein

the key frame is one of a plurality of key frames, and

the compensation component associated with the key frame is further based on a previous key frame of the plurality of key frames.

15. The device of claim 10, wherein

the compensation component includes a center position and a translation position associated with the key frame, and

the compensating for the camera movement includes centering the subsequent frame based on the key frame center position and the key frame translation position.

16. The device of claim 10, wherein

subsequent frame of the spherical video is stored as three dimensional (3D) data,

the compensation component includes a center position and a translation position associated with the key frame, and

the compensating for the camera movement includes rotating the subsequent frame in 3D based on the key frame center position and the key frame translation position.

17. The device of claim 10, wherein

subsequent frame of the spherical video is stored as two dimensional (2D) data,

the compensation component includes a center position and a translation position associated with the key frame, and

the compensating for the camera movement includes shifting the subsequent frame in 2D based on the key frame center position and the key frame translation position, the method further comprising:

correcting for border image loss.

18. The device of claim 10, wherein

the subsequent frame of the spherical video is stored as two dimensional (2D) data,

the compensation component includes a center position and a translation position associated with the key frame,

the compensating for the camera movement includes shifting the subsequent frame in 2D based on the key frame center position and the key frame translation position, and

upon determining the spherical video was not captured as partial spherical data, correcting for border image loss.

19. A non-transitory computer readable medium having code segments stored thereon, the code segments, when executed by a processor cause the processor to:

identify a frame of a spherical video as a key frame;

store a compensation component based on position data in association with the key frame;

compensate for a camera movement in a subsequent frame of the spherical video based on the key frame and the compensation component associated with the key frame; and

in response to determining the spherical video was not captured as partial spherical data, correcting for border image loss.

20. The non-transitory computer readable medium of claim 19, wherein

the subsequent frame of the spherical video is stored as two dimensional (2D) data,

the compensation component includes a center position and a translation position associated with the key frame,

the compensating for the camera movement includes shifting the subsequent frame in 2D based on the key frame center position and the key frame translation position,

upon determining the spherical video was captured as partial spherical data, set pixels of the movement compensated subsequent frame outside a display region to a default value.