Stabilization of panoramic video

• US 9,843,724 B1
• Filed: 09/21/2015
• Issued: 12/12/2017
• Est. Priority Date: 09/21/2015
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Brailovskiy, Ilya Vladimirovich
• Primary Examiner(s): Mikeska, Neil
• Application Number:
  US14/859,969
• Time in Patent Office: 813 Days
• CPC Class Codes: G06T 3/12 Panospheric to cylindrical image transformations
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: H04N 23/6811 based on the image signal
• CPC Class Codes: H04N 23/6812 based on additional sensors, e.g. acceleration sensors
• CPC Class Codes: H04N 23/6815 by distinguishing pan or tilt from motion
• CPC Class Codes: H04N 23/683 performed by a processor, e.g. controlling the readout of an image memory
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 5/2624 for obtaining an image which is composed of whole input images, e.g. splitscreen
• CPC Class Codes: H04N 5/2628 Alteration of picture size, shape, position or orientation, e.g. zooming, rotation, rolling, perspective, translation

Abstract

Various examples described herein are directed to systems and methods for stabilizing a panoramic video. The panoramic video may comprise a first frame captured by a panoramic camera. The panoramic camera may be positioned in a three-dimensional camera space described by an x-axis, a y-axis, and a z-axis that are mutually orthogonal. The first frame may comprise a plurality of pixel values arranged in a pixel plane and may be continuous in a first dimension. An image processor may receive from a motion sensor data describing a rotation of the panoramic camera at a time when the panoramic camera captured the first frame. The image processor may determine a planar component of an unintended rotation of the panoramic camera about the z-axis and convert the planar component of the unintended rotation to a first frame shift in the first dimension.

Claims

1. A method for stabilizing a 360°

panoramic video, the method comprising;

receiving, from a panoramic camera, video data comprising a plurality of frames, wherein the panoramic camera was positioned in a three-dimensional camera space described by an x-axis, a y-axis, and a z-axis that are mutually orthogonal, wherein receiving the video data comprises receiving first frame data of the plurality of frames that represents a 360°

field-of-view of the panoramic camera, the first frame data including a plurality of pixel values and spatial arrangement data describing positions for the plurality of pixel values on a pixel plane comprising a plurality of pixel value positions, and wherein pixel values positioned at a first vertical edge of the pixel plane and at a second vertical edge of the pixel plane opposite the first vertical edge depict adjacent portions of the three-dimensional camera space;

receiving, from a gyro sensor, rotation data of the panoramic camera in the three-dimensional camera space;

high-pass filtering the rotation data of the panoramic camera to generate unintended rotation data describing unintended rotation of the panoramic camera in the three-dimensional camera space;

using the unintended rotation data, determining a planar component of the unintended rotation, wherein the planar component of the unintended rotation is about the z-axis and in an xy plane;

for the first frame data from the plurality of frames of the video data, converting the planar component of the unintended rotation to a first frame shift in a horizontal dimension of the pixel plane; and

writing the first frame shift to a user data location associated with the first frame data.

2. The method of claim 1, further comprising:

receiving, from an accelerometer, velocity data indicating a velocity of the panoramic camera in the three-dimensional camera space;

high-pass filtering the velocity data to generate unintended velocity data indicating an unintended velocity of the panoramic camera;

using the unintended velocity data, determining a planar component of the unintended velocity of the panoramic camera in the xy plane;

identifying a first set of pixel values from the plurality of pixel values of the first frame data, wherein the spatial arrangement data describes a first set of pixel value positions for the first set of pixel values;

determining that a first object depicted by the first set of pixel values is a foreground object;

shifting the first set of pixel values from the first set of pixel value positions to a second set of pixel value positions to compensate for the planar component of the unintended velocity; and

write to the first set of pixel value positions, a second set of pixel values from a third set of pixel value positions at second frame data immediately after the first frame data in the panoramic video, wherein the third set of pixel value positions correspond to the first set of pixel value positions.

3. The method of claim 1, further comprising:

receiving from an accelerometer a velocity of the panoramic camera in the three-dimensional camera space;

high-pass filtering the velocity of the panoramic camera to find an unintended velocity of the panoramic camera;

determining an unintended rotation of the panoramic camera about the x-axis;

determining an unintended rotation of the panoramic camera about the y-axis;

determining an unintended velocity of the panoramic camera in a direction of the z-axis;

determining into a plurality of columns across the horizontal dimension of the pixel plane of the first frame data;

projecting the unintended rotation of the panoramic camera about the x-axis, the unintended rotation of the panoramic camera about the y-axis, and the unintended velocity of the panoramic camera in the direction of the z-axis to a first column of the plurality of columns;

cropping the first column in a vertical dimension of the pixel plane to compensate for the unintended rotation of the panoramic camera about the y-axis, and the unintended velocity of the panoramic camera in the direction of the z-axis;

projecting the unintended rotation of the panoramic camera about the x-axis, the unintended rotation of the panoramic camera about the y-axis, and the unintended velocity of the panoramic camera in the direction of the z-axis to a second column of the plurality of columns; and

cropping the second column in the vertical dimension to compensate for the unintended rotation of the panoramic camera about the y-axis, and the unintended velocity of the panoramic camera in the direction of the z-axis.

4. A system for stabilizing a panoramic video, the system comprising:

an image processor, the image processor comprising at least one processor and operatively associated memory, wherein the image processor is programmed to;

receive first frame data from the panoramic video, wherein the first frame data comprises a plurality of pixel values and spatial arrangement data describing positions for the plurality of pixel values on a pixel plane, and wherein the plurality of pixel values are continuous in a first dimension;

receive, from a motion sensor, data describing a rotation of at least one image sensor that captured the first frame data at a time when the at least one image sensor captured the first frame data;

high-pass filter the data describing the rotation resulting in unintended rotation data describing an unintended rotation of the at least one image sensor that captured the first frame data at the time when the at least one image sensor captured the first frame data;

determine a planar component of the unintended rotation about a first axis perpendicular to the first dimension of the pixel plane; and

determine, based at least in part on the planar component of the unintended rotation, a first frame data shift in the first dimension.

5. The system of claim 4, wherein the image processor is further programmed to generate playback instruction data for the first frame data, wherein the playback instruction data comprises the first frame data shift.

6. The system of claim 4, wherein the image processor is further programmed to shift the plurality of pixel values in the first dimension by the first frame data shift.

7. The system of claim 4, wherein determining the first frame data shift comprises projecting the planar component of the unintended rotation onto the first frame.

8. The system of claim 4, wherein projecting the planar component of the unintended rotation onto the first frame data comprises:

determining an opposite side length of an opposite side of a right triangle, wherein a hypotenuse the right triangle is a radius from the at least one image sensor to the first frame, and wherein the opposite side is opposite the planar component of the unintended rotation; and

applying a pixel conversion to the opposite side length to find the first frame data shift in pixel value positions.

9. The system of claim 4, wherein the image processor is further programmed to:

receive from an accelerometer data describing a velocity of the at least one image sensor at the time when the at least one image sensor captured the first frame data; and

high-pass filter the data describing the velocity resulting in an unintended velocity data describing an unintended velocity of the at least one image sensor at the time when the at least one image sensor captured the first frame data.

10. The system of claim 9, wherein the image processor is further programmed to:

determine a planar component of the unintended velocity of the at least one image sensor in a plane perpendicular to the pixel plane;

identify a set of pixel values from the plurality of pixel values, wherein the spatial arrangement data describes a first set of pixel value positions for the set of pixel values and wherein the set of pixel values depicts a first object;

determine that the first object is a foreground object; and

shift the set of pixel values from the first set of pixel value positions to a second set of pixel value positions to correct for the planar component of the unintended velocity.

11. The system of claim 10, wherein shifting the set of pixel values to the second set of pixel value positions comprises replacing an existing set of pixel values at the second set of pixel value positions with the first set of pixel values.

12. The system of claim 10, wherein the image processor is further programmed to:

add a new pixel value to the set of pixel values to change a size of the first object, wherein the new pixel value is described at the spatial arrangement data by a pixel value position adjacent the first set of pixel value positions.

13. The system of claim 10, wherein the image processor is further programmed to:

write to a first pixel value position from the first set of pixel value positions with a new pixel value.

14. The system of claim 13, wherein the image processor is further programmed to extrapolate the new pixel value from at least one adjacent pixel value of a pixel value position adjacent to the first pixel value position.

15. The system of claim 13, wherein the new pixel value is from a second pixel value position corresponding to the first pixel value position in a second frame adjacent to the first frame data in the panoramic video.

16. The system of claim 9, wherein the plurality of pixel values of the first frame data are discontinuous in a second dimension perpendicular to the first dimension, and wherein the image processor is further programmed to:

determine a second component of the unintended rotation about a second axis perpendicular to the first axis;

determine a third component of the unintended rotation about a third axis perpendicular to the first axis and the second axis;

determine a first component of the unintended velocity in a direction of the first axis;

determine a plurality of columns of the first frame across the first dimension of the pixel plane;

project the second component of the unintended rotation, the third component of the unintended rotation and the first component of the unintended velocity to a first column from the plurality of columns; and

crop the first column to correct for the second component of the unintended rotation, the third component of the unintended rotation and the first component of the unintended velocity.

17. A system for stabilizing a panoramic video, the system comprising:

an image processor, the image processor comprising at least one processor and operatively associated memory, wherein the image processor is programmed to;

receive a first frame from the panoramic video and captured by a panoramic camera positioned in a three-dimensional camera space described by an x-axis, a y-axis, and a z-axis that are mutually orthogonal, wherein the first frame comprises a plurality of pixel values and spatial arrangement data describing positions for the plurality of pixel values in a pixel plane, and wherein the first frame is continuous in a first dimension;

receiving, from a gyro sensor, data describing a rotation of the panoramic camera at a time when the panoramic camera captured the first frame;

high-pass filtering the data describing the rotation resulting in unintended rotation data describing an unintended rotation of the panoramic camera;

determine a planar component of the unintended rotation about the z-axis; and

determine, based at least in part on the planar component of the unintended rotation, a first frame shift in the first dimension.

18. The system of claim 17, wherein the image processor is further programmed to:

receive from an accelerometer data describing a velocity of the panoramic camera at the time when the panoramic camera captured the first frame; and

high-pass filter the velocity resulting in unintended velocity data describing an unintended velocity of the panoramic camera at the time when the panoramic camera captured the first frame.

19. The system of claim 18, wherein the image processor is further programmed to:

determine a planar component of the unintended velocity of the panoramic camera in an xy plane of the three-dimensional camera space;

identify a set of pixel values from the plurality of pixel values, wherein the spatial arrangement data describes a first set of pixel value positions for the set of pixel values, and wherein the set of pixel values depicts a first object;

determine that the first object is a foreground object; and

shift the set of pixel values from the first set of pixel value positions to a second set of pixel value positions to correct for the planar component of the unintended velocity.

20. The system of claim 18, wherein the plurality of pixel values of first frame data are discontinuous in a second dimension perpendicular to the first dimension, and wherein the image processor is further programmed to:

determine a second component of the unintended rotation about the x-axis;

determine a third component of the unintended rotation about the y-axis;

determine a first component of the unintended velocity in a direction of the z-axis;

determine a plurality of columns of the first frame across the first dimension of the pixel plane;

project the second component of the unintended rotation, the third component of the unintended rotation and the first component of the unintended velocity to a first column from the plurality of columns; and

crop the first column to correct for the second component of the unintended rotation, the third component of the unintended rotation and the first component of the unintended velocity.