Video Stabilization

• US 20190104255A1
• Filed: 10/03/2017
• Published: 04/04/2019
• Est. Priority Date: 10/03/2017
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Shi, Fuhao, Liang, Chia-Kai, Tu, Xue
• Granted Patent: US 10,462,370 B2
• Time in Patent Office: Days
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20201 Motion blur correction
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 5/73 Deblurring; Sharpening
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: H04N 23/6811 based on the image signal
• CPC Class Codes: H04N 23/6812 based on additional sensors, e.g. acceleration sensors
• CPC Class Codes: H04N 23/683 performed by a processor, e.g. controlling the readout of an image memory
• CPC Class Codes: H04N 23/687 by shifting the lens or sensor position

Abstract

Methods, systems, and apparatus, including computer programs stored on a computer-readable storage medium, for video stabilization. In some implementations, a computer system obtains frames of a video captured by a recording device using an optical image stabilization (OIS) system. The computing system receives (i) OIS position data indicating positions of the OIS system during capture of the frames, and (ii) device position data indicating positions of the recording device during capture of the frames. The computing system determines a first transformation for a particular frame based on the OIS position data for the particular frame and device position data for the particular frame. The computing system determines a second transformation for the particular frame based on the first transformation and positions of the recording device occurring after capture of the particular frame. The computing system generates a stabilized version of the particular frame using the second transformation.

Claims

1. A computer-implemented method, comprising:

receiving, by a computing system, a series of frames of a video captured by a recording device using an optical image stabilization (OIS) system;

receiving, by the computing system, (i) OIS position data indicating positions of the OIS system during capture of the series of frames and (ii) device position data indicating positions of the recording device during capture of the series of frames;

determining, by the computing system, a first transformation for a particular frame in the series of frames, the first transformation being determined based on the OIS position data for the particular frame and the device position data for the particular frame;

determining, by the computing system, a set of camera positions occurring over a set of multiple frames based on the OIS position data and the device position data, the set of multiple frames including one or more frames before the particular frame and one or more frames after the particular frame;

applying, by the computing system, a filter to the set of camera positions;

determining, by the computing system, a second transformation for the particular frame based on the first transformation and positions of the recording device determined, based on the device position data, for one or more frames in the series of frames that are captured after the particular frame, wherein the second transformation is determined based on a camera position determined based on applying the filter to the set of camera positions; and

generating, by the computing system, a stabilized version of the particular frame using the second transformation.

2. The computer-implemented method of claim 1, wherein receiving the OIS position data comprises receiving lens shift data indicating positions of a movable lens of the OIS system, wherein receiving the device position data comprises receiving orientation or movement data from a gyroscope or an accelerometer of the recording device.

3. The computer-implemented method of claim 1, wherein the OIS position data indicates multiple OIS position readings for each frame in the series of frames, wherein the device position data indicates multiple device position measurements for each frame in the series of frames.

4. The computer-implemented method of claim 1, further comprising determining a lens focus position corresponding to the particular frame, wherein the first transformation for the particular frame is determined using the lens focus position.

5. The computer-implemented method of claim 1, further comprising determining an image offset for the particular frame based on a position of a movable element of the OIS system as indicated by the OIS position data, wherein the first transformation for the particular frame is determined using the image offset.

7. The computer-implemented method of claim 1, further comprising:

determining an intermediate camera position for the particular frame from the output of applying the filter to the set of camera positions; and

comparing the intermediate camera position with a virtual camera position for a frame immediately before the particular frame wherein the second transformation is determined based on a camera position determined based on the comparison of the intermediate camera position with the virtual camera position for the frame immediately before the particular frame.

8. The computer-implemented method of claim 1, further comprising generating data characterizing an amount of motion blur in the particular frame, wherein the second transformation is determined based on a camera position determined using the data characterizing an amount of motion blur in the particular frame.

9. The computer-implemented method of claim 8, wherein generating data characterizing motion blur in the particular frame comprises estimating a motion blur kernel based on a projection matrix that determined for a beginning of exposure of the particular frame and a projection matrix for an end of exposure of the particular frame.

10. The computer-implemented method of claim 1, further comprising determining a camera position representing a position occurring during consistent movement of the recording device over a set of multiple frames, wherein the second transformation is determined based on the camera position representing a position occurring during consistent movement of the camera.

11. The computer-implemented method of claim 10, wherein the camera position is generated by applying a stable filter to data corresponding to a set of frames that includes one or more one or more frames before the particular frame and one or more frames after the particular frame.

12. The computer-implemented method of claim 10, further comprising:

determining a probability of consistent movement of the recording device based on analysis of camera positions over a set of frames that includes one or more frames before the particular frame and one or more frames after the particular frame; and

blending (i) the camera position representing a position occurring during consistent movement of the camera with (ii) a second camera position for the particular frame, wherein the camera positions are blended according to the determined probability, wherein the second transformation is determined based on a result of blending the camera position and the second camera position.

13. The computer-implemented method of claim 12, wherein:

the probability of consistent movement is determined using a linear regression over a set of camera positions that is sub-sampled from camera positions corresponding to the set of frames, and

the second camera position is determined using data indicating an amount of blur in the particular frame.

14. The computer-implemented method of claim 1, wherein determining the second transformation for the particular frame comprises:

determining, for each reference frame in a reference set including the particular frame and one or more frames occurring after the particular frame, a projection of image data of the frame based on the device position data and OIS position data corresponding to the frame;

determining a transformation for the particular frame that maps the projected image data of the particular frame to an output frame;

determining, for each reference frame in the reference set, whether the determined transformation for the particular frame defines each of the pixels of the output frame when applied to the reference frame;

identifying at least one reference frame for which application of the determined transformation to the projected image data does not define each of the pixels of the output frame;

identifying a minimal transition measure between a camera position determined for the particular frame and a camera position determined for the at least one reference frame; and

determining the second transformation for a camera position determined using the identified minimal transition measure, the second transformation being determined such that the second transformation maps image data to each of the pixels of an output frame when applied to the projection of each of the reference frames in the reference set.

15. The computer-implemented method of claim 1, further comprising:

determining a current transition measure for the particular frame, the current transition measure indicating a difference between a camera position determined for a frame immediately before the particular frame and a camera position determined for the particular frame;

determining a future transition measure for each of one or more future frames occurring after the particular frame, wherein each future transition measure indicates a difference between a camera position determined for the particular frame and a camera position determined for one of the future frames;

identifying, from among the current transition measure and the future transition measures, a largest transition measure indicating a largest difference in camera positions; and

determining a camera position based on the identified largest transition measure;

wherein the second mathematical transform is determined using the determined camera position.

16. The computer-implemented method of claim 15, wherein determining the camera position based on the identified largest transition measure comprises:

determining that the largest transition measure corresponds to a particular future frame of the one or more future frames;

adjusting a camera position for the particular frame toward the camera position for the particular future frame, wherein the camera position for the particular frame is adjusted in an amount that is scaled according to an amount of time between capture of the particular frame and capture of the particular future frame.

17. The computer-implemented method of claim 1, further comprising:

determining, for each of proper subset of scanlines of the particular frame, a mapping of the scanline with respect to an output frame or a virtual camera position for the scanline; and

determining a mapping for each scanline of the particular frame by interpolating between the determined mappings for the proper subset of scanlines or interpolating between the determined virtual camera positions for the proper subset of scanlines.

18. The computer-implemented method of claim 1, wherein the first transformation involves a first projection matrix for projecting image data for the particular frame to an intermediate space;

and

wherein the second transformation involves a second projection matrix for projecting the image data in the intermediate space to the stabilized version of the particular frame.

6. (canceled)

19. A recording device, comprising:

one or more processors; and

one or more non-transitory computer-readable devices, including instructions that when executed by the one or more processors, cause performance of operations that include;

receiving, by the one or more processors of the recording device, a series of frames of a video captured by a recording device using an optical image stabilization (OIS) system;

receiving, by the one or more processors of the recording device, (i) OIS position data indicating positions of the OIS system during capture of the series of frames, and (ii) device position data indicating positions of the recording device during capture of the series of frames;

determining, by the one or more processors of the recording device, a first transformation for a particular frame in the series of frames, the first transformation being determined based on the OIS position data for the particular frame and the device position data for the particular frame;

determining, by the one or more processors of the recording device, a set of camera positions occurring over a set of multiple frames based on the OIS position data and the device position data, the set of multiple frames including one or more frames before the particular frame and one or more frames after the particular frame;

applying, by the one or more processors of the recording device, a filter to the set of camera positions;

determining, by the one or more processors of the recording device, a second transformation for the particular frame based on the first transformation and positions of the recording device determined, based on the device position data, for one or more frames in the series of frames that are captured after the particular frame, wherein the second transformation is determined based on a camera position determined based on applying the filter to the set of camera positions; and

generating, by the one or more processors of the recording device, a stabilized version of the particular frame using the second transformation.

20. (canceled)

21. A computer-implemented method, comprising:

receiving a series of frames of a video captured by a recording device using an optical image stabilization (OIS) system;

receiving (i) OIS position data indicating positions of the OIS system during capture of the series of frames and (ii) device position data indicating positions of the recording device during capture of the series of frames;

determining a first transformation for a particular frame in the series of frames, the first transformation being determined based on the OIS position data for the particular frame and the device position data for the particular frame, wherein the first transformation involves a first projection matrix for projecting image data for the particular frame to an intermediate space;

determining a second transformation for the particular frame based on the first transformation and positions of the recording device determined, based on the device position data, for one or more frames in the series of frames that are captured after the particular frame, wherein the second transformation involves a second projection matrix for projecting the image data in the intermediate space to a stabilized version of the particular frame; and

generating a stabilized version of the particular frame using the second transformation.

22. A computer-implemented method, comprising:

receiving a series of frames of a video captured by a recording device using an optical image stabilization (OIS) system;

receiving (i) OIS position data indicating positions of the OIS system during capture of the series of frames and (ii) device position data indicating positions of the recording device during capture of the series of frames;

determining a first transformation for a particular frame in the series of frames, the first transformation being determined based on the OIS position data for the particular frame and the device position data for the particular frame;

determining a camera position representing a position occurring during consistent movement of the recording device over a set of multiple frames;

determining a second transformation for the particular frame based on the first transformation and positions of the recording device determined, based on the device position data, for one or more frames in the series of frames that are captured after the particular frame, wherein the second transformation is determined based on the camera position representing a position occurring during consistent movement of the camera; and

generating a stabilized version of the particular frame using the second transformation.