Method of stabilizing a sequence of images

• US 9,743,001 B1
• Filed: 02/19/2016
• Issued: 08/22/2017
• Est. Priority Date: 02/19/2016
• Status: Active Grant

• Current Assignee: FotoNation Limited (Adeia Inc.)
• Original Assignee: FotoNation Limited (Adeia Inc.)
• Inventors: Stec, Piotr
• Primary Examiner(s): Aggarwal, Yogesh
• Application Number:
  US15/048,149
• Publication Number:
  US 20170244881A1
• Time in Patent Office: 550 Days
• Field of Search: 34820899, 3482081-2088, 34820811
• CPC Class Codes: H04N 23/6812 based on additional sensors, e.g. acceleration sensors
• CPC Class Codes: H04N 23/683 performed by a processor, e.g. controlling the readout of an image memory
• CPC Class Codes: H04N 23/687 by shifting the lens or sensor position
• CPC Class Codes: H04N 23/689 Motion occurring during a rolling shutter mode

Abstract

A method operable within an image capture device for stabilizing a sequence of images captured by the image capture device is disclosed. The method comprises, using lens based sensors indicating image capture device movement during image acquisition, performing optical image stabilization (OIS) during acquisition of each image of the sequence of images to provide a sequence of OIS corrected images. Frame-to-frame movement of the device for each frame during which each OIS corrected image is captured is determined using inertial measurement sensors. At least an estimate of OIS control performed during acquisition of an image is obtained. The estimate is removed from the frame-to-frame movement determined for the frame during which the OIS corrected image was captured to provide a residual measurement of movement for the frame. Electronic image stabilization (EIS) of each OIS corrected image based on the residual measurement is performed to provide a stabilized sequence of images.

Claims

1. A method operable within an image capture device for stabilizing a sequence of images captured by the image capture device, comprising:

using lens based sensors indicating image capture device movement during image acquisition, performing optical image stabilization (OIS) during acquisition of each image of said sequence of images to provide a sequence of OIS corrected images;

using inertial measurement sensors, determining frame-to-frame movement of the device for each frame during which each OIS corrected image is captured;

obtaining at least an estimate of OIS controlled lens movement responsive to device movement during image acquisition with a lens system including an OIS controller providing, to a central processor of said image capture device, a record of OIS controlled lens movement applied during capture of each OIS corrected image;

removing said estimate from the frame-to-frame movement determined for the frame during which said OIS corrected image was captured to provide a residual measurement of movement for said frame wherein said removing comprises transforming said frame-to-frame movement according to a camera intrinsic matrix and multiplying an inverse of said record of OIS control with transformed frame-to-frame movement to provide said residual measurement of movement; and

performing electronic image stabilization (EIS) of each OIS corrected image based on said residual measurement to provide a stabilized sequence of images.

2. A method operable within an image capture device for stabilizing a sequence of images captured by the image capture device, comprising:

using lens based sensors indicating image capture device movement during image acquisition, performing optical image stabilization (OIS) during acquisition of each image of said sequence of images to provide a sequence of OIS corrected images;

using inertial measurement sensors, determining frame-to-frame movement of the device for each frame during which each OIS corrected image is captured;

obtaining at least an estimate of OIS controlled lens movement responsive to device movement during image acquisition said obtaining including comparing a pair of successively acquired OIS corrected images provided by an image sensor to provide an estimate of frame-to-frame movement between said OIS corrected images, and subtracting frame-to-frame movement of the device using inertial measurement sensors from said estimated frame-to-frame movement to provide said estimate of OIS controlled lens movement performed during acquisition of an image;

removing said estimate from the frame-to-frame movement determined for the frame during which said OIS corrected image was captured to provide a residual measurement of movement for said frame;

performing electronic image stabilization (EIS) of each OIS corrected image based on said residual measurement to provide a stabilized sequence of images;

prior to removing said estimate, subtracting a measurement of frame-to-frame lens rotation from said estimate of frame-to-frame movement; and

after removing said estimate, adding said measurement of frame-to-frame lens rotation to said residual measurement of movement for said frame.