Electronic image stabilization frequency estimator

• US 9,756,249 B1
• Filed: 04/27/2016
• Issued: 09/05/2017
• Est. Priority Date: 04/27/2016
• Status: Active Grant

• Current Assignee: GoPro, Inc.
• Original Assignee: GoPro, Inc.
• Inventors: Segapelli, Loic
• Primary Examiner(s): Ho, Tuan
• Assistant Examiner(s): Camargo, Marly
• Application Number:
  US15/140,391
• Time in Patent Office: 496 Days
• Field of Search: 34820899, 3482081, 3482082, 3482083, 3482084, 3482085, 348241, 348699-701
• CPC Class Codes: H04N 23/6812 based on additional sensors, e.g. acceleration sensors
• CPC Class Codes: H04N 23/6815 by distinguishing pan or tilt from motion
• CPC Class Codes: H04N 23/683 performed by a processor, e.g. controlling the readout of an image memory
• CPC Class Codes: H04N 23/685 performed by mechanical compensation
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 5/144 Movement detection for video coding H04N19/503; analysis of motion in general G06T7/20
• CPC Class Codes: H04N 5/21 Circuitry for suppressing or minimising disturbance, e.g. moiré or halo

Abstract

Disclosed are a system and a method for determining enabling or disabling electronic image stabilization (EIS) for a video frame. An image sensor of a camera system captures a video stream that comprises a plurality of video frames. An image processor determines availability of a computational resource that may process application of EIS on each video frame. Simultaneously, the image processor receives motion data of the camera system from a gyroscope. Based on the computational resource availability, a motion frequency threshold is determined. Based on the gyroscope motion data, a motion frequency of each video frame is estimated. The estimated motion frequency is compared to the determined motion frequency threshold. If the estimated motion frequency is greater than the determined motion frequency threshold, application of EIS is disabled. If the estimated motion frequency is less than or equal to the determined motion frequency threshold, application of EIS is enabled.

Claims

1. A method for controlling electronic image stabilization, the method comprising:

capturing a video stream;

determining an availability of a computational resource for performing the electronic image stabilization on one or more frames of the video stream;

receiving data from a gyroscope, the gyroscope connected to an imaging device, wherein the data from the gyroscope includes sensed motion of the imaging device;

determining a baseline motion frequency threshold based on the availability of the computational resource;

estimating a motion frequency of the imaging device during capture of the video stream from the data received from the gyroscope;

comparing the estimated motion frequency of the imaging device to the determined baseline motion frequency threshold;

performing, in response to the estimated motion frequency meeting a first criteria based on the baseline motion frequency threshold, the electronic image stabilization on the one or more frames of the video stream; and

disabling, in response to the estimated motion frequency meeting a second criteria based on the baseline motion frequency threshold, performing the electronic image stabilization on the one or more frames of the video stream.

2. The method of claim 1, wherein the gyroscope data further includes at least one of a positive displacement or a negative displacement, wherein the positive displacement indicates movement in a first direction and the negative displacement indicates movement in a second direction opposite the first direction.

3. The method of claim 2, further comprising:

determining a number of polarity changes in the sensed motion from the data from the gyroscope during capture of the one or more frames, wherein the polarity changes indicate at least one of a change in motion from the first direction to the second direction or a change in motion from the second direction to the first direction.

4. The method of claim 1, further comprising:

activating a switching element to enable or disable performing the electronic image stabilization on the one or more frames of the video stream.

5. The method of claim 1, wherein estimating the motion frequency further comprises:

determining, for each video frame of the video stream, a number of polarity changes in the gyroscope data indicating at least one of a change from a motion in first direction that exceeds a first noise threshold to a motion in a second direction that exceeds a second noise threshold or a change from a motion in the second direction that exceeds the second noise threshold to a motion in the first direction that exceeds the first noise threshold;

determining a total number of polarity changes over a window of video frames of the video stream;

determining a total time of capture of the window of the video frames of the video stream; and

estimating the motion frequency based on the total number of polarity changes and the total time of the window of the video frames of the video stream.

6. The method of claim 1, wherein the motion frequency is estimated on a sliding window of a plurality of video frames of the video stream.

7. The method of claim 1, further comprising:

determining a lower motion frequency threshold below the baseline motion frequency threshold and an upper motion frequency threshold above the baseline motion frequency threshold; and

determining that the estimated motion frequency meets the first criteria if the estimated motion frequency is below the lower motion frequency threshold and determining that the estimated motion frequency meets the second criteria if the estimated motion frequency is above the upper motion frequency threshold.

8. The method of claim 7, further comprising:

determining that the estimated motion frequency meets the first criteria if the estimated motion frequency is above the lower motion frequency threshold and below the upper motion frequency threshold and a last motion frequency met the first criteria; and

determining that the estimated motion frequency meets the second criteria if the estimated motion frequency is above the lower motion frequency threshold and below the upper motion frequency threshold and the last motion frequency met the second criteria.

9. The method of claim 1, further comprising:

determining a tapering motion frequency threshold based on the baseline motion frequency threshold;

determining that the estimated motion frequency meets the first criteria if the estimated motion frequency is below the tapering motion frequency threshold, determining that the estimated motion frequency meets the second criteria if the estimated motion frequency is above the baseline motion frequency threshold, and determining that the estimated motion frequency meets a third criteria if the estimation motion frequency is between the tapering motion frequency threshold and the baseline motion frequency threshold; and

responsive to the estimated motion frequency meeting the third criteria, applying a partial electronic image stabilization to the one or more frames, a stabilization level of the partial electronic image stabilization varying inversely to the estimated motion frequency.

10. A non-transitory computer readable medium configured to store instructions for controlling electronic image stabilization, the instructions when executed by a processor cause the processor to:

capture a video stream;

determine an availability of a computational resource, the computational resource to perform the electronic image stabilization on one or more frames of the video stream;

receive data from a gyroscope, the gyroscope connected to an imaging device, wherein the data from the gyroscope includes sensed motion of the imaging device;

determine a baseline motion frequency threshold based on the availability of the computational resource;

estimate a motion frequency of the imaging device during capture of the video stream from the data received from the gyroscope;

compare the estimated motion frequency of the imaging device to the determined baseline motion frequency threshold;

perform, in response to the estimated motion frequency meeting a first criteria based on the baseline motion frequency threshold, the electronic image stabilization on the one or more frames of the video stream; and

disable, in response to the comparison that the estimated motion frequency meeting a second criteria based on the baseline motion frequency threshold, performing the electronic image stabilization on the one or more frames of the video stream.

11. The non-transitory computer readable storage medium of claim 10, wherein the gyroscope data further includes at least one of a positive displacement or a negative displacement, wherein the positive displacement indicates movement in a first direction and the negative displacement indicates movement in a second direction opposite the first direction.

12. The non-transitory computer readable storage medium of claim 11, wherein the instructions further cause the processor to determine a number of polarity changes in the sensed motion from the data from the gyroscope during capture of the one or more frames, wherein the polarity changes indicate at least one of a change in motion from a first direction to a second direction or a change in motion from the second direction to the first direction.

13. The non-transitory computer readable storage medium of claim 10, wherein the instructions further cause the processor to activate a switching element to enable or disable performing the electronic image stabilization on the one or more frames of the video stream.

14. The non-transitory computer readable storage medium of claim 10, wherein estimating the motion frequency further comprises:

determining, for each video frame of the video stream, a number of polarity changes in the gyroscope data indicating at least one of a change from a motion in a first direction that exceeds a first noise threshold to a a motion in a second direction that exceeds a second noise threshold or a change from a motion in the second direction that exceeds the second noise threshold to a motion in the first direction that exceeds the first noise threshold;

determining a total number of polarity changes over a window of video frames of the video stream;

determining a total time of capture of the window of video frames of the video stream; and

estimating the motion frequency based on the total number of polarity changes and the total time of the window of the video frames of the video stream.

15. The non-transitory computer readable storage medium of claim 10, wherein the motion frequency is estimated on a sliding window of a plurality of video frames of the video stream.

16. The non-transitory computer readable storage medium of claim 10, wherein the instructions further cause the processor to:

determine a lower motion frequency threshold below the baseline motion frequency threshold and an upper motion frequency threshold above the baseline motion frequency threshold; and

determine that the estimated motion frequency meets the first criteria if the estimated motion frequency is below the lower motion frequency threshold and determining that the estimated motion frequency meets the second criteria if the estimated motion frequency is above the upper motion frequency threshold.

17. The non-transitory computer readable storage medium of claim 16, wherein the instructions further cause the processor to:

determine that the estimated motion frequency meets the first criteria if the estimated motion frequency is above the lower motion frequency threshold and below the upper motion frequency threshold and a last motion frequency met the first criteria; and

determine that the estimated motion frequency meets the second criteria if the estimated motion frequency is above the lower motion frequency threshold and below the upper motion frequency threshold and the last motion frequency met the second criteria.

18. The non-transitory computer readable storage medium of claim 10, wherein the instructions further cause the processor to:

determine a tapering motion frequency threshold based on the baseline motion frequency threshold;

determine that the estimated motion frequency meets the first criteria if the estimated motion frequency is below the tapering motion frequency threshold, determining that the estimated motion frequency meets the second criteria if the estimated motion frequency is above the baseline motion frequency threshold, and determining that the estimated motion frequency meets a third criteria if the estimation motion frequency is between the tapering motion frequency threshold and the baseline motion frequency threshold; and

responsive to the estimated motion frequency meeting the third criteria, apply a partial electronic image stabilization to the one or more frames, a stabilization level of the partial electronic image stabilization varying inversely to the estimated motion frequency.

19. A camera device comprising:

an imaging device to capture a video stream;

a gyroscope to sense motion of the imaging device;

a processor; and

a non-transitory computer readable medium configured to store instructions for controlling electronic image stabilization, the instructions when executed by the processor cause the processor to;

determine an availability of a computational resource, the computational resource to perform the electronic image stabilization on one or more frames of the video stream;

receive data from the gyroscope, the gyroscope connected to the imaging device, wherein the data from the gyroscope includes the sensed motion of the imaging device;

determine a baseline motion frequency threshold based on the availability of the computational resource;

estimate a motion frequency of the imaging device during capture of the video-from the data received from the gyroscope;

compare the estimated motion frequency of the imaging device to the determined baseline motion frequency threshold;

perform, in response to the estimated motion frequency meeting a first criteria based on the baseline motion frequency threshold, the electronic image stabilization on the one or more frames of the video stream; and

disable, in response to the comparison that the estimated motion frequency meeting a second criteria based on the baseline motion frequency threshold, performing the electronic image stabilization on the one or more frames of the video stream.

20. The camera device of claim 19, wherein estimating the motion frequency further comprises:

determining, for each video frame of the video stream, a number of polarity changes in the gyroscope data indicating at least one of a change from a motion in first direction that exceeds a first noise threshold to a motion in a second direction that exceeds a second noise threshold or a change from a motion in the second direction that exceeds the second noise threshold to a motion in the first direction that exceeds the first noise threshold;

determining a total number of polarity changes over a window of video frames of the video stream;

determining a total time of capture of the window of the video frames of the video stream; and

estimating the motion frequency based on the total number of polarity changes and the total time of the window of the video frames of the video stream.