Electronic image stabilization frequency estimator

• US 10,506,162 B2
• Filed: 11/20/2018
• Issued: 12/10/2019
• Est. Priority Date: 04/27/2016
• Status: Active Grant

• Current Assignee: GoPro, Inc.
• Original Assignee: GoPro, Inc.
• Inventors: Segapelli, Loic
• Primary Examiner(s): Camargo, Marly S
• Application Number:
  US16/195,991
• Publication Number:
  US 20190109990A1
• Time in Patent Office: 385 Days
• Field of Search: 34820899, 3482081, 3482082, 3482083, 3482084, 3482085, 348241, 348699-701, 382107, 382236
• CPC Class Codes: H04N 23/6812 based on additional sensors, e.g. acceleration sensors
• CPC Class Codes: H04N 23/6815 by distinguishing pan or tilt from motion
• CPC Class Codes: H04N 23/683 performed by a processor, e.g. controlling the readout of an image memory
• CPC Class Codes: H04N 23/685 performed by mechanical compensation
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 5/144 Movement detection for video coding H04N19/503; analysis of motion in general G06T7/20
• CPC Class Codes: H04N 5/21 Circuitry for suppressing or minimising disturbance, e.g. moiré or halo

Abstract

Disclosed are a system and a method for providing electronic image stabilization (EIS). The method comprises estimating a motion frequency of an imaging device during capture of media using data received from the imaging device, comparing the estimated motion frequency of the imaging device to a baseline motion frequency threshold, and performing, in response to the estimated motion frequency meeting a first criteria based on the baseline motion frequency threshold, electronic image stabilization on the media.

Claims

1. A method, comprising:

estimating a motion frequency of an imaging device during capture of media using data received from the imaging device;

comparing the estimated motion frequency of the imaging device to a baseline motion frequency threshold; and

performing, in response to the estimated motion frequency meeting a first criteria based on the baseline motion frequency threshold, electronic image stabilization on the media.

2. The method of claim 1, wherein the data is received from a gyroscope of the imaging device and includes sensed motion of the imaging device.

3. The method of claim 2, wherein the data includes positive displacement and a negative displacement, the positive displacement indicating movement in a first direction, the negative displacement indicating movement in a second direction opposite the first direction.

4. The method of claim 3, further comprising:

determining a number of polarity changes in the sensed motion, the number of polarity changes indicating any of a change in motion from the first direction to the second direction and a change in motion from the second direction to the first direction.

5. The method of claim 1, wherein the media comprises a video stream with a plurality of video frames, and wherein estimating the motion frequency further comprises:

determining, for each video frame of the video stream, a number of polarity changes in the data indicating any of a change from a motion in a first direction that exceeds a first noise threshold to a motion in a second direction that exceeds a second noise threshold and a change from a motion in the second direction that exceeds the second noise threshold to a motion in the first direction that exceeds the first noise threshold;

determining a total number of polarity changes over a window of video frames of the video stream;

determining a total time of capture of the window of video frames; and

estimating the motion frequency based on the total number of polarity changes and the total time of capture.

6. The method of claim 1, wherein the media comprises a video stream with a plurality of video frames, and wherein estimating the motion frequency further comprises:

estimating the motion frequency on a sliding window of the plurality of video frames.

7. The method of claim 1, further comprising:

determining a lower motion frequency threshold below the baseline motion frequency threshold and an upper motion frequency threshold above the baseline motion frequency threshold; and

determining that the estimated motion frequency meets the first criteria if the estimated motion frequency is below the lower motion frequency threshold and determining that the estimated motion frequency meets a second criteria if the estimated motion frequency is above the upper motion frequency threshold.

8. The method of claim 7, further comprising:

determining that the estimated motion frequency meets the first criteria if the estimated motion frequency is above the lower motion frequency threshold and below the upper motion frequency threshold and if a last motion frequency met the first criteria; and

determining that the estimated motion frequency meets the second criteria if the estimated motion frequency is above the lower motion frequency threshold and below the upper motion frequency threshold and if the last motion frequency met the second criteria.

9. The method of claim 1, further comprising:

determining a tapering motion frequency threshold based on the baseline motion frequency threshold;

determining that the estimated motion frequency meets the first criteria if the estimated motion frequency is below the tapering motion frequency threshold, determining that the estimated motion frequency meets a second criteria if the estimated motion frequency is above the baseline motion frequency threshold, and determining that the estimated motion frequency meets a third criteria if the estimated motion frequency is between the tapering motion frequency threshold and the baseline motion frequency threshold; and

responsive to the estimated motion frequency meeting the third criteria, applying a partial electronic image stabilization to the media, a stabilization level of the partial electronic image stabilization varying inversely to the estimated motion frequency.

10. A non-transitory computer readable storage medium configured to store instructions, the instructions when executed by a processor cause the processor to perform steps including:

estimating a motion frequency of an imaging device during capture of media using data received from the imaging device;

comparing the estimated motion frequency of the imaging device to a baseline motion frequency threshold; and

performing, in response to the estimated motion frequency meeting a first criteria based on the baseline motion frequency threshold, electronic image stabilization on the media.

11. The non-transitory computer readable storage medium of claim 10, wherein the data is received from a gyroscope of the imaging device and includes any of sensed motion of the imaging device, a positive displacement, and a negative displacement, the positive displacement indicating movement in a first direction and the negative displacement indicating movement in a second direction opposite the first direction.

12. The non-transitory computer readable storage medium of claim 11, wherein the instructions further cause the processor to:

determine a number of polarity changes in the sensed motion, the number of polarity changes indicating any of a change in motion from the first direction to the second direction and a change in motion from the second direction to the first direction.

13. The non-transitory computer readable storage medium of claim 10, wherein the media comprises a video stream with a plurality of video frames, and wherein estimating the motion frequency further comprises:

determining, for each video frame of the video stream, a number of polarity changes in the data indicating any of a change from a motion in a first direction that exceeds a first noise threshold to a motion in a second direction that exceeds a second noise threshold and a change from a motion in the second direction that exceeds the second noise threshold to a motion in the first direction that exceeds the first noise threshold;

determining a total number of polarity changes over a window of video frames of the video stream;

determining a total time of capture of the window of video frames; and

estimating the motion frequency based on the total number of polarity changes and the total time of capture.

14. The non-transitory computer readable storage medium of claim 10, wherein the instructions further cause the processor to perform steps including:

determining a lower motion frequency threshold below the baseline motion frequency threshold and an upper motion frequency threshold above the baseline motion frequency threshold; and

determining that the estimated motion frequency meets the first criteria if the estimated motion frequency is below the lower motion frequency threshold and determining that the estimated motion frequency meets a second criteria if the estimated motion frequency is above the upper motion frequency threshold.

15. A camera device, comprising:

a processor; and

a memory storing an application that when executed by the processor causes the processor to;

estimate a motion frequency of the camera device during capture of media using data received from the camera device;

compare the estimated motion frequency of the camera device to a baseline motion frequency threshold; and

perform, in response to the estimated motion frequency meeting a first criteria based on the baseline motion frequency threshold, electronic image stabilization on the media.

16. The camera device of claim 15, further comprising:

a gyroscope, wherein the data is received from the gyroscope, further wherein the data includes any of sensed motion of the camera device, a positive displacement, and a negative displacement, the positive displacement indicating movement in a first direction and the negative displacement indicating movement in a second direction opposite the first direction.

17. The camera device of claim 16, wherein the application when executed further causes the processor to:

determine a number of polarity changes in the sensed motion, the number of polarity changes indicating any of a change in motion from the first direction to the second direction and a change in motion from the second direction to the first direction.

18. The camera device of claim 15, wherein the media comprises a video stream with a plurality of video frames, and wherein to estimate the motion frequency further comprises to:

determining, for each video frame of the video stream, a number of polarity changes in the data indicating any of a change from a motion in a first direction that exceeds a first noise threshold to a motion in a second direction that exceeds a second noise threshold and a change from a motion in the second direction that exceeds the second noise threshold to a motion in the first direction that exceeds the first noise threshold;

determining a total number of polarity changes over a window of video frames of the video stream;

determining a total time of capture of the window of video frames; and

estimating the motion frequency based on the total number of polarity changes and the total time of capture.

19. The camera device of claim 15, wherein the media comprises a video stream with a plurality of video frames, and wherein to estimate the motion frequency further comprises to:

estimate the motion frequency on a sliding window of the plurality of video frames.

20. The camera device of claim 15, wherein the application when executed further causes the processor to:

determine a lower motion frequency threshold below the baseline motion frequency threshold and an upper motion frequency threshold above the baseline motion frequency threshold; and

determine that the estimated motion frequency meets the first criteria if the estimated motion frequency is below the lower motion frequency threshold and determine that the estimated motion frequency meets a second criteria if the estimated motion frequency is above the upper motion frequency threshold.