Synchronization of frame rate to a detected cadence in a time lapse image sequence using sampling

• US 9,779,777 B2
• Filed: 12/16/2015
• Issued: 10/03/2017
• Est. Priority Date: 12/16/2015
• Status: Active Grant

• Current Assignee: GoPro, Inc.
• Original Assignee: GoPro, Inc.
• Inventors: Weckel, Charles, Bhat, Vishwanatha
• Primary Examiner(s): Jerabek, Kelly L
• Application Number:
  US14/970,869
• Publication Number:
  US 20170178689A1
• Time in Patent Office: 657 Days
• Field of Search: 34820899, 3482081-2082, 3482084
• CPC Class Codes: G11B 27/031 Electronic editing of digitised analogue information signals, e.g. audio or video signals
• CPC Class Codes: G11B 27/10 Indexing; Addressing; Timing or synchronising; Measuring tape travel
• CPC Class Codes: H04N 23/6812 based on additional sensors, e.g. acceleration sensors
• CPC Class Codes: H04N 23/683 performed by a processor, e.g. controlling the readout of an image memory
• CPC Class Codes: H04N 23/684 performed by controlling the image sensor readout, e.g. by controlling the integration time

Abstract

A frame rate is synchronized to a detected cadence in order to generate an output image sequence that is substantially stabilized. In an in-camera process, a camera receives motion data of the camera while the camera captures the sequence of image frames. A dominant frequency of motion is determined and the capture frame rate is dynamically adjusted to match the frequency of detected motion so that each image frame is captured when the camera is at approximately the same position along the axis of motion. Alternatively, in a post-processing process, frames of a captured image sequence are selectively sampled at a sampling rate corresponding to the dominant frequency of motion so that each sampled frame corresponds to an image capture that occurred when the camera is at approximately the same position along the axis of motion.

Claims

1. A method for synchronization of a frame rate to a detected cadence, the method comprising:

receiving a sequence of image frames having a first frame rate;

receiving motion data of a camera recorded by one or more sensors tracking motion of the camera while the camera captures the sequence of image frames;

converting the motion data in a particular window of time from time domain data to frequency domain data;

determining a dominant frequency having the highest magnitude value in the frequency domain data for the particular window of time;

sampling frames from the sequence of image frames in the particular window of time that are at a sampling frequency that is within a predefined tolerance of the dominant frequency; and

creating a new image sequence with the sampled frames.

2. The method of claim 1, wherein the particular window of time comprises a predefined time window.

3. The method of claim 1, wherein the motion data comprises position data along an axis of oscillation derived from sensed data captured by a motion sensor of the camera.

4. The method of claim 1, wherein the converting the motion data from time domain data to frequency domain data is performed using Fast Fourier Transform.

5. The method of claim 1, wherein the determining the dominant frequency in the frequency domain data comprises:

determining a frequency value in the frequency domain data that has the largest amplitude of the plurality of frequency values in the frequency domain data;

selecting the frequency value as the dominant frequency.

6. The method of claim 1, wherein sampling frames from the sequence of image frames comprises:

sampling frames from the sequence of image frames at a sampling frequency that matches the dominant frequency.

7. A non-transitory computer-readable storage medium storing instructions for synchronization of frame rate to cadence, the instructions when executed by one or more processors cause the one or more processors to:

receive a sequence of image frames having a first frame rate;

receive motion data of a camera recorded by one or more sensors tracking motion of the camera while the camera captures the sequence of image frames;

convert the motion data in a particular window of time from time domain data to frequency domain data;

determine a dominant frequency having the highest magnitude value in the frequency domain data for the particular window of time;

sample frames from the sequence of image frames in the particular window of time that are at a sampling frequency that is within a predefined tolerance of the dominant frequency; and

create a new image sequence with the sampled frames.

8. The non-transitory computer-readable storage medium of claim 7, wherein the particular window of time comprises a predefined time window.

9. The non-transitory computer-readable storage medium of claim 7, wherein the motion data comprises position data along an axis of oscillation derived from sensed data captured by a motion sensor.

10. The non-transitory computer-readable storage medium of claim 7, wherein the conversion of the motion data from time domain data to frequency domain data is performed using Fast Fourier Transform.

11. The non-transitory computer-readable storage medium of claim 7, wherein the computer-readable storage medium stores further instructions for the determination of the dominant frequency in the frequency domain data, the instructions when executed by the one or more processors cause the one or more processors to:

determine a frequency value in the frequency domain data that has the largest amplitude of the plurality of frequency values in the frequency domain data;

select the frequency value as the dominant frequency.

12. The non-transitory computer-readable storage medium of claim 7, wherein the computer-readable storage medium stores further instructions for the sampling of frames from the sequence of image frames, the instructions when executed by the one or more processors cause the one or more processors to:

sample frames from the sequence of image frames at a sampling frequency that matches the dominant frequency.

13. A computing device, comprising:

one or more processors; and

a non-transitory computer-readable storage medium storing instructions, the instructions when executed by one or more processors causing the one or more processors to perform steps including;

receiving a sequence of image frames having a first frame rate;

receiving motion data of a camera recorded by one or more sensors tracking motion of the camera while the camera captures the sequence of image frames;

converting the motion data in a particular window of time from time domain data to frequency domain data;

determining a dominant frequency having the highest magnitude value in the frequency domain data for the particular window of time;

sampling frames from the sequence of image frames in the particular window of time that are at a sampling frequency that is within a predefined tolerance of the dominant frequency; and

creating a new image sequence with the sampled frames.

14. The device of claim 13, wherein the particular window of time comprises a predefined time window.

15. The device of claim 13, wherein the motion data comprises position data along an axis of oscillation derived from sensed data captured by a motion sensor.

16. The device of claim 13, wherein the conversion of the motion data from time domain data to frequency domain data is performed using Fast Fourier Transform.

17. The device of claim 13, wherein the computer-readable storage medium stores further instructions for the determination of the dominant frequency in the frequency domain data from the particular window of time, the instructions when executed by the one or more processors causing the processors to perform steps including:

determining a frequency value in the frequency domain data from the particular window of time that has the largest amplitude of the plurality of frequency values in the frequency domain data;

selecting the frequency value as the dominant frequency.

18. The device of claim 13, wherein the computer-readable storage medium stores further instructions for the sampling of the frames from the time lapse image sequence, the instructions when executed by the one or more processors causing the processors to perform steps including:

sampling frames from the sequence of image frames at a sampling frequency that matches the dominant frequency.