Systems and methods for synchronizing sensor data

• US 10,506,163 B2
• Filed: 06/10/2016
• Issued: 12/10/2019
• Est. Priority Date: 06/10/2016
• Status: Active Grant

• Current Assignee: Movea
• Original Assignee: Invensense Incorporated (TDK Corporation)
• Inventors: de Foras, Etienne, Hall, Daniela
• Primary Examiner(s): Camargo, Marly S
• Application Number:
  US15/179,291
• Publication Number:
  US 20170359518A1
• Time in Patent Office: 1,278 Days
• Field of Search: 3482201, 34820899-20816, 3482191, 348154-155, 3482081, 348699, 396 13, 396 52- 55, 359225-557, 382107, 37524016, 3752401, 375145, 375149, 37524028, 375354-370, 37524029, 375343, 375350, 3752101, 375144, 702 89, 702187, 370241, 370248, 370304, 370324, 370350, 37039562, 370503, 370310, 370507, 370509
• CPC Class Codes: H04N 23/6812 based on additional sensors, e.g. acceleration sensors
• CPC Class Codes: H04N 23/687 by shifting the lens or sensor position
• CPC Class Codes: H04N 5/04 Synchronising for television systems using pulse code modulation H04N7/56

Abstract

A method for synchronizing data provided by an image sensor and a motion sensor rigidly linked together, comprising the steps of:

• • obtaining image sensor data and motion sensor data;
  • processing at least one of the image sensor data and the motion sensor data;
  • extracting motion characteristics from filtering said processed data, wherein said filtering comprises at least applying a high pass filter; and
    synchronizing the image sensor data and the motion sensor data provided by the rigidly linked sensors using synchronization data derived from identification of substantially similar motion characteristics in the filtered image sensor data and the filtered motion sensor data.

Claims

1. A method for synchronizing data provided by an image sensor and a motion sensor rigidly linked together, comprising the steps of:

obtaining image sensor data and motion sensor data, wherein the image sensor data is unsynchronized and the motion sensor data is unsynchronized; and

providing at least one processor configured to;

a) process the motion sensor data;

b) extract motion characteristics from the motion sensor data independently of the image sensor data by filtering said processed motion sensor data, wherein said filtering comprises at least applying a frequency filter;

c) extract motion characteristics from the image sensor data independently of the motion sensor data; and

d) synchronize the image sensor data and the motion sensor data provided by the rigidly linked sensors using synchronization data derived from matching the motion characteristics extracted from the image sensor data to the motion characteristics extracted from the motion sensor data and applying a correction based on the synchronization data to at least one of the image sensor data and the motion sensor data.

2. The method of claim 1, wherein the correction consists of applying a time delay to at least one of the image sensor data and the motion sensor data.

3. The method of claim 1, wherein the correction consists of modifying a time interval between at least two successive data samples from at least one of the image sensor data and the motion sensor data.

4. The method of claim 1, wherein the correction comprises an interpolation of one of the image sensor data and the motion sensor data.

5. The method of claim 1, wherein the image sensor data and the motion sensor data comprise timestamp data, and wherein the step of synchronizing consists in modifying the timestamp data for at least one of the image sensor data and the motion sensor data.

6. The method of claim 1, wherein the synchronization data is derived using a pattern matching of the motion characteristics of the image sensor data and the motion sensor data.

7. The method of claim 1, wherein the motion sensor has a plurality of measuring axes, and the step of processing processes the motion sensor data to determine a motion vector.

8. The method of claim 7, further comprising the step of determining a norm of the motion vector.

9. The method of claim 1, wherein the image sensor produces a plurality of image frames, and the step of processing of the image sensor data comprises determining displacement data related to points of interest in the image derived from a comparison of at least two image frames.

10. The method of claim 9, wherein a motion vector is derived based on the displacement data.

11. A sensor synchronization unit, comprising an image sensor and a motion sensor rigidly linked together, configured to:

obtain image data from the image sensor, wherein the image sensor data is unsynchronized;

obtain motion data from the motion sensor, wherein the motion sensor data is unsynchronized; and

use at least one processor to;

a) process the motion sensor data;

b) extract motion characteristics from the motion sensor data independently of the image sensor data by filtering said processed motion sensor data, wherein said filtering comprises at least applying a frequency filter;

c) extract motion characteristics from the image sensor data independently of the motion sensor data; and

d) synchronize the image sensor data and the motion sensor data provided by the rigidly linked sensors using synchronization data derived from matching the motion characteristics extracted from the image sensor data to the motion characteristics extracted from the motion sensor data and applying a correction based on the synchronization data to at least one of the image sensor data and the motion sensor data.

12. The sensor synchronization unit of claim 11, further configured to provide the synchronized data to an image stabilization system.

13. A system for synchronizing image sensor data and motion sensor data, comprising:

an image sensor;

a motion sensor rigidly linked to the image sensor; and

at least one processor configured to;

obtain image data from the image sensor, wherein the image sensor data is unsynchronized;

obtain motion data from the motion sensor, wherein the motion sensor data is unsynchronized;

process the motion sensor data;

filter the processed data, wherein said filtering comprises applying frequency filter;

extract motion characteristics from the motion sensor data independently of the image sensor data by filtering said processed motion sensor data;

extract motion characteristics from the image sensor data independently of the motion sensor data; and

synchronize the image sensor data and the motion sensor data provided by the rigidly linked sensors using synchronization data derived by matching the motion characteristics extracted from the image sensor data to the motion characteristics extracted from the motion sensor data and applying a correction based on the synchronization data to at least one of the image sensor data and the motion sensor data.

14. The system of claim 13, further comprising a memory for buffering at least one of the image sensor data and the motion sensor data.

15. The system of claim 13, wherein the motion sensor comprises a gyroscope.

16. The system of claim 13, wherein the motion sensor comprises a gyroscope and an accelerometer, and the motion data is based on an orientation determined based on fusion of at least the gyroscope data and accelerometer data.