Multi-sensor image stabilization techniques

• US 10,594,937 B2
• Filed: 12/22/2018
• Issued: 03/17/2020
• Est. Priority Date: 06/08/2016
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Harris, Scott Raymond, Kamon, Ishay, Watson, Joshua John
• Primary Examiner(s): Hsu, Amy R
• Application Number:
  US16/231,448
• Publication Number:
  US 20190149735A1
• Time in Patent Office: 451 Days
• CPC Class Codes: G06V 20/13 Satellite images
• CPC Class Codes: G06V 20/17 taken from planes or by drones
• CPC Class Codes: H04N 23/45 for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
• CPC Class Codes: H04N 23/68 for stable pick-up of the scene, e.g. compensating for camera body vibrations
• CPC Class Codes: H04N 23/6811 based on the image signal
• CPC Class Codes: H04N 23/6812 based on additional sensors, e.g. acceleration sensors
• CPC Class Codes: H04N 23/687 by shifting the lens or sensor position
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums

Abstract

An aerial vehicle may include a first sensor, such as a digital camera, having a lens or other component that includes a second sensor mounted thereto. Information or data, such as digital images, captured using the second sensor may be used to determine or predict motion of the lens, which may include components of translational and/or rotational motion. Once the motion of the lens has been determined or predicted, such motion may be used to stabilize information or data, such as digital images, captured using the first sensor, according to optical or digital stabilization techniques. Where operations of the first sensor and the second sensor are synchronized, motion of the second sensor may be modeled based on information or data captured thereby, and imputed to the first sensor.

Claims

1. An aerial vehicle comprising:

a frame;

a primary imaging device joined to the frame, wherein the primary imaging device comprises a primary lens module;

a secondary imaging device joined to an external surface of the primary lens module, wherein the secondary imaging device comprises a secondary lens module; and

a control unit having at least one computer processor configured to at least;

cause a plurality of primary images to be captured by the primary imaging device;

cause a plurality of secondary images to be captured by the secondary imaging device;

detect motion of the secondary imaging device based at least in part on at least two of the secondary images; and

stabilize at least one of the primary images based at least in part on the motion of the secondary imaging device.

2. The aerial vehicle of claim 1, wherein the at least one computer processor is further configured to at least:

identify a salient feature in a first one of the secondary images; and

identify the salient feature in a second one of the secondary images,

wherein the motion of the secondary imaging device is detected based at least in part on the salient feature in the first one of the secondary images and the salient feature in the second one of the secondary images.

3. The aerial vehicle of claim 1, wherein the at least one computer processor is further configured to at least:

derive a model of the motion of the secondary imaging device based at least in part on at least some of the plurality of secondary images,

wherein the at least one vector is defined based at least in part on the model.

4. The aerial vehicle of claim 1, wherein an angular orientation of a lens of the primary lens module is oriented perpendicular to an angular orientation of a lens of the secondary lens module.

5. The aerial vehicle of claim 1, wherein the primary imaging device is configured to capture images at a first level of resolution,

wherein the secondary imaging device is configured to capture images at a second level of resolution, and

wherein the first level of resolution is greater than the second level of resolution.

6. A method comprising:

capturing, by a first sensor, first data during an operation of an aerial vehicle, wherein the first sensor comprises a first imaging device having a first lens module, wherein the first data comprises a first plurality of images, and wherein the first sensor is mounted to the aerial vehicle;

capturing, by a second sensor, second data during the operation of the aerial vehicle, wherein the second sensor comprises a second imaging device having a second lens module, wherein the second imaging device is joined to an external surface of the first imaging device, and wherein the second data comprises a second plurality of images captured by the second imaging device;

determining, by at least one computer processor, motion of the first sensor based at least in part on the second data, wherein the motion of the first sensor is determined based at least in part on at least two of the second plurality of images;

modifying, by the at least one computer processor, a first image of the first plurality of images based at least in part on the motion of the first sensor; and

storing the modified first image in at least one data store.

7. The method of claim 6, wherein the at least two of the second plurality of images comprises a second image and a third image, and

wherein modifying the first image comprises:

identifying a salient feature within the second image;

identifying the salient feature within the third image;

computing a transformation from the second image to the third image; and

modifying the first image or the second image based at least in part on the transformation.

8. The method of claim 6, wherein an angular orientation of the first lens module is perpendicular to an angular orientation of the second lens module.

9. The method of claim 8, wherein the angular orientation of the first lens module is aligned along or parallel to a yaw axis of the aerial vehicle, and

wherein the angular orientation of the second lens module is aligned along or parallel to a roll axis of the aerial vehicle.

10. The method of claim 6, wherein the first image was captured at a first time,

wherein the at least two of the second plurality of images comprises a second image captured prior to the first time and a third image captured after the first time, and

wherein modifying the first image comprises:

determining a first vector based at least in part on the second image and the third image; and

stabilizing the first image based at least in part on the first vector.

11. The method of claim 10, wherein the third image was captured prior to a second time,

wherein the at least two of the second plurality of images comprises a fourth image captured after the second time,

wherein the first plurality of images comprises a fifth image captured at the second time, and

wherein the method further comprises:

determining a second vector based at least in part on the third image and the fourth image; and

stabilizing the fifth image based at least in part on the second vector.

12. The method of claim 6, further comprising:

repositioning the first lens module based at least in part on the motion of the first sensor; and

after repositioning the first lens module,

capturing, by the first sensor, third data during the operation of the aerial vehicle,

wherein the third data comprises a second plurality of images.

13. The method of claim 6, wherein determining the motion of the first sensor comprises:

determining at least one of a forward velocity, a vertical velocity or a lateral velocity of the first sensor based at least in part on the second data.

14. The method of claim 6, wherein determining the motion of the first sensor comprises:

determining at least one of a yaw function, a pitch function or a roll function of the first sensor based at least in part on the second data.

15. The method of claim 6, wherein the first imaging device is configured to capture imaging data at a first level of resolution,

wherein the second sensor is a second imaging device configured to capture imaging data at a second level of resolution, and

wherein the first level of resolution is greater than the second level of resolution.

16. The method of claim 6, wherein the external surface of the first imaging device is an external surface of the first lens module of the first imaging device, and

wherein the second sensor further comprises at least one of a gyroscope or an accelerometer mounted to the first lens module of the first imaging device.

17. An imaging device coupled to an aerial vehicle, wherein the imaging device:

a first lens module, wherein the first lens module has an angular orientation aligned along or parallel to a yaw axis of the aerial vehicle;

a second lens module joined to the first lens module, wherein the second lens module has an angular orientation aligned along or parallel to a roll axis of the aerial vehicle;

a data store; and

at least one computer processor configured to at least;

capture a first plurality of images using the first lens module;

modeling motion of the second lens module based at least in part on at least the first plurality of images;

capture a second plurality of images using the second lens module;

stabilize a first image of the second plurality of images with respect to a second image of the second plurality of images based at least in part on the motion of the second lens module; and

store a stream of the second plurality of images in the data store, wherein the stream of images comprises the stabilized first image of the second plurality of images and the second image of the second plurality of images.

18. The imaging device of claim 17, wherein modeling the motion of the second lens module based at least in part on the first plurality of images comprises:

identifying a salient feature within a first image of the first plurality of images;

identifying the salient feature within a second image of the first plurality of images;

computing a transformation from the second image of the first plurality of images to the first image of the first plurality of images; and

modeling the motion of the second lens module based at least in part on the transformation.

19. The imaging device of claim 17, wherein the model represents at least one of:

a forward velocity function;

a vertical velocity function;

a lateral velocity function;

a yaw function;

a pitch function;

or

a roll function.

20. The imaging device of claim 17, wherein each of the first plurality of images has a first level of resolution,

wherein each of the second plurality of images has a second level of resolution, and

wherein the first level of resolution is greater than the second level of resolution.