Multi-sensor image stabilization techniques

• US 10,212,346 B2
• Filed: 06/08/2016
• Issued: 02/19/2019
• Est. Priority Date: 06/08/2016
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Harris, Scott Raymond, Kamon, Ishay, Watson, Joshua John
• Primary Examiner(s): Hsu, Amy R
• Application Number:
  US15/177,194
• Publication Number:
  US 20170359515A1
• Time in Patent Office: 986 Days
• CPC Class Codes: G06F 18/00 Pattern recognition
• CPC Class Codes: G06V 20/13 Satellite images
• CPC Class Codes: G06V 20/17 taken from planes or by drones
• CPC Class Codes: H04N 23/45 for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
• CPC Class Codes: H04N 23/68 for stable pick-up of the scene, e.g. compensating for camera body vibrations
• CPC Class Codes: H04N 23/6811 based on the image signal
• CPC Class Codes: H04N 23/6812 based on additional sensors, e.g. acceleration sensors
• CPC Class Codes: H04N 23/687 by shifting the lens or sensor position
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums

Abstract

An aerial vehicle may include a first sensor, such as a digital camera, having a lens or other component that includes a second sensor mounted thereto. Information or data, such as digital images, captured using the second sensor may be used to determine or predict motion of the lens, which may include components of translational and/or rotational motion. Once the motion of the lens has been determined or predicted, such motion may be used to stabilize information or data, such as digital images, captured using the first sensor, according to optical or digital stabilization techniques. Where operations of the first sensor and the second sensor are synchronized, motion of the second sensor may be modeled based on information or data captured thereby, and imputed to the first sensor.

Claims

1. An unmanned aerial vehicle comprising:

a frame;

a primary imaging device joined to the frame, wherein the primary imaging device comprises a primary lens module;

a secondary imaging device joined to an external surface of the primary lens module, wherein the secondary imaging device comprises a secondary lens module; and

a control unit having at least one computer processor configured to at least;

cause a plurality of primary images to be captured by the primary imaging device;

cause a plurality of secondary images to be captured by the secondary imaging device;

define at least one vector representative of motion of the secondary imaging device based at least in part on the plurality of secondary images; and

stabilize at least one of the plurality of primary images based at least in part on the at least one vector representative of the motion of the secondary imaging device.

2. The unmanned aerial vehicle of claim 1, wherein the at least one computer processor is further configured to at least:

identify at least one salient feature in a first one of the secondary images; and

identify the at least one salient feature in a second one of the secondary images,

wherein the at least one vector representative of the motion of the secondary imaging device is defined based at least in part on the at least one salient feature in the first one of the secondary images and the at least one salient feature in the second one of the secondary images.

3. The unmanned aerial vehicle of claim 1, wherein the at least one vector is representative of at least one of a yaw, a pitch or a roll of the secondary imaging device.

4. The unmanned aerial vehicle of claim 1, wherein the primary imaging device is aligned in a substantially forward orientation when the unmanned aerial vehicle is engaged in forward flight operations, and

wherein the secondary imaging device is aligned in a substantially downward orientation when the unmanned aerial vehicle engaged in forward flight operations.

5. The unmanned aerial vehicle of claim 1, wherein the at least one computer processor is further configured to at least:

derive a model of the motion of the secondary imaging device based at least in part on at least some of the plurality of secondary images,

wherein the at least one vector is defined based at least in part on the model.

6. A method comprising:

capturing, by a first imaging device having a first lens, first imaging data during an operation of an aerial vehicle, wherein the first imaging device is mounted to the aerial vehicle;

capturing, by a second imaging device having a second lens, second imaging data during the operation of the aerial vehicle, wherein the second imaging device is mounted to at least a portion of the first imaging device;

determining, by at least one computer processor, at least one vector representative of motion of the first imaging device based at least in part on the second imaging data; and

modifying, by the at least one computer processor, the first data based at least in part on the at least one vector, wherein modifying the first imaging data comprises stabilizing the first imaging data by the at least one computer processor; and

storing the modified first imaging data in at least one data store.

7. The method of claim 6, wherein the first imaging data comprises a first image and a second image, and

wherein stabilizing the first imaging data comprises:

identifying at least one of a plurality of anchor points within the first image;

identifying at least one of the plurality of anchor points within the second image;

computing a transformation from the first image to the second image; and

modifying at least one of the first image or the second image based at least in part on the transformation.

8. The method of claim 6,

wherein the first imaging device is aligned in a substantially forward orientation with respect to a direction of travel of the aerial vehicle during the operation,

wherein the second imaging device is embedded in an underside of the first lens, and

wherein the second imaging device is aligned in a substantially downward orientation with respect to the direction of travel of the aerial vehicle during the operation.

9. The method of claim 6, wherein the first imaging data comprises a first plurality of images,

wherein the second imaging data comprises a second plurality of images, and

wherein determining the at least one vector representative of the motion of the first imaging device based at least in part on the second imaging data comprises:

determining a first vector based at least in part on a first subset of the second plurality of images, wherein the first subset comprises a predetermined number of the second plurality of images captured prior to a first time and the predetermined number of the second plurality of images captured following the first time;

identifying a first one of the first plurality of images captured at the first time,

wherein stabilizing the first imaging data comprises;

stabilizing the first one of the first plurality of images based at least in part on the first vector.

10. The method of claim 9, wherein determining the at least one vector representative of the motion of the first imaging device based at least in part on the second imaging data further comprises:

determining a second vector based at least in part on a second subset of the second plurality of images, wherein the second subset comprises a predetermined number of the second plurality of images captured prior to a second time and the predetermined number of the second plurality of images captured following the second time;

identifying a second one of the first plurality of images captured at the second time,

wherein stabilizing the first imaging data comprises;

stabilizing the second one of the first plurality of images based at least in part on the second vector.

11. The method of claim 6, wherein the second imaging data comprises a second plurality of images, and

wherein determining the at least one vector representative of the motion of the first imaging device based at least in part on the second data comprises:

identifying, by the at least one computer processor, at least a first position of at least one salient feature in a first one of the second plurality of images captured at a first time;

identifying, by the at least one computer processor, at least a second position of the at least one salient feature in a second one of the second plurality of images captured at a second time; and

determining the at least one vector based at least in part on the first position, the second position, the first time and the second time.

12. The method of claim 6, further comprising:

repositioning the first lens based at least in part on the at least one vector; and

after repositioning the first lens,

capturing, by the first imaging device, third data during the operation of the aerial vehicle.

13. The method of claim 6, wherein determining the at least one vector representative of the motion of the first imaging device comprises:

determining at least one of a forward velocity, a vertical velocity or a lateral velocity of the first imaging device based at least in part on the second imaging data.

14. The method of claim 6, wherein determining the at least one vector representative of the motion of the first imaging device comprises:

determining at least one of a yaw function, a pitch function or a roll function of the first imaging device based at least in part on the second imaging data.

15. The method of claim 6, wherein the first imaging device is configured to capture imaging data at a first level of resolution,

wherein the second imaging device is configured to capture imaging data at a second level of resolution, and

wherein the first level of resolution is greater than the second level of resolution.

16. An imaging device comprising:

a first lens;

a second lens joined to the first lens, wherein the second lens is aligned substantially perpendicular to the first lens;

a data store; and

at least one computer processor configured to at least;

capture a first plurality of images using the first lens, wherein the first plurality of images comprises a first image;

derive a model of motion of the second lens based at least in part on the plurality of images captured using the first lens;

determine a vector corresponding to the motion of the second lens at a first time based at least in part on the model;

capture a second plurality of images using the second lens, wherein the second plurality of images comprises a second image and a third image;

determine that the second image was captured at the first time;

stabilize the second image with respect to at least the third image based at least in part on the vector corresponding to the motion of the second lens at the first time; and

store a third plurality of images in the data store, wherein the stream of images comprises the stabilized second image and the third image.

17. The imaging device of claim 16, wherein the at least one computer processor is further configured to at least:

select a first predetermined number of the plurality of images captured using the first lens, wherein each of the first predetermined number of the plurality of images was captured prior to the first time;

select a second predetermined number of the plurality of images captured using the first lens, wherein each of the second predetermined number of the plurality of images was captured following the first time;

determine the vector corresponding to the motion of the second lens at the first time based at least in part on the first predetermined number of the plurality of images captured using the first lens and the second predetermined number of the first plurality of images captured using the first lens.

18. The imaging device of claim 16, wherein the vector represents at least one of:

a forward velocity function;

a vertical velocity function;

a lateral velocity function;

a yaw function;

a pitch function;

or

a roll function.

19. The imaging device of claim 16, wherein the at least one computer processor is further configured to at least:

identify at least one salient feature in the first image; and

identify the at least one salient feature in a fourth image, wherein the fourth image is one of the first plurality of images,

wherein the model of the motion of the second lens is derived based at least in part on the at least one salient feature in the first image and the at least one salient feature in the fourth image.

20. The imaging device of claim 16,

wherein the imaging device is coupled to an aerial vehicle,

wherein the first lens is aligned along or parallel to a yaw axis of the aerial vehicle, and

wherein the second lens is aligned along or parallel to a roll axis of the aerial vehicle.