Active alignment of stereo cameras

• US 10,499,037 B1
• Filed: 05/28/2015
• Issued: 12/03/2019
• Est. Priority Date: 05/28/2015
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Wilcox, Scott Michael
• Primary Examiner(s): Philippe, Gims S
• Assistant Examiner(s): Chang, Daniel
• Application Number:
  US14/724,669
• Time in Patent Office: 1,650 Days
• Field of Search: 348 47, 348117, 348144, 348147, 382104, 385 97, 356400, 356 401, 356496, 324 96, 702 39
• CPC Class Codes: G03B 15/006 Apparatus mounted on flying objects
• CPC Class Codes: G03B 17/561 Support related camera accessories
• CPC Class Codes: G03B 35/08 by simultaneous recording
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 13/246 Calibration of cameras
• CPC Class Codes: H04N 17/002 for television cameras
• CPC Class Codes: H04N 23/50 Constructional details
• CPC Class Codes: H04N 23/57 Mechanical or electrical details of cameras or camera modules specially adapted for being embedded in other devices

Abstract

In some examples, an unmanned aerial vehicle (UAV) may include a stereo camera including two cameras. To maintain proper alignment of the stereo camera as the UAV moves about, a management device may access calibration information for the stereo camera and receive sensing information indicating movement of the two cameras relative to each other. Based at least in part on the calibration information and the sensing information, the management device may instruct an actuator to move one of the two cameras to the proper alignment or may rectify frames captured by the two cameras to return to the proper alignment.

Claims

1. An unmanned aerial vehicle, comprising:

a frame;

a propulsion system comprising at least one propeller, the propulsion system connected to the frame;

a stereo camera array comprising;

a first camera connected to the frame and configured to capture a plurality of first image frames; and

a second camera connected to the frame via an actuator that extends between an exterior body of the second camera and the frame, the second camera configured to capture a plurality of second image frames;

a sensing system connected to the frame and external from the first camera and the second camera and in communication with a management device, the sensing system comprising;

an optical sensor device connected to the frame at a location between the first camera and the second camera, the optical sensor device configured to produce a first light at a first side of the optical sensor device and a second light at a second side of the optical sensor device, the optical sensor device comprising;

a first optical sensor disposed at the first side of the optical sensor device; and

a second optical sensor disposed at the second side of the optical sensor device;

a first reflecting device attached to the first camera and in optical communication with the optical sensor device, the first side of the optical sensor device oriented in a first direction toward the first reflecting device, the first reflecting device positioned to produce a first reflection from the first light; and

a second reflecting device attached to the second camera and in optical communication with the optical sensor device, the second side of the optical sensor device oriented in a second direction toward the second reflecting device, and the first direction being different from the second direction, the second reflecting device positioned to produce a second reflection from the second light;

the management device configured with at least a memory and a processor configured to;

access calibration information for the stereo camera array, the calibration information at least indicating a calibrated orientation for the first camera relative to the second camera;

receive, via the sensing system, position information that indicates at least one of the first camera or the second camera has changed position, the position information based at least in part on the first reflection and the second reflection, and the sensing system sensing at least a portion of the position information when the unmanned aerial vehicle is being moved by the at least one propeller of the propulsion system;

identify a change to the calibrated orientation of the first camera relative to the second camera based at least in part on the position information; and

instruct the actuator to at least displace the second camera at least until the first camera and the second camera are returned to the calibrated orientation.

2. The unmanned aerial vehicle of claim 1, wherein the change to the calibrated orientation is caused at least in part by one or more of vibration, deflection, linear extension, linear contraction, or twisting.

3. The unmanned aerial vehicle of claim 1, wherein the actuator comprises a piezoelectric linear actuator, and wherein instructing the actuator to at least displace the second camera into the calibrated orientation comprises providing an electrical signal to the piezoelectric linear actuator that causes a portion of the piezoelectric linear actuator to move the second camera from a first position to a second position, the second camera in the calibrated orientation when at the second position.

4. The unmanned aerial vehicle of claim 1, wherein the first camera is connected to the frame via a second actuator, the management device further configured to, in response to identifying the change to the calibrated orientation, instruct the second actuator to at least displace the first camera at least until the first camera and the second camera are returned to the calibrated orientation.

5. The unmanned aerial vehicle of claim 1, wherein the optical sensor device further comprises at least one of a laser sensor, a photoelectric sensor, or a depth sensor.

18. The unmanned aerial vehicle of claim 1, wherein:

the actuator is one of a plurality of actuators by which the second camera is connected to the frame; and

instructing the actuator to at least displace the second camera comprises instructing the plurality of actuators to displace the second camera into the calibrated orientation by adjusting a position of the second camera in at least two dimensions.

19. The unmanned aerial vehicle of claim 1, wherein the first direction is opposite the second direction.

6. A vehicle, comprising:

a frame;

at least one stereo camera configured to output image data, comprising;

a first camera connected to the frame via at least one actuator that extends between an exterior body of first camera and the frame, the first camera being oriented in a first orientation; and

a second camera connected to the frame and oriented in a second orientation;

at least one sensing device connected to the frame at a location between the first camera and the second camera, the at least one sensing device configured to produce a first light at a first side of the at least one sensing device and a second light at a second side of the at least one sensing device, the at least one sensing device comprising;

a first optical sensor disposed at the first side of the at least one sensing device; and

a second optical sensor disposed at the second side of the at least one sensing device;

a first reflecting device attached to the first camera and in optical communication with the at least one sensing device, the first side of the at least one sensing device oriented in a first direction toward the first reflecting device, the first reflecting device positioned to produce a first reflection from the first light; and

a second reflecting device attached to the second camera and in optical communication with the at least one sensing device, the second side of the at least one sensing device oriented in a second direction toward the second reflecting device, and the first direction being different from the second direction, the second reflecting device positioned to produce a second reflection from the second light; and

a management device configured with at least a memory and a processor configured to;

access calibration information for the at least one stereo camera, the calibration information at least indicating a calibration relationship between the first orientation and the second orientation;

receive, via the at least one sensing device, position information that indicates at least one of the first camera or the second camera has changed position, the position information being distinct from the image data and being based at least in part on the first reflection and the second reflection;

determine, based at least in part on the position information, a change in a relative orientation corresponding to a change in at least one of the first orientation or the second orientation;

determine that the change in the relative orientation exceeds a predetermined threshold; and

perform, based at least in part on determining that the change in the relative orientation exceeds the predetermined threshold, one or more adjustments to at least one of the first camera or the second camera to restore the calibration relationship between the first orientation and the second orientation.

7. The vehicle of claim 6, wherein performing the one or more adjustments comprises instructing the at least one actuator to move the first camera from a first position to a second position, the first camera, when in the second position, being substantially aligned with the first orientation.

8. The vehicle of claim 6, wherein performing the one or more adjustments to at least one of the first camera or the second camera further comprises:

identifying an object in a first image frame captured by the first camera at a first time, the object located at a first position in the first image frame at the first time;

identifying the object in a second image frame captured by the second camera at the first time, the object located at a second position in the second image frame at the first time, the first position distinct from the second position; and

adjusting, based at least in part on the position information and the calibration information, the first image frame such that the object in the first image frame is substantially aligned with the object in the second image frame.

9. The vehicle of claim 8, wherein the vehicle comprises a navigation system, and wherein the management device is further configured to provide the one or more adjustments to the navigation system.

10. The vehicle of claim 6, further comprising a propulsion system including at least one propeller, and wherein at least a portion of the position information is received when the vehicle is being moved by the at least one propeller of the propulsion system.

11. The vehicle of claim 6, wherein the at least one sensing device further comprises at least one of a laser sensor, a photoelectric sensor, a depth sensor, an acoustic sensor, or a flex sensor.

12. The vehicle of claim 6, wherein the one or more adjustments comprise at least one of rotation of the first camera relative to the second camera or translation of the first camera relative to the second camera.

13. The vehicle of claim 6, wherein the calibration relationship corresponds to a calibrated orientation for the at least one stereo camera, and wherein first image frames captured by the first camera are substantially aligned with second image frames captured by the second camera when the at least one stereo camera is aligned according to the calibrated orientation.

14. A computer-implemented method, comprising:

accessing calibration information for a stereo camera array of an unmanned aerial vehicle, the stereo camera array comprising a first camera and a second camera, the first camera configured to capture a first plurality of image frames, the second camera configured to capture a second plurality of image frames, and the calibration information at least indicating an initial orientation of the first camera relative to the second camera, image frames of the first plurality of image frames aligned with other image frames of the second plurality of image frames when the stereo camera array is oriented according to the initial orientation;

receiving, via a sensing system connected to a frame of an unmanned aerial vehicle, position information, the sensing system comprising;

an optical sensor device connected to the frame at a location between the first camera and the second camera, the optical sensor device configured to produce a first light at a first side of the optical sensor device and a second light at a second side of the optical sensor device, the optical sensor device comprising;

a first optical sensor disposed at the first side of the optical sensor device; and

a second optical sensor disposed at the second side of the optical sensor device;

a first reflecting device attached to the first camera and in optical communication with the optical sensor device, the first side of the optical sensor device oriented in a first direction toward the first reflecting device, the first reflecting device positioned to produce a first reflection from the first light, the first reflection corresponding to the position information; and

a second reflecting device attached to the second camera and in optical communication with the optical sensor device, the second side of the optical sensor device oriented in a second direction toward the second reflecting device, and the first direction being different from the second direction, the second reflecting device positioned to produce a second reflection from the second light, the second reflection corresponding to the position information;

determining a change to a position of the first camera or the second camera based at least in part on the position information;

identifying a misalignment of at least one image frame of the first plurality of image frames or the second plurality of image frames based at least in part on the change to the position of the first camera or the second camera;

determining that the change to the position of the first camera or the second camera exceeds a predetermined threshold;

determining an adjustment for the at least one image frame, based on at least the determination that the change to the position of the first camera or the second camera exceeds the predetermined threshold, to account for the misalignment; and

performing the adjustment on the at least one image frame.

15. The computer-implemented method of claim 14, wherein the at least one image frame was captured by the first camera at a first time, and wherein identifying the at least one image frame comprises:

identifying an object located at a first position at the first time in the at least one image frame; and

identifying the object at a second position at the first time in a second image frame captured by the second camera, the first position distinct from the second position.

16. The computer-implemented method of claim 15, wherein performing the adjustment on the at least one image frame comprises adjusting, based at least in part on the sensing information and the calibration information, the at least one image frame such that the object in the at least one image frame is substantially aligned with the object in the second image frame.

17. The computer-implemented method of claim 14, wherein the at least one image frame is captured by the first camera, and wherein performing the adjustment on the at least one image frame comprises one or more of:

performing planar rectification of the at least one image frame relative to a second image frame captured by the second camera;

performing cylindrical rectification of the at least one image frame relative to the second image frame;

or

performing polar rectification of the at least one image frame relative to the second image frame.