Combining depth and thermal information for object detection and avoidance

• US 9,963,246 B2
• Filed: 03/28/2016
• Issued: 05/08/2018
• Est. Priority Date: 03/28/2016
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Harris, Scott Raymond, O'Brien, Barry James, Watson, Joshua John
• Primary Examiner(s): Rao, Andy
• Application Number:
  US15/083,153
• Publication Number:
  US 20170275023A1
• Time in Patent Office: 771 Days
• Field of Search: 348143, 348144, 348148
• CPC Class Codes: B64C 39/024 of the remote controlled vehicle type, i.e. RPV
• CPC Class Codes: B64D 47/08 Arrangements of cameras
• CPC Class Codes: B64U 10/13 Flying platforms
• CPC Class Codes: B64U 2101/30 for imaging, photography or videography
• CPC Class Codes: B64U 2101/60 for transporting passengers; for transporting goods other than weapons
• CPC Class Codes: B64U 2201/104 using satellite radio beacon positioning systems, e.g. GPS
• CPC Class Codes: B64U 30/20 Rotors; Rotor supports
• CPC Class Codes: B64U 50/13 using external fans or propellers
• CPC Class Codes: B64U 50/19 using electrically powered motors
• CPC Class Codes: G05D 1/102 specially adapted for vertical take-off of aircraft
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/10048 Infrared image
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: G06V 10/16 using multiple overlapping images; Image stitching
• CPC Class Codes: G06V 20/13 Satellite images
• CPC Class Codes: G06V 20/17 taken from planes or by drones
• CPC Class Codes: G06V 20/194 using hyperspectral data, i.e. more or other wavelengths than RGB
• CPC Class Codes: G08G 5/045 Navigation or guidance aids, e.g. determination of anti-collision manoeuvers radar or analogous systems specially adapted for anti-collision between aircraft G01S13/933
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 13/25 : using two or more image sensors with different characteristics other than in their location or field of view, e.g. having different resolutions or colour pickup characteristics; using image signals from one sensor to control the characteristics of another sensor
• CPC Class Codes: H04N 13/271 : wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 5/33 : Transforming infrared radiation cameras or camera modules for generating image signals from infrared radiation H04N23/20; circuitry of SSIS for transforming infrared radiation into image signals H04N25/20

Abstract

Described is an imaging component for use by an unmanned aerial vehicle (“UAV”) for object detection. As described, the imaging component includes one or more cameras that are configured to obtain images of a scene using visible light that are converted into a depth map (e.g., stereo image) and one or more other cameras that are configured to form images, or thermograms, of the scene using infrared radiation (“IR”). The depth information and thermal information are combined to form a representation of the scene based on both depth and thermal information.

Claims

1. An unmanned aerial vehicle (“

UAV”

) comprising;

a frame;

a plurality of propulsion mechanisms to aerially lift and navigate the UAV;

an imaging component, including;

a first camera at a first position, wherein the first camera is configured to form a first image of a scene using visible light;

a second camera at a second position, wherein the first position and the second position are separated by a known baseline distance, wherein the second camera is configured to form a second image of the scene using visible light;

a third camera positioned approximately equidistant between the first camera and the second camera, wherein the third camera is configured to form a third image of the scene using infrared radiation;

an image-processing component, configured to at least;

receive the first image and the second image;

process the first image and the second image to form a depth map wherein each pixel representative of a position includes a horizontal dimension, a vertical dimension, and a depth dimension;

receive the third image;

process the third image to determine for each corresponding horizontal dimension and vertical dimension, a thermal dimension; and

generate combined information representative of the scene that includes for each pixel of the combined information the horizontal dimension, the vertical dimension, the depth dimension, and the thermal dimension.

2. The UAV of claim 1, wherein the first camera and the second camera form a stereo pair of cameras.

3. The UAV of claim 1, wherein the third camera is a long wave infrared camera.

4. The UAV of claim 1, wherein a first field of view of the first camera, a second field of view of the second camera, and a third field of view of the third camera overlap to form an effective field of view.

5. The UAV of claim 1, further comprising:

a controller configured to detect an object represented in the combined information;

determine, based at least in part on the depth dimension and the thermal dimension, whether the object is a warm-blooded object; and

send instructions to a UAV control system to alter a navigation of the UAV in response to a determination that the object is potentially a warm-blooded object.

6. An image capture apparatus, comprising:

a frame;

a first image capture device coupled to the frame at a first position;

a second image capture device coupled to the frame at a second position;

a third image capture device coupled to the frame at a third position;

wherein the first image capture device and the second image capture device have an overlapping field of view and are configured to form images of a scene using a visible light spectrum; and

wherein the third image capture device is positioned such that a field of view of the third image capture device comprises the overlapping field of view and the third image capture device is configured to form an image of the scene using an infrared spectrum.

7. The image capture apparatus of claim 6, further comprising:

a processing component configured to;

receive the first image and the second image and form a depth map corresponding to the scene, wherein each position of the depth map corresponds to a vertical dimension, a horizontal dimension, and a depth dimension; and

receive the third image and form a thermogram corresponding to the scene wherein each position of the thermogram corresponds to a vertical dimension, a horizontal dimension, and a thermal dimension.

8. The image capture apparatus of claim 7, wherein the processing component is further configured to combine the depth map and the thermogram and form combined information representative of the scene wherein each position corresponds to a horizontal dimension, a vertical dimension, a depth dimension, a thermal dimension, and original image information.

9. The image capture apparatus of claim 6, wherein the first image capture device, the second image capture device and the third image capture device are positioned in a same plane.

10. The image capture apparatus of claim 6, wherein the image capture apparatus is coupled to an unmanned aerial vehicle.

11. The image capture apparatus of claim 6, wherein:

the image capture apparatus is coupled to an unmanned aerial vehicle; and

the scene includes a delivery location to which the unmanned aerial vehicle is to descend.

12. The image capture apparatus of claim 6, wherein the first image, the second image, and the third image are processed to detect a presence of a warm-blooded object within an effective field of view.

13. The image capture apparatus of claim 12, wherein the effective field of view corresponds to a scene within a field of view of the first image capture device, the second image capture device, and the third image capture device.

14. A method to detect a warm-blooded object, the method comprising:

initiating with an unmanned aerial vehicle (“

UAV”

) a descent toward a delivery location;

obtaining combined information corresponding to a position, a depth, and a thermal representation of each portion of a delivery location within a field of view of an imaging component, wherein each pixel of the image includes a horizontal dimension, a vertical dimension, a depth dimension, and a thermal dimension;

determining, based at least in part on the combined information, whether an object within the field of view corresponds to a warm-blooded object; and

altering a descent of the aerial vehicle based at least in part on a determination that the object corresponds to a warm-blooded object.

15. The method of claim 14, wherein determining that the object corresponds to a warm-blooded object includes determining that a thermal dimension of the object exceeds a threshold.

16. The method of claim 14, further comprising:

determining that a shape of the object and a thermal dimension of the object, as represented by the combined information, corresponds to a non-warm-blooded object; and

continuing a descent of the aerial vehicle.

17. The method of claim 16, wherein the determination is further based at least in part on a movement of the object over a defined period of time.

18. The method of claim 16, wherein the determination that the shape of the object and the thermal dimension of the object correspond to a non-warm-blooded object is based at least in part on stored information corresponding to non-warm-blooded objects and warm-blooded objects.

19. The method of claim 14, further comprising:

determining that a shape of the object and a thermal dimension of the object, as represented by the image, correspond to a warm-blooded object; and

altering a descent of the aerial vehicle in response to the determination that the shape of the object and the thermal dimension of the object correspond to a warm-blooded object.

20. The method of claim 19, wherein the determination that the shape of the object and the thermal dimension of the object correspond to a warm-blooded object is based at least in part on stored information corresponding to non-warm-blooded objects and warm-blooded objects.