Combined information for object detection and avoidance

• US 10,364,044 B2
• Filed: 03/06/2018
• Issued: 07/30/2019
• Est. Priority Date: 03/28/2016
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Harris, Scott Raymond, O'Brien, Barry James, Watson, Joshua John
• Primary Examiner(s): Rao, Anand S
• Application Number:
  US15/913,599
• Publication Number:
  US 20180194489A1
• Time in Patent Office: 511 Days
• Field of Search: 348143-159
• CPC Class Codes: B64C 39/024 of the remote controlled vehicle type, i.e. RPV
• CPC Class Codes: B64D 47/08 Arrangements of cameras
• CPC Class Codes: B64U 10/13 Flying platforms
• CPC Class Codes: B64U 2101/30 for imaging, photography or videography
• CPC Class Codes: B64U 2101/60 for transporting passengers; for transporting goods other than weapons
• CPC Class Codes: B64U 2201/104 using satellite radio beacon positioning systems, e.g. GPS
• CPC Class Codes: B64U 30/20 Rotors; Rotor supports
• CPC Class Codes: B64U 50/13 using external fans or propellers
• CPC Class Codes: B64U 50/19 using electrically powered motors
• CPC Class Codes: G05D 1/102 specially adapted for vertical take-off of aircraft
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/10048 Infrared image
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: G06V 10/16 using multiple overlapping images; Image stitching
• CPC Class Codes: G06V 20/13 Satellite images
• CPC Class Codes: G06V 20/17 taken from planes or by drones
• CPC Class Codes: G06V 20/194 using hyperspectral data, i.e. more or other wavelengths than RGB
• CPC Class Codes: G08G 5/045 Navigation or guidance aids, e.g. determination of anti-collision manoeuvers radar or analogous systems specially adapted for anti-collision between aircraft G01S13/933
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 13/25 : using two or more image sensors with different characteristics other than in their location or field of view, e.g. having different resolutions or colour pickup characteristics; using image signals from one sensor to control the characteristics of another sensor
• CPC Class Codes: H04N 13/271 : wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 5/33 : Transforming infrared radiation cameras or camera modules for generating image signals from infrared radiation H04N23/20; circuitry of SSIS for transforming infrared radiation into image signals H04N25/20

Abstract

Described is an imaging component for use by an unmanned aerial vehicle (“UAV”) for object detection. As described, the imaging component includes one or more cameras that are configured to obtain images of a scene using visible light that are converted into a depth map (e.g., stereo image) and one or more other cameras that are configured to form images, or thermograms, of the scene using infrared radiation (“IR”). The depth information and thermal information are combined to form a representation of the scene based on both depth and thermal information.

Claims

1. An unmanned aerial vehicle (“

UAV”

), comprising;

a frame;

a plurality of propulsion mechanisms to aerially lift and navigate the UAV;

an imaging component, including;

a first camera coupled to the frame and having a first orientation, wherein the first camera is configured to form a first image data of a scene using visible light;

a second camera coupled to the frame and having the first orientation, wherein the second camera is configured to form a second image data of the scene using visible light; and

a sensor coupled to the frame and having the first orientation, wherein the sensor is configured to form a sensor data representative of the scene; and

a processing component, configured to at least;

receive the first image data, the second image data, and the sensor data; and

process the first image data, the second image data, and the sensor data to produce a combined information representative of the scene.

2. The UAV of claim 1, wherein the combined information includes a horizontal dimension, a vertical dimension, and at least one of a depth dimension or a thermal dimension.

3. The UAV of claim 1, further comprising:

a controller configured to detect an object represented in the combined information, the controller configured to at least;

determine, based at least in part on the first image data, the second image data, and the sensor data, whether the object is an object to avoid; and

send instructions to a UAV control system to alter a navigation of the UAV in response to a determination that the object is an object to avoid.

4. The UAV of claim 1, wherein the sensor is at least one of an infrared sensor or an ultrasonic sensor.

5. The UAV of claim 1, wherein a first field of view of the first camera, a second field of view of the second camera, and a third field of view of the sensor overlap to form an effective field of view.

6. The UAV of claim 1, wherein the first orientation is downward with respect to the frame of the UAV and the scene includes an area toward which the UAV is descending.

7. A vision system of an aerial vehicle, comprising:

a first image capture device at a first position on the aerial vehicle, the first image capture device configured to form a first image data representative of at least a portion of a scene using a visible light spectrum;

a second image capture device at a second position on the aerial vehicle, the second image capture device configured to form a second image data representative of at least a portion of the scene using the visible light spectrum; and

a third image capture device at a third position on the aerial vehicle, the third image capture device configured to form a third image data representative of at least a portion of the scene using a light spectrum other than the visible light spectrum.

8. The vision system of claim 7, further comprising:

a processing component configured to at least;

receive the first image data and the second image data and form a depth map corresponding to at least a portion of the scene, wherein each position of the depth map corresponds to a vertical dimension, a horizontal dimension, and a depth dimension; and

receive the third image data and form a representation of the scene based on the third image data, wherein each position of the representation corresponds to a vertical dimension, a horizontal dimension, and a signature.

9. The vision system of claim 8, wherein the processing component is further configured to combine the depth map and the representation and form combined information representative of at least a portion of the scene.

10. The vision system of claim 7, wherein the first image capture device, the second image capture device and the third image capture device are positioned in a same plane.

11. The vision system of claim 7, wherein:

the scene includes a location to which the aerial vehicle is to descend.

12. The vision system of claim 7, wherein the first image data, the second image data, and the third image data are processed to detect a presence of a living object within an effective field of view.

13. The vision system of claim 12, wherein the effective field of view corresponds to at least a portion of the scene within a field of view of the first image capture device, the second image capture device, and the third image capture device.

14. A method to detect an object, the method comprising:

initiating, with an aerial vehicle, a descent toward a location;

obtaining combined information corresponding to a position, a depth, and a representation of each portion of the location within a field of view of an imaging component of the aerial vehicle, wherein each position of the combined information includes a horizontal dimension, a vertical dimension, a depth dimension, and a signature;

determining, based at least in part on the combined information, whether an object within the field of view corresponds to an object to avoid; and

altering the descent of the aerial vehicle based at least in part on a determination that the object corresponds to an object to avoid.

15. The method of claim 14, wherein:

the signature corresponds to a thermal dimension; and

determining that the object corresponds to an object to avoid includes determining that a thermal dimension of the object exceeds a threshold.

16. The method of claim 14, further comprising:

determining that a shape of the object and a signature of the object, as represented by the combined information, correspond to an object that is not to be avoided; and

continuing a descent of the aerial vehicle.

17. The method of claim 16, further comprising:

determining a movement of the object over a period of time; and

wherein determining whether the object corresponds to an object to avoid is further based at least in part on the movement.

18. The method of claim 16, wherein the determining the shape of the object and the signature of the object is based at least in part on stored object information.

19. The method of claim 14, further comprising:

determining that a shape of the object and a signature of the object, as represented by the combined information, correspond to a warm-blooded object; and

altering a descent of the aerial vehicle in response to the determination that the shape of the object and the signature of the object correspond to a warm-blooded object.

20. The method of claim 14, wherein the signature is a thermogram.