Systems and methods for autonomous imaging and structural analysis

• US 9,734,397 B1
• Filed: 11/23/2016
• Issued: 08/15/2017
• Est. Priority Date: 11/04/2016
• Status: Active Grant

• Current Assignee: Loveland Innovations, Inc.
• Original Assignee: Loveland Innovations, LLC
• Inventors: Larson, Leif, Loveland, Jim, Christiansen, Dan, Christiansen, Tad, Christiansen, Cam
• Primary Examiner(s): Pham, Nam
• Application Number:
  US15/360,630
• Time in Patent Office: 265 Days
• CPC Class Codes: B64C 39/024 of the remote controlled vehicle type, i.e. RPV
• CPC Class Codes: B64D 47/08 Arrangements of cameras
• CPC Class Codes: B64U 10/13 Flying platforms
• CPC Class Codes: B64U 2101/30 for imaging, photography or videography
• CPC Class Codes: G01C 11/02 Picture taking arrangements specially adapted for photogrammetry or photographic surveying, e.g. controlling overlapping of pictures
• CPC Class Codes: G05D 1/0044 by providing the operator with a computer generated representation of the environment of the vehicle, e.g. virtual reality, maps maps used for automatic navigation G05D1/0274; flight directors G01C23/005
• CPC Class Codes: G05D 1/0088 characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours using knowledge based models G06N5/00
• CPC Class Codes: G05D 1/0094 involving pointing a payload, e.g. camera, weapon, sensor, towards a fixed or moving target
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06T 2207/10032 Satellite or aerial image; Remote sensing
• CPC Class Codes: G06T 2207/20101 Interactive definition of point of interest, landmark or seed
• CPC Class Codes: G06T 2207/20104 Interactive definition of region of interest [ROI]
• CPC Class Codes: G06T 2207/30108 Industrial image inspection
• CPC Class Codes: G06T 2207/30184 Infrastructure
• CPC Class Codes: G06T 7/0004 Industrial image inspection
• CPC Class Codes: G06T 7/30 Determination of transform parameters for the alignment of images, i.e. image registration
• CPC Class Codes: G06T 7/38 Registration of image sequences
• CPC Class Codes: G06V 20/176 Urban or other man-made structures
• CPC Class Codes: H04N 1/00204 with a digital computer or a digital computer system, e.g. an internet server programmed control between transmitter and receiver or between image input and image output device H04N1/32561
• CPC Class Codes: H04N 23/45 for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
• CPC Class Codes: H04N 23/62 : Control of parameters via user interfaces
• CPC Class Codes: H04N 23/66 : Remote control of cameras or camera parts, e.g. by remote control devices
• CPC Class Codes: H04N 7/185 : from a mobile camera, e.g. for remote control

Abstract

An unmanned aerial vehicle (UAV) assessment and reporting system may utilize one or more scanning techniques to provide useful assessments and/or reports for structures and other objects. The scanning techniques may be performed in sequence and optionally used to further fine tune each subsequent scan. The system may include shadow elimination, annotation, and/or reduction for the UAV itself and/or other objects. A UAV may be used to determine a pitch of roof of a structure. The pitch of the roof may be used to fine tune subsequent scanning and data capture.

Claims

1. An unmanned aerial vehicle (UAV) assessment system for imaging a structure and determining a pitch of a roof of a structure, comprising:

a site selection user interface to receive an electronic input from a user identifying a geographic location of a structure based on at least one of a street address, a coordinate, and a satellite image;

a UAV to receive the geographic location of the structure from the site selection interface, the UAV comprising;

a camera to capture images of the structure;

a pitch determination system to determine a pitch of a planar surface of the roof that is at an angle relative to a downward direction based on at least two distance measurements; and

an imaging system to;

adjust a tilt angle of the camera of the UAV to a non-zero angle relative to a downward direction to align an optical axis of the camera perpendicular to the planar surface of the roof of the structure, and

capture an image of at least a portion of the roof of the structure with the optical axis of the camera aligned perpendicular to the planar surface at each of the plurality of locations by adjusting a tilt angle of the camera of the UAV based on the determined pitch of the planar surface of the roof.

2. The UAV assessment system of claim 1, wherein the pitch determination system is configured to determine the pitch of the planar surface of the roof based on a rise over run determination in which a rise is equal to a vertical distance downward between a location of the UAV and the roof of the structure and in which the run is equal to a horizontal distance to the roof of the structure, such that the roof represents a hypotenuse of a right triangle with the UAV positioned at a 90 degree corner of the right triangle.

3. The UAV assessment system of claim 1, further comprising a boundary identification interface to receive electronic input identifying geographic boundaries associated with the location of the structure.

4. The UAV assessment system of claim 1, further comprising:

a processor in communication with the camera; and

a non-transitory computer-readable medium for receiving and storing instructions that, when executed by the processor, cause the UAV to conduct a structural assessment including;

a boustrophedonic scan of the structure that includes image capture during a boustrophedonic flight pattern within a first altitude range,

a loop scan of the structure that includes image capture during a second flight pattern for the UAV to travel around the perimeter of the structure at a second altitude range lower than the first altitude range, and

a micro scan of the structure that includes image capture in a third flight pattern that includes vertical approaches proximate the structure to capture detail images of the structure.

5. The UAV assessment system of claim 4, further comprising an interface for a user to identify a portion of interest on the roof of the structure, and wherein the micro scan includes at least one detail image of the portion of interest in which the optical axis of the camera is aligned perpendicular to the planar surface of the roof of the structure.

6. The UAV assessment system of claim 1, further comprising a hazard identification interface to receive an input from a user identifying at least one obstacle proximate the structure that is within the geographic boundaries.

7. The UAV assessment system of claim 1, wherein the imaging system is configured to capture the image of the roof of the structure with the optical axis of the camera aligned perpendicular to the planar surface of the roof of the structure by further adjusting the location of the UAV relative to the planar surface of the roof.

8. The UAV assessment system of claim 1, further comprising:

a processor in communication with the camera; and

a non-transitory computer-readable medium for receiving and storing instructions that, when executed by the processor, cause the UAV to conduct a structural assessment including;

a loop scan of the structure that includes image capture during a flight pattern for the UAV to travel around the perimeter of the structure, and

a micro scan of the structure that includes image capture in a flight pattern that includes vertical approaches proximate the structure to capture detail images of the structure.

9. The UAV assessment system of claim 1, further comprising:

a proximate object determination system to identify at least one object proximate the structure;

a shadow determination system to calculate a location of a shadow cast by the proximate object onto the structure based on a current location of the sun;

an imaging system to account for the shadow during imaging by implementing at least one of;

annotating images of the structure that include the calculated shadow,

adjusting an exposure of images of the structure that include the calculated shadow, and

identifying a subsequent time to return to the structure to capture non-shadowed images of the portions of the structure that are currently shadowed.

10. The UAV assessment system of claim 1, further comprising:

a shadow determination system to calculate a location of a shadow of the UAV on the structure based on the relative positions of the UAV and the sun; and

a shadow avoidance system to adjust a location of the UAV as it captures images of the structure to ensure that the shadow of the UAV is not in any of the images.

11. The UAV assessment system of claim 1, further comprising:

a proximate object determination system to identify at least one object proximate the structure;

a shadow determination system to calculate;

a location of a shadow cast by the proximate object onto the structure based on a current location of the sun, and

a location of a shadow of the UAV on the structure based on the relative positions of the UAV and the sun; and

a shadow avoidance system to;

re-capture images from different UAV positions to obtain images of the structure that do not include the shadow of the UAV, and

annotate images of the structure that include a shadow cast by the proximate object for subsequent imaging at a time when the shadow cast by the proximate object will not be present.

12. An unmanned aerial vehicle (UAV) for imaging a roof of a structure, comprising:

a camera to capture images of the structure;

a pitch determination system to determine a pitch of a planar surface of the roof that is at an angle relative to a downward direction in real time during a flight based on at least two distance measurements; and

an imaging system to capture an image of at least a portion of the roof of the structure during the same flight with an optical axis of the camera aligned perpendicular to the planar surface of the roof of the structure by adjusting a tilt angle of the camera of the UAV based on the determined pitch of the planar surface of the roof.

13. The UAV of claim 12, wherein at least one additional image of the roof captured with the optical axis misaligned relative to the planar surface of the roof is de-skewed using the determined pitch after image capture.

14. The UAV of claim 12, wherein the pitch determination system is configured to determine the pitch of the planar surface of the roof based on a rise over run determination in which a rise is equal to a vertical distance downward between a location of the UAV and the roof of the structure and in which the run is equal to a horizontal distance to the roof of the structure, such that the roof represents a hypotenuse of a right triangle with the UAV positioned at a 90-degree corner of the right triangle.

15. The UAV of claim 12, further comprising:

a processor in communication with the camera; and

a non-transitory computer-readable medium for receiving and storing instructions that, when executed by the processor, cause the UAV to conduct a structural assessment including;

a boustrophedonic scan of the structure that includes image capture during a boustrophedonic flight pattern within a first altitude range,

a loop scan of the structure that includes image capture during a second flight pattern for the UAV to travel around the perimeter of the structure at a second altitude range lower than the first altitude range, and

a micro scan of the structure that includes image capture in a third flight pattern that includes vertical approaches proximate the structure to capture detail images of the structure.

16. The UAV of claim 12, further comprising:

a processor in communication with the camera; and

a non-transitory computer-readable medium for receiving and storing instructions that, when executed by the processor, cause the UAV to conduct a structural assessment including a micro scan of the structure that includes image capture in a third flight pattern that includes vertical approaches proximate the structure to capture detail images of the structure.

17. The UAV of claim 16, further comprising an interface to enable a user to identify a portion of interest on the roof of the structure, and wherein the micro scan includes at least one detail image of the portion of interest in which the optical axis of the camera is aligned perpendicular to the planar surface of the roof of the structure by further adjusting a location of the UAV relative to the planar surface of the roof.

18. The UAV of claim 12, wherein the imaging system is configured to capture the image of the roof of the structure with the optical axis of the camera aligned perpendicular to the planar surface of the roof of the structure by further adjusting the location of the UAV relative to the planar surface of the roof.

19. The UAV of claim 12, further comprising a boundary identification interface to receive electronic input identifying geographic boundaries associated with the location of the structure.

20. A method for determining a pitch of a roof and imaging the same, comprising:

receiving, via a communications network, a relative location of a structure provided by a site selection user interface;

transmitting to a UAV the relative location of the structure as provided by the site selection user interface;

calculating, via a pitch determination system, a pitch of a planar surface of a roof of the structure during a flight of the UAV proximate the structure; and

capturing a plurality of images of at least a portion of the roof of the structure using an imaging system, including;

at least a first image with an optical axis of a camera aligned perpendicular to the planar surface of the roof of the structure by adjusting a tilt angle of the camera of the UAV, based on the calculated pitch of the planar surface of the roof; and

at least a second image with the optical axis of a camera misaligned with respect to the planar surface of the roof of the structure,

wherein the first image does not need de-skewing in at least one direction and the second image can be de-skewed in at least one direction using the calculated pitch of the planar surface of the roof.