System and method for structural inspection and construction estimation using an unmanned aerial vehicle

• US 10,217,207 B2
• Filed: 01/20/2017
• Issued: 02/26/2019
• Est. Priority Date: 01/20/2016
• Status: Expired due to Fees

• Current Assignee: EZ3D Technologies, Inc.
• Original Assignee: EZ3D LLC
• Inventors: Marra, Martin, Smyth, James F.
• Primary Examiner(s): Perungavoor, Sathyanaraya V
• Assistant Examiner(s): Brown, Jr., Howard D
• Application Number:
  US15/411,182
• Publication Number:
  US 20170206648A1
• Time in Patent Office: 767 Days
• CPC Class Codes: B64U 10/13 Flying platforms
• CPC Class Codes: B64U 2101/00 UAVs specially adapted for particular uses or applications
• CPC Class Codes: B64U 2101/26 for manufacturing, inspections or repairs
• CPC Class Codes: B64U 2101/30 for imaging, photography or videography
• CPC Class Codes: B64U 2201/20 Remote controls
• CPC Class Codes: G01C 11/00 Photogrammetry or videogrammetry, e.g. stereogrammetry; Photographic surveying
• CPC Class Codes: G01C 21/20 Instruments for performing navigational calculations G01C21/24, G01C21/26 take precedence
• CPC Class Codes: G05D 1/0038 by providing the operator with simple or augmented images from one or more cameras located onboard the vehicle, e.g. tele-operation images analyzed by a computer and used for automatic navigation G05D1/0246
• CPC Class Codes: G05D 1/0094 involving pointing a payload, e.g. camera, weapon, sensor, towards a fixed or moving target
• CPC Class Codes: G06T 19/00 Manipulating 3D models or images for computer graphics
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2200/08 involving all processing steps from image acquisition to 3D model generation
• CPC Class Codes: G06T 2207/10032 Satellite or aerial image; Remote sensing
• CPC Class Codes: G06T 2207/30108 Industrial image inspection
• CPC Class Codes: G06T 2215/16 Using real world measurements to influence rendering
• CPC Class Codes: G06T 7/0004 Industrial image inspection
• CPC Class Codes: G06V 20/17 taken from planes or by drones
• CPC Class Codes: G06V 20/176 Urban or other man-made structures
• CPC Class Codes: G08G 5/0034 Assembly of a flight plan
• CPC Class Codes: H04N 13/204 using stereoscopic image cameras stereoscopic photography G03B35/00

Abstract

An automated image capturing and processing system and method may allow a field user to operate a UAV via a mobile computing device to capture images of a structure area of interest (AOI). The mobile computing device receives user and/or third party data and creates UAV control data and a flight plan. The mobile computing device executes a flight plan by issuing commands to the UAV'"'"'s flight and camera controller that allows for complete coverage of the structure AOI.

After data acquisition, the mobile computing device then transmits the UAV output data to a server for further processing. At the server, the UAV output data can be used for a three-dimensional reconstruction process. The server then generates a vector model from the images that precisely represents the dimensions of the structure. The server can then generate a report for inspection and construction estimation.

Claims

1. An image and information capturing and processing system, comprising:

a mobile computing device configured to;

receive user input data and/or third party data at the mobile computing device;

create unmanned aerial vehicle control data based at least in part on the user input data and/or the third party data;

create a flight plan based at least in part on the unmanned aerial vehicle control data comprising a generally crude outline of a structure area of interest to insure images and data capturing are taken at optimal distances and intervals for three-dimensional reconstruction and visual inspection;

transmit the flight plan to an unmanned aerial vehicle via a communication link;

execute the flight plan at least in part by issuing commands to flight and camera controllers of the unmanned aerial vehicle, wherein the commands comprise an orbit at calculated ranges with a specified minimum depression angle to insure complete image coverage of the structure area of interest from each perspective, omnidirectional orbital imaging capable of reducing obstructions for inspection and three-dimensional reconstruction of the structure of interest in order to allow three-dimensional point cloud reconstruction;

receive unmanned aerial vehicle output data from the unmanned aerial vehicle; and

transmit the unmanned aerial vehicle output data to a server via a wireless or wired communication link,

wherein the unmanned aerial vehicle output data comprises highly redundant imagery with full generality of structural shape, height, obstructions, and operator error, which generally requires no topographical aerial image data, that can be used to generate a three-dimensional structural reconstruction that is accurately scaled in three dimensions with less than one-percent systematic relative error with or without GPS/GNSS;

wherein the unmanned aerial vehicle output data comprises information used to create a point cloud density over an entire structure area of interest, which is substantially uniform ranging from 100-10,000 points per square meter while retaining at least two centimeters vertical precision, that can be converted into a regularized vector model of the structure area of interest.

2. The system of claim 1, further comprising a server configured to:

receive the transmitted unmanned aerial vehicle output data, wherein the unmanned aerial vehicle output data comprises reliable geotagged images of a structure area of interest and global positioning system information, comprising highly redundant imagery with full generality of structural shape, height, obstructions, and operator errors, which requires no a priori topographic or aerial image data, that can be used to generate a reconstruction accurately scaled in three dimensions with less than one percent systematic relative error with or without GPS/GNSS;

store the modified unmanned aerial vehicle data in an image database;

generate a three-dimensional photogrammetric point cloud density over the entire structure area of interest, which is substantially uniform ranging from 100-10,000 points per square meter while retaining at least two centimeters vertical precision, that can be converted into the regularized vector model of the structure area of interest, wherein the three-dimensional photogrammetric point cloud density is generated based at least in part on the received modified unmanned aerial vehicle data;

reconstruct surface points using dense matching algorithms to correlate neighboring images with a sufficiently low angular separation, with minimal point cloud voids even with limited surface texture, which allows three-dimensional reconstruction and image inspection to be complete despite obstructions occluding portions of the structural area of interest in at least some of the geotagged images; and

create three-dimensional regularized vector models using the point cloud model and regularization algorithms.

3. The system of claim 1, wherein the flight plan comprises a launch or home location, property bounds, a crude outline of the structure area of interest, structure height, and the obstacle clearance height.

4. The system of claim 3, wherein the home location comprises the global positioning system location of where the unmanned aerial vehicle is initialized, will launch from, and will land near.

5. The system of claim 1, wherein the third party data comprises geographic or property information from public or commercially available databases based at least in part on the property mailing address or geolocation.

6. The system of claim 1, wherein the unmanned aerial vehicle control data comprises a flight plan for the unmanned aerial vehicle comprising an obstacle free launch trajectory and an acquisition trajectory.

7. The system of claim 6, wherein the flight plan is executed by issuing commands to the unmanned aerial vehicle'"'"'s flight and camera controller at least in part via the unmanned aerial vehicle'"'"'s application program interface, to traverse a series of geographic waypoints, capture images in orientations ensuring substantially complete coverage of the structure area of interest, and record camera position and orientation, wherein the flight plan, when paused, allows the field user to manually control the unmanned aerial vehicle to capture supplemental images of interest before resuming the flight plan.

8. The system of claim 1, wherein the unmanned aerial vehicle control data comprises commands for image capturing at camera exposure stations along the acquisition trajectory such that successive images are angularly separated by less than an angle θ

with respect to the centroid of the structure area of interest, and wherein the camera at each exposure station is generally orthogonal to the acquisition trajectory such that the unmanned aerial vehicle'"'"'s camera gimbal does not require yaw steering control.

9. The system of claim 8, wherein the unmanned aerial vehicle control data comprises two geometric transformation algorithms that use the input data, third party data or data about the structure area of interest to ensure that the camera exposure stations enable three-dimensional reconstruction of the structure area of interest, wherein

a convex hull is computed, eliminating concavities in the trajectory which could lead to diverging perspectives and reduced overlap or low parallax displacement between successive exposures;

and

a dilation operation is applied to maintain adequate range so that the entire structure area of interest is contained within each image field of view, wherein photos are captured at intervals along the acquisition trajectory to insure that the angular separation between camera exposures does not exceed an angle θ

that would reduce image match accuracy but also minimizes the total number of images required.

10. The system of claim 1, wherein the unmanned aerial vehicle output data comprises, at least in part, geotagged images of the structure area of interest, GPS/GNSS information, or calibration targets deployed to accurately scale a three-dimensional structure model.

11. The system of claim 1, wherein the unmanned aerial vehicle output data is generally immediately made available to a server to assure accuracy of the unmanned aerial vehicle data and to facilitate rapid remote inspection of the structure area of interest.

12. The system of claim 1, wherein the unmanned aerial vehicle output data is limited to the structure area of interest and eliminates or degrades imagery of objects or people outside the property bounds based on an image mask derived from the three-dimensional reconstruction polygon mesh and the property boundary.

13. A method of capturing and processing automated images comprising:

receiving the transmitted unmanned aerial vehicle output data by a server, wherein the unmanned aerial vehicle output data comprises reliable geotagged images of a structure area of interest and global positioning system information, comprising highly redundant imagery with full generality of structural shape, height, obstructions, and operator errors, which requires no a priori topographic or aerial image data, that can be used to generate a regularized vector model accurately scaled in three dimensions with less than one percent systematic relative error;

storing the modified unmanned aerial vehicle data in an image database on the server;

generating a three-dimensional photogrammetric point cloud density by the server over the entire structure area of interest, which is substantially uniform ranging from 100-10,000 points per square meter while retaining at least two centimeters vertical precision, that can be converted into a regularized vector model of the structure area of interest, wherein the three-dimensional photogrammetric point cloud density is generated based at least in part on the received modified unmanned aerial vehicle data;

reconstructing surface points using dense matching algorithms on a server to correlate neighboring images with a sufficiently low angular separation and minimal point cloud voids, even with limited surface texture, which allows three-dimensional reconstruction and image inspection to be complete despite obstructions occluding portions of the structural area of interest in at least some of the fully redundant set geotagged images; and

creating three-dimensional regularized vector models using the point cloud model and regularization algorithms.

14. The method of claim 13, wherein the unmanned aerial vehicle output data comprises precise visual reference images based on captured images, which are a generally exact orthographical projection, containing image features that are not shifted by a parallax based on their elevation.

15. The method of claim 13, further comprising a structure model generator, on the server, configured to perform initial quality checks on the modified unmanned aerial vehicle data to verify the modified unmanned aerial vehicle data is usable, and wherein the structure model generator issues a notification if the modified unmanned aerial vehicle data needs to be reacquired.

16. The method of claim 14, wherein the structure model generator retrieves a collection of images with corresponding geotags, and a nominal camera model from the image database, and performs image matching to solve for a photogrammetric bundle adjustment, dense matching, and vector surface fitting.

17. The method of claim 15, wherein the structure model generator performs vector cleaning, wherein facets of the initial structural vector model are regularized with algorithms to create a regularized watertight vector model with edges labeled according to their topology and construction conventions.

18. The method of claim 16, wherein facet edge classifications can be made within tolerances to accommodate systematic tilts of a few degrees with the three-dimensional reconstruction process due to GNSS errors.

19. The method of claim 17, wherein the structure model generator creates a report with data suitable to estimate a needed task by the structure report engine, wherein the structure report engine uses the regularized vector model as input, and annotates the regularized vector model with descriptive information, wherein the descriptive information comprises dimensions and selected inspection annotations stored in the structure database, to form a report design plan diagram, and wherein the report design plan diagram is published into a structure database and is capable of being retrieved by an enterprise user or other users along with other three dimensional data products available for self annotation, interactive measurement, and report modification.

20. A non-transitory computer readable medium with instructions stored thereon which, if executed by a processor, causes the processor to:

receive the transmitted unmanned aerial vehicle output data by a server, wherein the unmanned aerial vehicle output data comprises reliable geotagged images of an area of interest and global positioning system information, comprising highly redundant imagery with full generality of structural shape, height, obstructions, and operator errors, which requires no a priori topographic or aerial image data, that can be used to generate a reconstruction accurately scaled in three dimensions with less than one percent systematic relative error;

store the modified unmanned aerial vehicle data in an image database on the server;

generate a three-dimensional photogrammetric point cloud by the server over the entire area of interest, which is substantially uniform ranging from 100-10,000 points per square meter while retaining at least two centimeters vertical precision, that can be converted into a regularized vector model of the area of interest, wherein the three-dimensional photogrammetric point cloud density is generated based at least in part on the received modified unmanned aerial vehicle data;

reconstruct surfaces of the area of interest using dense matching algorithms on a server to correlate neighboring images with a sufficiently low angular separation and minimal point cloud voids, even with limited surface texture, which allows three-dimensional reconstruction and image inspection to be complete despite obstructions occluding portions of the structural area of interest in at least some of the geotagged images; and

create three-dimensional regularized vector models using the point cloud model and regularization algorithms.