SYSTEM FOR COLLECTING & PROCESSING AERIAL IMAGERY WITH ENHANCED 3D & NIR IMAGING CAPABILITY

• US 20150022656A1
• Filed: 07/16/2014
• Published: 01/22/2015
• Est. Priority Date: 07/17/2013
• Status: Active Grant

• Current Assignee: James L. Carr, Joseph R. Fox-Rabinovitz, Lawrence A. Herron Jr, Nathaniel F. Allen, Stephen J. Fujikawa
• Original Assignee: Lawrence A. Herron Jr
• Inventors: Carr, James L., Fujikawa, Stephen J., Herron, Lawrence A. Jr., Allen, Nathaniel F., Fox-Rabinovitz, Joseph R.
• Granted Patent: US 9,798,928 B2
• Time in Patent Office: Days
• US Class Current: 348/117
• CPC Class Codes: G01C 11/025 by scanning the object
• CPC Class Codes: G06V 20/13 Satellite images
• CPC Class Codes: G06V 20/188 Vegetation

Abstract

A system for guided geospatial image capture, registration and 2D or 3D mosaicking, that employs automated imagery processing and cutting-edge airborne image mapping technologies for generation of geo-referenced Orthomosaics and Digital Elevation Models from aerial images obtained by UAVs and/or manned aircraft.

Claims

1. A system for collection and processing aerial imagery, comprising:

at least one camera;

an avionics box comprising a microprocessor;

a GPS receiver;

a remote computing device; and

a computing cloud comprising a plurality of cloud computing nodes, at least one data storage device and at least one processing device running at least one software application, c

wherein said at least one software application comprises an accelerated Normalized Cross-Correlation algorithm that locates image tie points as between two or more images via a multi-resolution matching method.

2. The system of claim 1, wherein said at least one camera comprises two consumer market digital single lens reflex cameras.

3. The system of claim 1, wherein at least one of said at least one cameras is modified for acquisition of Near-Infrared imagery.

4. The system of claim 1, wherein said remote computing device further comprises a pilot interface displaying pilot guidance imagery pertaining to a predetermined flight path.

5. The system of claim 4, wherein said predetermined flight path comprises a plurality of rasters oriented towards or away from the sun and covering a predetermined geographic area.

6. The system of claim 5, wherein said remote computing device is mounted in an aircraft, and wherein said pilot interface comprises a display showing the position of said aircraft relative to said rasters.

7. The system of claim 5, wherein said remote computing device further comprises a solar ephemeris.

8. The system of claim 1, wherein said avionics box further comprises a triggering system coupled to said at least one camera.

9. The system of claim 8, wherein said avionics box is in cooperative communication with said GPS receiver, wherein said avionics box further comprises a counter that increments with each one of a plurality of GPS top-of-second pulses, and wherein said counter is in cooperative communication with said triggering mechanism.

10. The system of claim 9, further comprising a digital connection between said avionics box and said remote computing device for transmission of command dialog from said remote computing device to said avionics box and for transmission and storage of GPS location data from said GPS device.

11. The system of claim 1, wherein said at least one camera, said avionics box, said GPS receiver and said remote computing device are mounted in an aircraft.

12. In a system comprising a cloud computing system comprising a plurality of cloud computing nodes, at least one data storage device and at least one processing device, said cloud computing system being connected via a communications network to at least one remote computer workstation, and a portable computing device coupled with at least one imaging device and at least one GPS receiver, a method for collecting and processing aerial imagery comprising the steps of:

mapping, by said portable computing device, one or more flight paths over a target geographic area;

collecting, by said remote computer workstation, a plurality of GPS data elements pertaining to said flight path from said portable computing device, and a plurality of images from said at least one imaging device, wherein each one of said plurality of images is associated with at least one of said plurality of GPS data elements;

applying, by at least one of said plurality of cloud computing nodes, a multiresolution linking process to said plurality of images whereby said linking process identifies a plurality of image tie points between two or more of said plurality of images;

applying, by at least one of said plurality of cloud computing nodes, a navigation process to said plurality of images whereby a roll, pitch, yaw, and absolute three-dimensional location of said imaging device in the geographic coordinate reference system is determined with respect to each one of said plurality of images; and

applying, by at least one of said plurality of cloud computing nodes, an orthorectification process to each one of said plurality of images, whereby each one of said plurality of images is assigned a geographic referencing metadata tag indicating the association between at least one pixel in each of said plurality of images with a point in the geographic coordinate reference system.

13. The method of claim 12, wherein said mapping step further comprises

determining, by said portable computing device, the position of the sun in the sky;

calculating, by said portable computing device, the direction of flight towards and away from the sun; and

mapping, by said portable computing device, in real-time a plurality of parallel rasters of alternating direction along said direction of flight towards and away from the sun and covering said target geographic area;

wherein two or more of said plurality of rasters collectively comprise one of said one or more flight paths.

14. The method of claim 12, further comprising, after said step of mapping, guiding, by said portable computing device, the flight of an aircraft containing said at least one portable computing device and said at least one imaging device along at least one of said one or more flight paths.

15. The method of claim 14, further comprising the steps of

requesting, by said remote computing device, a user input for an amount of time between images;

and

transmitting, by said remote computing device, said user input to an avionics box on said aircraft, wherein said user input directs said avionics box to trigger at least one of said imaging devices at a GPS top-of-second based on said user input when said aircraft is above said target geographic area.

16. The method of claim 15, wherein said step of collecting further comprises the step of importing, by said portable computing device, said plurality of GPS data elements into flight planning software running on said portable computing device.

17. The method of claim 12, further comprising, after the step of collecting, uploading, via said remote computer workstation, said plurality of GPS data elements and said plurality of images to said cloud computing system.

18. The method of claim 17, further comprising, after said step of collecting, uploading, via said remote computer workstation, public reference imagery relating to said target geographic area to said cloud computing device.

19. The method of claim 18, wherein said multiresolution linking process further comprises, in one of said plurality of cloud computing nodes

selecting, from said plurality of images, a home frame image and a target frame image;

binning said home frame and said target frame by combining arrays of pixels in said frames to reduce the number of pixels by a predetermined factor;

extracting one or more chips from said home frame, wherein each of said chips comprises one or more adjacent pixels;

reprojecting said one or more chips into the perspective of said target frame;

applying an accelerated Normalized Cross-Correlation algorithm to each of said one or more chips relative to said target frame to locate any successful image tie point matches;

saving said successful image tie point matches to said at least one data storage device;

re-binning said home frame and said target frame to increase the number of pixels in said frames by a predetermined factor;

using said successful image tie point matches to re-navigate said re-binned target frame; and

repeating said selecting, binning, extracting, reprojecting, applying, saving, re-binning, and using steps for a predetermined number of iterations.

20. The method of claim 12, further comprising, comparing, by one of said plurality of cloud computing nodes, the locations of fixed landmarks seen in reference imagery with locations of said fixed landmarks as they appear in said plurality of images.

21. The method of claim 20, further comprising, in one of said plurality of cloud computing nodes, repeating said steps of applying a navigation process and applying an orthorectification process, wherein the locations of said fixed landmarks are used as inputs for said step of applying a navigation process.

22. The method of claim 12, further comprising, in one of said plurality of cloud computing nodes, rendering two successive overlapping images from said plurality of images to create a 3D mosaic of said target geographic area.

23. The method of claim 22, further comprising using said 3D mosaic to measure the heights of objects in said target geographic area.

24. The method of claim 22, further comprising converting said 3D mosaic to a Normalized Differential Vegetation Index for diagnosis of plant health.

25. The method of claim 12, further comprising applying a Normalized Differential Vegetation Index to said plurality of images for diagnosis of plant health.

26. A cloud-based system for collecting, processing and distributing aerial imagery, comprising:

a first cloud computing node connected in the cloud computing network comprising a processor and a computer readable medium storing a user software application accessible by any number of remote computers, said user software application comprising computer-executable instructions for uploading a plurality of aerial image photos and a corresponding flight log and calibration data from said remote computer(s) to the cloud computing network, and

a second cloud computing node connected in the cloud computing network comprising a processor and a computer readable medium storing a cloud-based application comprising computer-executable instructions for carrying out the steps of,

performing a first-pass of an image tie point registration by applying an accelerated Normalized Cross-Correlation (NCC) algorithm to determine image tie-point matches between successive pairs of said uploaded aerial image photos at a plurality of different binned resolutions,

performing state solution estimation by applying a least squares estimation method to estimate camera roll (r), pitch (p), and yaw (y) states for said uploaded aerial image photos, GPS corrections along three axes (x, y, z), and camera calibration states,

ortho-rectifying each of said uploaded aerial image photos to a map projection in accordance with said estimated state solution, and

displaying said orthorectified map projection to one of said remote users.

27. The cloud-based system according to claim 26, wherein said cloud-based application further comprises computer-executable instructions for carrying out the step of performing a second-pass of an image tie point registration by the steps of,

determining deviations in geo-registration of reference imagery from said ortho-rectified uploaded aerial image photos,

correcting said deviations in geo-registration of reference imagery, and

remapping said map projection in accordance with said corrected reference imagery.