Airborne multispectral imaging system with integrated navigation sensors and automatic image stitching

• US 9,945,828 B1
• Filed: 10/23/2015
• Issued: 04/17/2018
• Est. Priority Date: 10/23/2015
• Status: Active Grant

• Current Assignee: Sentek Systems, LLC
• Original Assignee: Sentek Systems, LLC
• Inventors: Poling, Bryan T., Salant, Kimberly A., Poling, Theodore C., Launer, Christopher A.
• Primary Examiner(s): Yang, Weiwen
• Application Number:
  US14/921,335
• Time in Patent Office: 907 Days
• Field of Search: None
• CPC Class Codes: G01N 33/0098 Plants or trees wood G01N33/46
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/30188 Vegetation; Agriculture
• CPC Class Codes: G06T 7/0004 Industrial image inspection
• CPC Class Codes: G06T 7/30 Determination of transform parameters for the alignment of images, i.e. image registration
• CPC Class Codes: H01L 27/14645 Colour imagers
• CPC Class Codes: H01L 27/14649 Infrared imagers
• CPC Class Codes: H04N 23/11 for generating image signals from visible and infrared light wavelengths
• CPC Class Codes: H04N 23/951 by using two or more images to influence resolution, frame rate or aspect ratio
• CPC Class Codes: H04N 7/185 from a mobile camera, e.g. for remote control
• CPC Class Codes: Y02A 40/10 in agriculture

Abstract

A self-contained UAV sensor payload and on-board or off-board hardware and software for precision agriculture. The invention combines multiple cameras and navigation sensors to create a system for automatically collecting, geo-referencing, and stitching red, green, blue, and near infrared imagery and derived vegetation indices. The invention is able to produce all of these quantities in a single flight. The tight integration with integrated navigation sensors eliminates the need for integration with external sensors or avionics hardware and enables innovative solutions to image processing and analysis which greatly improve accuracy, processing speed, and reliability over alternative approaches.

Claims

1. A system for use in precision agriculture comprising:

a. a self-contained unmanned aerial vehicle (UAV) sensor payload including;

i. a plurality of sensors including gyroscopes, accelerometers, magnetometers, and a GNSS receiver,

ii. an image sensor sensitive to near-infrared (NIR),

iii. an image sensor sensitive to red-green-blue (RGB),

iv. an integrated processor and supporting hardware, wherein all sensors are time-synchronized and co-located, and the relative orientations of the sensors are known;

b. on-board or off-board hardware and software configured to;

i. compute position and attitude estimates of the self-contained unmanned aerial vehicle (UAV) sensor payload at each instant images are collected by the NIR and RGB imager sensors a(ii) and a(iii) using data from the time-synchronized and co-located plurality of sensors in a(i),

ii. determine a transformation between NIR and RGB imagery from the co-located and time-synchronized NIR and RGB image sensor a(ii) and a(iii) that maps a projection of each scene point in an image from one of the image sensors to a projection of the same point in an image from the other image sensor,

iii. compute vegetation indices using the NIR and RGB imagery obtained by the NIR and RBG image sensor a(ii) and a(iii) using the transformation from b(ii),

iv. use the computed position and attitude estimates from b(i) and the known relative orientations of the NIR and RGB image sensors a(ii) and a(iii) relative to the plurality of sensors in a(i) to geo-register and stitch the RGB and NIR imagery and the computed vegetation indices from (b)(iii).

2. The system according to claim 1 wherein the GNSS receiver is a standard GPS receiver.

3. The system according to claim 1 wherein the GNSS receiver is a differential GPS receiver.

4. The system according to claim 1 wherein the GNSS receiver is an RTK GPS receiver.

5. The system according to claim 1 wherein the GNSS receiver is a Galileo, Glonass, or other constellation receiver or a multi-constellation receiver.

6. The system according to claim 1 wherein an incident light sensor is used to measure incident irradiance, for use in computing surface reflectance.

7. The system according to claim 1 wherein the plurality of sensors and the onboard processor are configured to detect take-off and landing events and autonomously begin recording at take-off and terminates recording at landing.

8. The system according to claim 1 wherein the plurality of sensors and the onboard processor are configured to begin and stop recording when a signal is received from an external source, or where images are collected on demand according to an externally provided trigger signal.

9. The system according to claim 1 wherein multi-threading and/or GPU acceleration is used to accelerate the processing.

10. The system according to claim 1 wherein the processor uses real-time navigation data to continually optimize camera settings.

11. The system according to claim 1 wherein sub-pixel level spectral alignment is achieved at all altitudes through a combination of factory alignment and image processing that produces an estimate of residual misalignment due to physical hardware changes over time and changing perspective.

12. The system according to claim 1 wherein the processor or separate computer is configured to use the navigation solution to compute prior estimates of image geo-registration and image alignment.

13. The system according to claim 1 wherein prior estimates of image alignment are used to screen possible feature correspondences to accelerate and improve robustness of image stitching.

14. The system according to claim 1 wherein prior-based correspondence screening is performed in a multi-hypothesis framework to dynamically determine alignment prediction accuracy.

15. The system according to claim 1 wherein the image sensors and associated lenses and optical filters are radiometrically characterized to produce vegetation indices that are consistent across multiple camera settings.

16. The system according to claim 1 wherein navigation measurements are used in place of uncomputable or unreliable photogrammetric measurements for system pose recovery and to perform stitching.

17. The system according to claim 1 wherein sensor height above ground is estimated by computing changes in GNSS altitude and stabilized using additional on-board sensors.

18. The system according to claim 1 wherein a software module is used to assess the noise environment on the UAV, including vibrational noise, EMI, and magnetic interference due to soft or hard iron anomalies or nearby electrical currents.