Unmanned aerial vehicle image processing system and method

• US 9,336,568 B2
• Filed: 06/15/2012
• Issued: 05/10/2016
• Est. Priority Date: 06/17/2011
• Status: Active Grant

• Current Assignee: National Cheng KUNG University (Government of The Republic of China)
• Original Assignee: National Cheng KUNG University (Government of The Republic of China)
• Inventors: Liu, Cheng Chien
• Primary Examiner(s): Prince, Jessica M
• Assistant Examiner(s): Mung, On S
• Application Number:
  US13/524,045
• Publication Number:
  US 20120320203A1
• Time in Patent Office: 1,425 Days
• Field of Search: 348/144
• US Class Current: 1/1
• CPC Class Codes: G01C 11/02 Picture taking arrangements specially adapted for photogrammetry or photographic surveying, e.g. controlling overlapping of pictures
• CPC Class Codes: G01C 11/04 Interpretation of pictures
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: H04N 7/18 Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources

Abstract

An unmanned aerial vehicle image processing system includes a digital camera mounted on an unmanned aerial vehicle, an autopilot controller installed in the unmanned aerial vehicle and pre-loaded with a base image, and a computer device. The autopilot controller programs a flight path as a series of mission lags to cover a target polygon on the base image in a lawnmower pattern for controlling the unmanned aerial vehicle to perform a flight mission in which the autopilot controller records plural GPS/INS data and triggers the digital camera to take plural photos which are further calibrated to generate plural calibrated photos corresponding to the plural GPS/INS data, respectively. The computer device processes the plural calibrated photos to generate plural projected photos by using the corresponding plural GPS/INS data, respectively, and stitches the plural projected photos to a mosaic image by a photo-stitching process.

Claims

1. An unmanned aerial vehicle image processing system, comprising:

a digital camera mounted on an unmanned aerial vehicle;

an autopilot controller installed in the unmanned aerial vehicle and pre-loaded with a fly file generated from a base image drawn thereon a target polygon, wherein the autopilot controller is configured to program a flight path as a series of mission lags to cover the target polygon on the base image in a lawnmower pattern for controlling the unmanned aerial vehicle to perform a flight mission by traveling along the flight path in which the autopilot controller records plural Global Positioning System/Inertial Navigation System (GPS/INS) data and triggers the digital camera plural times to take plural photos which are further calibrated based on calibration parameters of lens distortion of the digital camera to calculate the amount of distortion for each pixel of plural photos, so as to generate plural calibrated photos corresponding to the plural GPS/INS data, respectively; and

a computer device for receiving the plural calibrated photos and the plural GPS/INS data, processing the plural calibrated photos to generate plural projected photos by using the corresponding plural GPS/INS data, respectively, and stitching the plural projected photos to a mosaic image by a photo-stitching process,

wherein the calibration parameters of lens distortion include principal point deviation, decentric lens distortion and radial lens distortion, and the amount of distortion for each pixel (x,y) of one photo is calculated based on the following equation;

2. The unmanned aerial vehicle image processing system as claimed in claim 1, wherein the computer device further geo-referencing the mosaic image by specifying coordinates of three points referenced from the base image.

3. The unmanned aerial vehicle image processing system as claimed in claim 1, wherein the base image is cut into plural reference images corresponding to the plural photos taken in the flight mission, respectively, and the computer device compares each projected photo to its corresponding reference image to generate an Rotation-Scale-Translation (RST)-corrected GPS/INS data, applies RST transformation to geo-reference the corresponding calibrated photo with the RST-corrected GPS/INS data, so as to generate an RST photo, and stitches all RST photos to a mosaic image by a photo-stitching process.

4. The unmanned aerial vehicle image processing system as claimed in claim 3, wherein each RST-corrected GPS/INS data is generated by selecting three pairs of conjugate points to conduct an iterated calculation.

5. The unmanned aerial vehicle image processing system as claimed in claim 1, wherein a Digital Terrain Model (DTM) for the base image is cut into plural reference DTMs corresponding to the plural photos taken in the flight mission, respectively, and the computer device determines detailed relief displacement for each pixel in each photo taken in the flight mission with a corresponding reference DTM and three pairs of conjugate points, orthorectifies each calibrated photo to an ortho photo based on the detailed relief displacements of the corresponding photo, and stitches all ortho photos to a mosaic image by a photo-stitching process.

6. An image processing method for use in an unmanned aerial vehicle image processing system including a digital camera mounted on an unmanned aerial vehicle, an autopilot controller installed in the unmanned aerial vehicle, and a computer device, the method comprising the steps of:

(A) configuring the autopilot controller, which is pre-loaded with a fly file generated from a base image drawn thereon a target polygon, to program a flight path as a series of mission lags to cover the target polygon on the base image in a lawnmower pattern for controlling the unmanned aerial vehicle to perform a flight mission by traveling along the flight path in which the autopilot controller records plural Global Positioning System/Inertial Navigation System (GPS/INS) data and triggers the digital camera plural times to take plural photos which are further calibrated based on calibration parameters of lens distortion of the digital camera to calculate the amount of distortion for each pixel of plural photos, so as to generate plural calibrated photos corresponding to the plural GPS/INS data, respectively;

(B) downloading the plural calibrated photos and the plural GPS/INS data into the computer device for processing the plural calibrated photos to generate plural projected photos by using the corresponding plural GPS/INS data, respectively; and

(C) stitching the plural projected photos to a mosaic image by a photo-stitching process,

wherein the calibration parameters of lens distortion include principal point deviation, decentric lens distortion and radial lens distortion, and the amount of distortion for each pixel (x,y) of one photo is calculated based on the following equation

7. The image processing method as claimed in claim 6, further comprises:

geo-referencing the mosaic image by specifying coordinates of three points referenced from the base image.

8. The image processing method as claimed in claim 6, wherein the flight path is programmed to specify flight courses to be parallel wind direction.

9. The image processing method as claimed in claim 6, wherein the base image is cut into plural reference images corresponding to the plural photos taken in the flight mission, respectively, and the computer device compares each projected photo to its corresponding reference image to generate an Rotation-Scale-Translation (RST)-corrected GPS/INS data, applies RST transformation to geo-reference the corresponding calibrated photo with the RST-corrected GPS/INS data, so as to generate an RST photo, and stitches all RST photos to a mosaic image by a photo-stitching process.

10. The image processing method as claimed in claim 9, wherein each RST-corrected GPS/INS data is generated by selecting three pairs of conjugate points to conduct an iterated calculation.

11. The image processing method as claimed in claim 6, wherein a Digital Terrain Model (DTM) for the base image is cut into plural reference DTMs corresponding to the plural photos taken in the flight mission, respectively, and the computer device determines detailed relief displacement for each pixel in each photo taken in the flight mission with a corresponding reference DTM and three pairs of conjugate points, orthorectifies each calibrated photo to an ortho photo based on the detailed relief displacements of the corresponding photo, and stitches all ortho photos to a mosaic image by a photo-stitching process.