Machine vision-based method and system for aircraft docking guidance and aircraft type identification

• US 10,290,219 B2
• Filed: 07/02/2015
• Issued: 05/14/2019
• Est. Priority Date: 08/01/2014
• Status: Active Grant

• Current Assignee: Shenzhen Cimc-Tianda Airport Support, Ltd. (China International Marine Containers (Group) Co., Ltd.)
• Original Assignee: China International Marine Containers Limited (China International Marine Containers Company Limited), Shenzhen Cimc-Tianda Airport Support, Ltd. (China International Marine Containers (Group) Co., Ltd.)
• Inventors: Deng, Lan, Zhang, Zhaohong, Xiang, Wei, Yang, Yuefeng, Liu, Haiqiu, Wang, Haibin
• Primary Examiner(s): Bekele, Mekonen T
• Application Number:
  US15/329,994
• Publication Number:
  US 20170263139A1
• Time in Patent Office: 1,412 Days
• CPC Class Codes: B64F 1/002 Taxiing aids
• CPC Class Codes: B64F 1/18 Visual or acoustic landing aids
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/30236 Traffic on road, railway or crossing
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/136 involving thresholding
• CPC Class Codes: G06T 7/174 involving the use of two or more images
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06T 7/90 Determination of colour characteristics
• CPC Class Codes: G06V 10/20 Image preprocessing
• CPC Class Codes: G06V 10/26 Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
• CPC Class Codes: G06V 10/28 Quantising the image, e.g. histogram thresholding for discrimination between background and foreground patterns
• CPC Class Codes: G06V 10/435 Computation of moments
• CPC Class Codes: G06V 10/50 by performing operations within image blocks; by using histograms, e.g. histogram of oriented gradients [HoG]; by summing image-intensity values; Projection analysis
• CPC Class Codes: G06V 10/60 relating to illumination properties, e.g. using a reflectance or lighting model
• CPC Class Codes: G06V 10/7515 Shifting the patterns to accommodate for positional errors
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 20/54 of traffic, e.g. cars on the road, trains or boats
• CPC Class Codes: G08B 5/06 : using hydraulic transmission; using pneumatic transmission
• CPC Class Codes: G08G 5/0026 : located on the ground
• CPC Class Codes: G08G 5/065 : Navigation or guidance aids, e.g. for taxiing or rolling
• CPC Class Codes: H04N 7/00 : Television systems details H04N3/00, H04N5/00; methods or arrangements, for coding, decoding, compressing or decompressing digital video signals H04N19/00; selective content distribution H04N21/00

Abstract

A machine vision-based method and system for aircraft docking guidance and aircraft type identification, comprising: S1: a monitoring scenario is divided into different information processing function areas; S2: a captured image is pre-processed; S3: the engine and the front wheel of an aircraft are identified in the image, so as to confirm that the aircraft has appeared in the image; S4: continuous tracking and real-time updating are performed on the image of the engine and the front wheel of the aircraft captured in step S3; S5: real-time positioning of the aircraft is implemented and the degree of deviation of the aircraft with respect to a guide line and the distance with respect to a stop line are accurately determined; S6: the degree of deviation of the aircraft with respect to the guide line and the distance with respect to the stop line of step S5 are outputted and displayed.

Claims

1. A method for guiding an aircraft docking and recognizing an aircraft model based on machine vision, characterized in that the method comprises:

a step for configuring an aircraft berthing scene, in which a monitored scene is divided into regions for separate data processing;

an image pre-processing step, in which a captured image is pre-processed;

an aircraft capturing step, in which it is determined that an aircraft exists in the image by recognizing an engine and front wheels of the aircraft in the image;

an aircraft tracking step, in which the image of the engine and the front wheels of the aircraft recognized in the aircraft capturing step is continuously tracked and updated in real time;

an aircraft positioning step, in which the aircraft is positioned in real time, and a deviation degree of the aircraft from a guide line and a distance between the aircraft and a stop line are accurately determined; and

an information displaying step, in which the deviation degree of the aircraft from the guide line and the distance between the aircraft and the stop line determined in the aircraft positioning step are outputted and displayed, wherein the aircraft capturing step further comprises;

a background elimination step, in which a dynamic distribution of a background in the scene is simulated with a single Gaussian background model, a model of the background is established and the present image and the background model are differentiated to eliminate the background to obtain a foreground region;

a shade elimination step, gray values of respective pixels in the foreground region are computed to find the maximum gray value g max and the minimum gray value g min thereof, and shading elimination is performed on the regions with gray values lower than T=g min+(g max−

g min)*0.5;

a region classification step, in which a template of a standard front view of an aircraft region is created, a target region is extracted through change detection, and a vertical projection curve of the target region is calculated, a correlation coefficient between the vertical projection curve of the target region and a vertical projection curve of the template of the standard front view of the aircraft region is calculated, if the correlation coefficient is larger than or equals to a classification threshold, the target region is determined as an aircraft; and

a feature verification step, in which it is further determined whether the target region is an aircraft by detecting the recognized engine and front wheel of the aircraft.

2. The method of claim 1, characterized in that the image pre-processing step further comprises:

determining whether the image is an image of low illumination, high illumination or normal illumination according to an average of gray scales of the image, performing a step for processing a low-illumination image on the image if the image is an image of low illumination, and performing a step for processing a high-illumination image on the image if the image is an image of high illumination;

if the image is an image of normal illumination, determining whether the image is a normal image according to a variance of the gray scales of the image; and

if the image is not a normal image, determining whether the image is an image captured in rain or snow or an image captured in fog, performing a step for processing a rain-or-snow image on the image if the image is an image captured in rain or snow, and performing a step for processing a fog image on the image if the image is an image captured in fog.

3. The method of claim 2, characterized in that the step for processing a low-illumination image comprises processing the image with the following formula:

g(x,y)=f(x,y)+af(x,y)(255−

f(x,y))

where f(x, y) represents the image before the step for processing a low-illumination image, (x, y) represents coordinates of a pixel in the image, g(x, y) represents the image after the step for processing a low-illumination image, and a represents a parameter for processing a low-illumination image.

4. The method of claim 2, characterized in that the step for processing a rain-or-snow image comprises:

searching in the image for a pixel to be processed which is contaminated by rain or snow by using a photometric model;

for a pixel to be processed in the present image, extracting luminance values of corresponding pixels in a previous image and in a subsequent image of the present image, determining whether the corresponding pixels in the previous image and in the subsequent image of the present image are pixels to be processed according to the extracted luminance values, if the corresponding pixels are pixels to be processed, calculating an average value of the luminance values of the corresponding pixels and replacing a luminance value of the pixel to be processed in the present image with the average value, and if the corresponding pixels are not pixels to be processed, replacing a luminance value of the pixel to be processed in the present image with a minimum value of the luminance values of the corresponding pixels or an average value of two smallest luminance values of the corresponding pixels.

5. The method of claim 2, characterized in that the fog image processing step is performed by homomorphic filtering.

6. The method of claim 1, characterized in that the feature verification step further comprises:

(a) extracting a blackest image region, in which a gray scale histogram of the target region of a current frame of image is made, the maximum gray value and the minimum gray value are acquired in a gray level range of 1% to 99%, the darkest part of the image is extracted from the image by means of the maximum gray value, the minimum gray value and a preset threshold for judging a blackest part, to obtain a darkest region;

(b) circular-like detection, in which all outer boundaries of the blackest region are extracted, and for each boundary, a gravity center coordinate of the boundary is calculated with a moment of the boundary, a ji^th-order of the moment m_ji of the boundary is defined as follows;

7. The method of claim 6, characterized in that in the step (c), for M detected like-circular regions, a similarity degree Similarity_ij between an i^th like-circular region and a j^th like-circular region is calculated as:

Similarity_ij=|Height_i−

Height_j|*|Radius_i−

Radius_j|

where Height is a height of the gravity center, and Radius is the radius, when the similarity degree Similarity_ij is less than a preset similarity threshold, it is determined that the like-circular regions i and j are the aircraft engine.

8. The method of claim 7, characterized in that in step (c), if no aircraft engine is detected, the detection in the steps (a)-(c) are performed iteratively, with the threshold for judging a blackest part, the threshold for determining a circle and the similarity threshold being increased;

if after that, still no aircraft engine is detected, an open operation is performed for all the blackest regions using a circular template of 7*7, and the steps (a)-(c) are performed for another time;

if still no aircraft engine is detected, the above detection steps are performed iteratively for another 2 times; and

if still not aircraft engine is detected, it is determined that there is no engine in the image.

9. The method of claim 8, characterized in that the threshold for judging a blackest part, the threshold for determining a circle and the similarity threshold are increased respectively by 0.05, 0.5 and 20.

10. The method of claim 6, characterized in that step (d) further comprises:

in the search region, 256 levels of grayscale are quantized to 64 levels, a first peak and a first valley are searched out in the histogram of the 64 levels of grayscale after the quantization, the optimal peak position BestPeak and the optimal valley position BestValley in the original histogram of the 256 levels of grayscale are defined as;

11. The method of claim 6, characterized in that the aircraft tracking step further comprises:

after a position of an engine in a previous frame of image is obtained, a region of an engine in a current frame of image is determined and tracked using a flood filling method;

if filling result is invalid, a dark environment detection and tracking step is performed to detect and track the engine region with parameters used in processing the previous frame of image;

after information about a region of an engine is acquired, the front wheel of the aircraft is detected in accordance with step (d); and

a front wheel tracking emergency processing step, in which when it is determined that the region of the front wheel has an incorrect shape, or the position of the front wheel deviates significantly from that in the previous frames of images, from information of the previous frame of image and the current frame of image, a shift of the front wheel in the frame is estimated using a shift of an engine in adjacent two frames, the estimated result is taken as a front wheel tracking result, and if no front wheel is detected in more than N frames of images, an error message is outputted.

12. The method of claim 11, characterized in that the aircraft positioning step comprises:

a photographing device calibration and image correction step, configured to determine optical parameters of a photographing device and optical parameters of the photographing device with respect to the world coordinate system;

en aircraft front wheel deviation degree calculating step; and

an aircraft front wheel actual distance calculating step.

13. The method of claim 12, characterized in that the photographing device calibration and image correction step further comprises:

retrieving N calibration pictures;

a corner of a chessboard is found by using a function cvFindChessboardCorners ( ) of OpenCV, the retrieved N calibration pictures are respectively substituted to the function cvFindChessboardCorners ( );

if all the corners are successfully found, the function returns 1, and coordinates of the corners in the image coordinate system are acquired; and

if it is not successful, the function returns 0; and

coordinates of the corners which are successfully searched out in the calibration template are substituted into the function cvCalibrateCamera2( ), and the function returns a parameter matrix, a distortion coefficient, a rotation vector and a translation vector of the photographing device.

14. The method of claim 12, characterized in that the aircraft front wheel deviation degree calculating step further comprises:

according to the coordinates (x₀, y₀) of the obtained position of the front wheel, by utilizing a relationship between the coordinates of the position and the stop line, a linear equation of the guide line is calculated as y₁=k₁x₁+b₁, and a linear equation of the stop line is calculated as y₂=k₂x₂+b₂, and a distance from the coordinate point to the straight line is;

15. The method of claim 14, characterized in that the aircraft front wheel deviation degree calculating step further comprises:

determining whether |d₁|>

width/2 is satisfied, where width is a threshold equal to a width of a front wheel of an aircraft detected;

if |d₁|>

width/2 is satisfied, it means that the aircraft has deviated from the guide line.

16. The method of claim 12, characterized in that the aircraft front wheel actual distance calculating step further comprises:

establishing a corresponding relationship between image coordinates and geodetic coordinates;

obtaining image coordinates from the mark points in the configuration of the scene at the step for configuring an aircraft berthing scene, performing quadratic curve fitting on the image coordinates with a least squares method, to obtain a curve equation y=ax²+bx+c, where x is a distance in the image, y is an actual distance;

projecting the front wheel of the aircraft on the image on the guide line along a direction of the stop line, denoting an Euclidean distance from the projection point to the stop line by x, and obtaining the actual distance from the front wheel of the aircraft to the stop line from y=ax²+bx+c.

17. The method of claim 1, characterized in that, an aircraft identification and identity verification step is performed after the aircraft capturing step and comprises:

parameter verification, in which parameters of the aircraft in the image are extracted and compared with parameters of a corresponding model type preset in a database, to obtain a model similarity degree;

template matching, in which the image is compared with a model template preset in the database, to obtain a template similarity degree;

comprehensive judgment, in which when the model similarity degree and the template similarity degree are larger than or equal to a verification threshold, it is deemed that the identity verification is passed.

18. The method of claim 17, characterized in that the parameter verification step further comprises:

(a) parameters of the aircraft engine in the image are extracted and compared with parameters of the aircraft engine of a corresponding model type preset in a database, to obtain a first ratio;

(b) the aircraft wing parameters in the image are extracted and compared to aircraft wing parameters of the corresponding model preset in the database, to obtain a second ratio;

(c) parameters of aircraft head in the image are extracted and compared with parameters of aircraft head corresponding to the model type preset in the database, to obtain a third ratio;

(d) parameters of the aircraft tail in the image are extracted and compared with aircraft tail parameters preset to the corresponding model type in the database, to obtain a fourth ratio;

(e) a minimum value and a maximum value among the first ratio, the second ratio, the third ratio, and the fourth ratio are taken, and the ratio of minimum/maximum is taken as the model similarity degree.

19. The method of claim 18, characterized in that the template matching step further comprises:

global template matching, in which a global template similarity degree is calculated by taking the whole image as the image to be searched and the standard aircraft image as the template;

local template matching, in which with the aircraft engine, the aircraft wings, the aircraft head, and the aircraft tail respectively extracted in the steps (a)-(d) as the images to be searched, and the engine, the wings, the head and the tail of the standard aircraft image as templates, 4 similarity degrees between the images to be searched and the templates are calculated, a minimum value of the 4 similarity degrees is removed, and an average of the remaining 3 similarity degrees of the 4 similarity degrees is taken as a local template similarity degree.

20. The method of claim 19, characterized in that the comprehensive judgment step further comprises:

if at least two of the model similarity degree, the global template similarity degree and the local template similarity degree are larger than or equal to a first verification threshold, it is deemed that the identity verification is passed, or, if each of the model similarity degree, the global template similarity degree and the local template similarity degree is larger than a second verification threshold, it is deemed that the identity verification is passed.

21. A system for guiding an aircraft docking and recognizing an aircraft model based on machine vision, characterized in that the system comprises:

a photographing device, a central processing device and a display device,

wherein the central processing device is configured to perform;

a step for configuring an aircraft berthing scene, in which a monitored scene is divided into regions for separate data processing;

an image pre-processing step, in which a captured image is pre-processed;

an aircraft capturing step, in which it is determined that an aircraft exists in the image by recognizing an engine and front wheels of the aircraft in the image;

an aircraft tracking step, in which the image of the engine and the front wheels of the aircraft recognized in the aircraft capturing step is continuously tracked and updated in real time;

an aircraft positioning step, in which the aircraft is positioned in real time, and a deviation degree of the aircraft from a guide line and a distance between the aircraft and a stop line are accurately determined; and

an information displaying step, in which the deviation degree of the aircraft from the guide line and the distance between the aircraft and the stop line determined in the aircraft positioning step are outputted and displayed, wherein the aircraft capturing step further comprises;

a background elimination step, in which a dynamic distribution of a background in the scene is simulated with a single Gaussian background model, a model of the background is established and the present image and the background model are differentiated to eliminate the background to obtain a foreground region;

a shade elimination step, gay values of respective pixels in the foreground region are computed to find the maximum gray value g max and the minimum gray value g min thereof, and shading elimination is performed on the regions with gray values lower than T=g min+(g max−

g min)*0.5;

a region classification step, in which a template of a standard front view of an aircraft region is created, a target region is extracted through change detection, and a vertical projection curve of the target region is calculated, a correlation coefficient between the vertical projection curve of the target region and a vertical projection curve of the template of the standard front view of the aircraft region is calculated, if the correlation coefficient is larger than or equals to a classification threshold the target region is determined as an aircraft; and

a feature verification step, in which it is further determined whether the target region is an aircraft by detecting the recognized engine and front wheel of the aircraft.

22. The system of claim 21, characterized in that after the aircraft capturing step, the central processing device is configured to perform:

an aircraft identification and identity verification step that comprises;

parameter verification, in which parameters of the aircraft in the image are extracted and compared with parameters of a corresponding model type preset in a database, to obtain a model similarity degree;

template matching, in which the image is compared with a model template preset in the database, to obtain a template similarity degree; and

comprehensive judgment, in which when the model similarity degree and the template similarity degree are larger than or equal to a verification threshold, it is deemed that the identity verification is passed,

wherein the parameter verification step further comprises;

(a) parameters of the aircraft engine in the image are extracted and compared with parameters of the aircraft engine of a corresponding model type preset in a database, to obtain a first ratio;

(b) the aircraft wing parameters in the image are extracted and compared to aircraft wing parameters of the corresponding model preset in the database, to obtain a second ratio;

(c) parameters of aircraft head in the image are extracted and compared with parameters of aircraft head corresponding to the model type preset in the database, to obtain a third ratio;

(d) parameters of the aircraft tail in the image are extracted and compared with aircraft tail parameters preset to the corresponding model type in the database, to obtain a fourth ratio; and

(e) a minimum value and a maximum value among the first ratio, the second ratio, the third ratio, and the fourth ratio are taken, and the ratio of minimum/maximum is taken as the model similarity degree;

the template matching step further comprises;

global template matching, in which a global template similarity degree is calculated by taking the whole image as the image to be searched and the standard aircraft image as the template;

local template matching, in which with the aircraft engine, the aircraft wings, the aircraft head, and the aircraft tail respectively extracted in the steps (a)-(d) as the images to be searched, and the engine, the wings, the head and the tail of the standard aircraft image as templates, 4 similarity degrees between the images to be searched and the templates are calculated, a minimum value of the 4 similarity degrees is removed, and an average of the remaining 3 similarity degrees of the 4 similarity degrees is taken as a local template similarity degree;

the comprehensive judgment step further comprises;

if at least two of the model similarity degree, the global template similarity degree and the local template similarity degree are larger than or equal to a first verification threshold, it is deemed that the identity verification is passed, or, if each of the model similarity degree, the global template similarity degree and the local template similarity degree is larger than a second verification threshold, it is deemed that the identity verification is passed.