MACHINE VISION-BASED METHOD AND SYSTEM FOR AIRCRAFT DOCKING GUIDANCE AND AIRCRAFT TYPE IDENTIFICATION

• US 20170263139A1
• Filed: 07/02/2015
• Published: 09/14/2017
• Est. Priority Date: 08/01/2014
• Status: Active Grant

• Current Assignee: Shenzhen Cimc-Tianda Airport Support, Ltd. (Pteris Global Limited)
• Original Assignee: China International Marine Containers Limited (China International Marine Containers Company Limited), Shenzhen Cimc-Tianda Airport Support, Ltd. (Pteris Global Limited)
• Inventors: DENG, Lan, ZHANG, Zhaohong, XIANG, Wei, YANG, Yuefeng, LIU, Haiqiu, WANG, Haibin
• Granted Patent: US 10,290,219 B2
• Time in Patent Office: Days
• CPC Class Codes: B64F 1/002 Taxiing aids
• CPC Class Codes: B64F 1/18 Visual or acoustic landing aids taxiing aids B64F1/002
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/30236 Traffic on road, railway or crossing
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/136 involving thresholding
• CPC Class Codes: G06T 7/174 involving the use of two or more images
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06T 7/90 Determination of colour characteristics
• CPC Class Codes: G06V 10/20 Image preprocessing
• CPC Class Codes: G06V 10/26 Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
• CPC Class Codes: G06V 10/28 Quantising the image, e.g. histogram thresholding for discrimination between background and foreground patterns
• CPC Class Codes: G06V 10/435 Computation of moments
• CPC Class Codes: G06V 10/50 by performing operations within image blocks; by using histograms, e.g. histogram of oriented gradients [HoG]; by summing image-intensity values; Projection analysis
• CPC Class Codes: G06V 10/60 relating to illumination properties, e.g. using a reflectance or lighting model
• CPC Class Codes: G06V 10/7515 Shifting the patterns to accommodate for positional errors
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 20/54 of traffic, e.g. cars on the road, trains or boats
• CPC Class Codes: G08B 5/06 : using hydraulic transmission; using pneumatic transmission
• CPC Class Codes: G08G 5/0026 : located on the ground
• CPC Class Codes: G08G 5/065 : Navigation or guidance aids, e.g. for taxiing or rolling
• CPC Class Codes: H04N 7/00 : Television systems details H04N3/00, H04N5/00; methods or arrangements, for coding, decoding, compressing or decompressing digital video signals H04N19/00; selective content distribution H04N21/00

Abstract

A machine vision-based method and system for aircraft docking guidance and aircraft type identification, comprising: S1, a monitoring scenario is divided into different information processing function areas; S2 a captured image is pre-processed; S3 the engine and the front wheel of an aircraft are identified in the image, so as to confirm that the aircraft has appeared in the image; S4 continuous tracking and real-time updating are performed on the image of the engine and the front wheel of the aircraft captured in step S3; S5 real-time positioning of the aircraft is implemented and the degree of deviation of the aircraft with respect to a guide line and the distance with respect to a stop line are accurately determined; S6 the degree of deviation of the aircraft with respect to the guide line and the distance with respect to the stop line of step S5 are outputted and displayed.

Claims

1. A method for guiding an aircraft docking and recognizing an aircraft model based on machine vision, characterized in that the method comprises:

step S1, a step for configuring an aircraft berthing scene, in which a monitored scene is divided into regions for separate data processing;

step S2, an image pre-processing step, in which a captured image is pre-processed;

step S3, an aircraft capturing step, in which it is determined that an aircraft exists in the image by recognizing an engine and front wheels of the aircraft in the image;

step S4, an aircraft tracking step, in which the image of the engine and the front wheels of the aircraft recognized in step S3 is continuously tracked and updated in real time;

step S5, an aircraft positioning step, in which the aircraft is positioned in real time, and a deviation degree of the aircraft from a guide line and a distance between the aircraft and a stop line are accurately determined; and

step S6, an information displaying step, in which the deviation degree of the aircraft from the guide line and the distance between the aircraft and the stop line determined in step S5 are outputted and displayed.

2. The method of claim 1, characterized in that the image pre-processing step further comprises:

step S21, determining whether the image is an image of low illumination, high illumination or normal illumination according to an average of gray scales of the image, performing a step for processing a low-illumination image on the image if the image is an image of low illumination, and performing a step for processing a high-illumination image on the image if the image is an image of high illumination;

step S22, if the image is an image of normal illumination, determining whether the image is a normal image according to a variance of the gray scales of the image; and

step S23, if the image is not a normal image, determining whether the image is an image captured in rain or snow or an image captured in fog, performing a step for processing a rain-or-snow image on the image if the image is an image captured in rain or snow, and performing a step for processing a fog image on the image if the image is an image captured in fog.

3. The method of claim 2, characterized in that the step for processing a low-illumination image comprises processing the image with the following formula:

g(x, y)=f(x, y)+af (x, y)(255−

f(x, y))

where f(x, y) represents the image before the step for processing a low-illumination image, (x, y) represents coordinates of a pixel in the image, g(x, y) represents the image after the step for processing a low-illumination image, and a represents a parameter for processing a low-illumination image.

4. The method of claim 2, characterized in that the step for processing a rain-or-snow image comprises:

searching in the image for a pixel to be processed which is contaminated by rain or snow by using a photometric model;

for a pixel to be processed in the present image, extracting luminance values of corresponding pixels in a previous image and in a subsequent image of the present image, determining whether the corresponding pixels in the previous image and in the subsequent image of the present image are pixels to be processed according to the extracted luminance values, if the corresponding pixels are pixels to be processed, calculating an average value of the luminance values of the corresponding pixels and replacing a luminance value of the pixel to be processed in the present image with the average value, and if the corresponding pixels are not pixels to be processed, replacing a luminance value of the pixel to be processed in the present image with a minimum value of the luminance values of the corresponding pixels or an average value of two smallest luminance values of the corresponding pixels.

5. The method of claim 2, characterized in that the fog image processing step is performed by homomorphic filtering.

6. The method of claim 1, characterized in that the aircraft capturing step further comprises:

step S31, a background elimination step, in which a dynamic distribution of a background in the scene is simulated with a single Gaussian background model, a model of the background is established and the present image and the background model are differentiated to eliminate the background to obtain a foreground region;

step S32, a shade elimination step, gray values of respective pixels in the foreground region are computed to find the maximum gray value g max and the minimum gray value g min thereof, and shading elimination is performed on the regions with gray values lower than T=g min+(g max−

g min)*0.5;

step S33, a region classification step, in which a template of a standard front view of an aircraft region is created, a target region is extracted through change detection, and a vertical projection curve of the target region is calculated, a correlation coefficient between the vertical projection curve of the target region and a vertical projection curve of the template of the standard front view of the aircraft region is calculated, if the correlation coefficient is larger than or equals to a classification threshold, the target region is determined as an aircraft;

step S34, a feature verification step, in which it is further determined whether the target region is an aircraft by detecting the recognized engine and front wheel of the aircraft.

7. The method of claim 6, characterized in that the feature verification step further comprises:

step S341, extracting a blackest image region, in which a gray scale histogram of the target region of a current frame of image is made, the maximum gray value and the minimum gray value are acquired in a gray level range of 1% to 99%, the darkest part of the image is extracted from the image by means of the maximum gray value, the minimum gray value and a preset threshold for judging a blackest part, to obtain a darkest region;

step S342, circular-like detection, in which all outer boundaries of the blackest region are extracted, and for each boundary, a gravity center coordinate of the boundary is calculated with a moment of the boundary, a ji^th-order of the moment m_ji of the boundary is defined as follows;

8. The method of claim 7, characterized in that in the step S343, for M detected like-circular regions, a similarity degree Similarity_ij between an i^th like-circular region and an j^th like-circular region is calculated as:

Similarity_ij=|Height_i−

Height_j|*|Radius_i−

Radius_j|

where Height is a height of the gravity center, and Radius is the radius, when the similarity degree Similarity_ij is less than a preset similarity threshold, it is determined that the like-circular regions i and j are the aircraft engine.

9. The method of claim 8, characterized in that in the step S343, if no aircraft engine is detected, the detection in the steps S341-343 are performed iteratively, with the threshold for judging a blackest part, the threshold for determining a circle and the similarity threshold being increased;

if after that, still no aircraft engine is detected, an open operation is performed for all the blackest regions using a circular template of 7*7, and the steps S341-343 are performed for another time;

if still no aircraft engine is detected, the above detection steps are performed iteratively for another 2 times; and

if still not aircraft engine is detected, it is determined that there is no engine in the image.

10. The method of claim 9, characterized in that the threshold for judging a blackest part, the threshold for determining a circle and the similarity threshold are increased respectively by 0.05, 0.5 and 20.

11. The method of claim 7, characterized in that the step 5344 further comprises:

in the search region, 256 levels of grayscale are quantized to 64 levels, a first peak and a first valley are searched out in the histogram of the 64 levels of grayscale after the quantization, the optimal peak position BestPeak and the optimal valley position BestValley in the original histogram of the 256 levels of grayscale are defined as;

12. The method of claim 7, characterized in that the aircraft tracking step further comprises:

step S41, after a position of an engine in a previous frame of image is obtained, a region of an engine in a current frame of image is determined and tracked using a flood filling method;

step S42, if filling result of the step S41 is invalid, a dark environment detection and tracking step is performed to detect and track the engine region with parameters used in processing the previous frame of image;

step S43, after information about a region of an engine is acquired, the front wheel of the aircraft is detected with the step S344; and

step S44, a front wheel tracking emergency processing step, in which when it is determined that the region of the front wheel has an incorrect shape, or the position of the front wheel deviates significantly from that in the previous frames of images, from information of the previous frame of image and the current frame of image, a shift of the front wheel in the frame is estimated using a shift of an engine in adjacent two frames, the estimated result is taken as a front wheel tracking result, and if no front wheel is detected in more than N frames of images, an error message is outputted.

13. The method of claim 12, characterized in that the aircraft positioning step comprises:

step S51, a photographing device calibration and image correction step, configured to determine optical parameters of a photographing device and optical parameters of the photographing device with respect to the world coordinate system;

step S52, an aircraft front wheel deviation degree calculating step; and

step S53, an aircraft front wheel actual distance calculating step.

14. The method of claim 13, characterized in that the step S51 further comprises:

step S511, retrieving N calibration pictures;

step S512, a corner of a chessboard is found by using a function cvFindChessboardCorners ( ) of OpenCV, the retrieved N calibration pictures are respectively substituted to the function cvFindChessboardCorners ( );

if all the corners are successfully found, the function returns 1, and coordinates of the corners in the image coordinate system are acquired; and

if it is not successful, the function returns 0; and

step S513, coordinates of the corners which are successfully searched out in the calibration template are substituted into the function cyCalibrateCamera2( ), and the function returns a parameter matrix, a distortion coefficient, a rotation vector and a translation vector of the photographing device.

15. The method of claim 13, characterized in that the step S52 further comprises:

according to the coordinates (x₀, y₀) of the position of the front wheel obtained in the step S43, by utilizing a relationship between the coordinates of the position and the stop line, a linear equation of the guide line is calculated as y₁=k₁x₁b₁, and a linear equation of the stop line is calculated as y₂=k₂x₂+b₂, and a distance from the coordinate point to the straight line is;

16. The method of claim 15, characterized in that the step S52 further comprises:

it is determined whether |d₁|>

width/2 is satisfied, where width is a threshold equal to a width of a front wheel of an aircraft detected;

if |d₁|>

width/2 is satisfied, it means that the aircraft has deviated from the guide line.

17. The method of claim 13, characterized in that the step S53 further comprises:

a corresponding relationship between image coordinates and geodetic coordinates is established;

image coordinates are obtained from the mark points in the configuration of the scene at the step S1, quadratic curve fitting is performed on the image coordinates with a least squares method, to obtain a curve equation y=ax²+bx+c, where x is a distance in the image, y is an actual distance;

the front wheel of the aircraft on the image is projected on the guide line along a direction of the stop line, an Euclidean distance from the projection point to the stop line is denoted by x, and the actual distance from the front wheel of the aircraft to the stop line is obtained from y=ax²+bx+c.

18. The method of claim 1, characterized in that after the step S3, step S7 is performed, the step S7 is an aircraft identification and identity verification step and further comprises:

step S71, parameter verification, in which parameters of the aircraft in the image are extracted and compared with parameters of a corresponding model type preset in a database, to obtain a model similarity degree;

step S72, template matching, in which the image is compared with a model template preset in the database, to obtain a template similarity degree;

step S73, comprehensive judgment, in which when the model similarity degree and the template similarity degree are larger than or equal to a verification threshold, it is deemed that the identity verification is passed.

19. The method of claim 18, characterized in that the step S71 further comprises:

step S711, parameters of the aircraft engine in the image are extracted and compared with parameters of the aircraft engine of a corresponding model type preset in a database, to obtain a first ratio;

step S712, the aircraft wing parameters in the image are extracted and compared to aircraft wing parameters of the corresponding model preset in the database, to obtain a second ratio;

step S713, parameters of aircraft head in the image are extracted and compared with parameters of aircraft head corresponding to the model type preset in the database, to obtain a third ratio;

step S714, parameters of the aircraft tail in the image are extracted and compared with aircraft tail parameters preset to the corresponding model type in the database, to obtain a fourth ratio;

step S715, a minimum value and a maximum value among the first ratio, the second ratio, the third ratio, and the fourth ratio are taken, and the ratio of minimum/maximum is taken as the model similarity degree.

20. The method of claim 19, characterized in that the step S72 further comprises:

step S721, global template matching, in which a global template similarity degree is calculated by taking the whole image as the image to be searched and the standard aircraft image as the template;

step S722, local template matching, in which with the aircraft engine, the aircraft wings, the aircraft head, and the aircraft tail respectively extracted in the steps S711-S714 as the images to be searched, and the engine, the wings, the head and the tail of the standard aircraft image as templates, 4 similarity degrees between the images to be searched and the templates are calculated, a minimum value of the 4 similarity degrees is removed, and an average of the remaining 3 similarity degrees of the 4 similarity degrees is taken as a local template similarity degree.

21. The method of claim 20, characterized in that the step S73 further comprises:

if at least two of the model similarity degree, the global template similarity degree and the local template similarity degree are larger than or equal to a first verification threshold, it is deemed that the identity verification is passed, or, if each of the model similarity degree, the global template similarity degree and the local template similarity degree is larger than a second verification threshold, it is deemed that the identity verification is passed.

22-42. -42 (canceled)

43. A system for guiding an aircraft docking and recognizing an aircraft model based on machine vision, characterized in that the system comprises:

a photographing device, a central processing device and a display device, wherein the central processing device is configured to perform;

step S1, a step for configuring an aircraft berthing scene, in which a monitored scene is divided into regions for separate data processing;

step S2, an image pre-processing step, in which a captured image is pre-processed;

step S3, an aircraft capturing step, in which it is determined that an aircraft exists in the image by recognizing an engine and front wheels of the aircraft in the image;

step S4, an aircraft tracking step, in which the image of the engine and the front wheels of the aircraft recognized in step S3 is continuously tracked and updated in real time;

step S5, an aircraft positioning step, in which the aircraft is positioned in real time, and a deviation degree of the aircraft from a guide line and a distance between the aircraft and a stop line are accurately determined; and

step S6, an information displaying step, in which the deviation degree of the aircraft from the guide line and the distance between the aircraft and the stop line determined in step S5 are outputted and displayed.

44. The system of claim 43, characterized in that after the step S3, the central processing device is configured to perform:

step S7, an aircraft identification and identity verification step and further comprises;

step S71, parameter verification, in which parameters of the aircraft in the image are extracted and compared with parameters of a corresponding model type preset in a database, to obtain a model similarity degree;

step S72, template matching, in which the image is compared with a model template preset in the database, to obtain a template similarity degree; and

step S73, comprehensive judgment, in which when the model similarity degree and the template similarity degree are larger than or equal to a verification threshold, it is deemed that the identity verification is passed,

wherein the step S71 further comprises;

step S711, parameters of the aircraft engine in the image are extracted and compared with parameters of the aircraft engine of a corresponding model type preset in a database, to obtain a first ratio;

step S712, the aircraft wing parameters in the image are extracted and compared to aircraft wing parameters of the corresponding model preset in the database, to obtain a second ratio;

step S713, parameters of aircraft head in the image are extracted and compared with parameters of aircraft head corresponding to the model type preset in the database, to obtain a third ratio;

step S714, parameters of the aircraft tail in the image are extracted and compared with aircraft tail parameters preset to the corresponding model type in the database, to obtain a fourth ratio; and

step S715, a minimum value and a maximum value among the first ratio, the second ratio, the third ratio, and the fourth ratio are taken, and the ratio of minimum/maximum is taken as the model similarity degree;

the step S72 further comprises;

step S721, global template matching, in which a global template similarity degree is calculated by taking the whole image as the image to be searched and the standard aircraft image as the template;

step S722, local template matching, in which with the aircraft engine, the aircraft wings, the aircraft head, and the aircraft tail respectively extracted in the steps S711-S714 as the images to be searched, and the engine, the wings, the head and the tail of the standard aircraft image as templates, 4 similarity degrees between the images to be searched and the templates are calculated, a minimum value of the 4 similarity degrees is removed, and an average of the remaining 3 similarity degrees of the 4 similarity degrees is taken as a local template similarity degree;

the step S73 further comprises;

if at least two of the model similarity degree, the global template similarity degree and the local template similarity degree are larger than or equal to a first verification threshold, it is deemed that the identity verification is passed, or, if each of the model similarity degree, the global template similarity degree and the local template similarity degree is larger than a second verification threshold, it is deemed that the identity verification is passed.