System and method for aircraft docking guidance and aircraft type identification

• US 10,255,520 B2
• Filed: 07/02/2015
• Issued: 04/09/2019
• Est. Priority Date: 08/01/2014
• Status: Active Grant

• Current Assignee: Shenzhen Cimc-Tianda Airport Support, Ltd. (China International Marine Containers (Group) Co., Ltd.)
• Original Assignee: China International Marine Containers Limited (China International Marine Containers Company Limited), Shenzhen Cimc-Tianda Airport Support, Ltd. (China International Marine Containers (Group) Co., Ltd.)
• Inventors: Deng, Lan, Chang, Shaomin, Xiang, Wei, Yang, Yuefeng, Wang, Haibin, Liu, Haiqiu
• Primary Examiner(s): Bekele, Mekonen T
• Application Number:
  US15/329,990
• Publication Number:
  US 20170262732A1
• Time in Patent Office: 1,377 Days
• CPC Class Codes: B64F 1/00 Ground or aircraft-carrier-deck installations
• CPC Class Codes: B64F 1/002 Taxiing aids
• CPC Class Codes: G01S 17/06 Systems determining position data of a target
• CPC Class Codes: G01S 17/88 Lidar systems specially adapted for specific applications
• CPC Class Codes: G06V 10/10 Image acquisition document image scanning and transmission H04N1/00; control of digital cameras H04N23/60
• CPC Class Codes: G06V 10/20 Image preprocessing
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G06V 20/588 Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road
• CPC Class Codes: G08G 5/06 for control when on the ground
• CPC Class Codes: G08G 5/065 Navigation or guidance aids, e.g. for taxiing or rolling
• CPC Class Codes: H04N 7/00 Television systems details H04N3/00, H04N5/00; methods or arrangements, for coding, decoding, compressing or decompressing digital video signals H04N19/00; selective content distribution H04N21/00

Abstract

A system and method for aircraft docking guidance and aircraft type identification. The method is executed in the system for aircraft docking guidance and aircraft type identification comprising a machine vision sub-system, a laser scanning sub-system and a fusion module. The method includes: in step 1000, obtaining, by the machine vision sub-system, an image via image capturing means, and calculating a first aircraft front wheel position therefrom; in step 2000, obtaining, by the laser scanning sub-system, the position of the nose of an aircraft via laser scanning means, and calculating a second aircraft front wheel position; in step 3000, fusing the first aircraft front wheel position and the second aircraft front wheel position according to a fusion rule, to obtain deviation of an aircraft front wheel.

Claims

1. An aircraft docking guidance and type identification method, performed by an aircraft docking guidance and type identification system which comprises a machine vision subsystem, a laser scanning subsystem and a fusion module, wherein the method comprises:

obtaining images, by the machine vision subsystem, through image capturing and obtaining a first position of a front wheel of an aircraft by calculation based on the images;

obtaining a position of an aircraft nose of the aircraft, by the laser scanning subsystem, by means of laser scanning, and obtaining a second position of the front wheel of the aircraft by calculation; and

performing fusion with respect to the first position and the second position of the front wheel of the aircraft according to a fusion rule to identify deviation of the front wheel of the aircraft

wherein the fusion rule comprises;

for a case where the front wheel of the aircraft deviates from a guidance line;

determining whether a distance between x coordinates of the first position and the second position of the front wheel of the aircraft is smaller than or equal to a threshold;

if the distance is smaller than or equal to the threshold, obtaining the deviation degree of the front wheel of the aircraft calculated by the machine vision subsystem;

if the distance is greater than the threshold, performing smooth filtering on the first position and the second position of the front wheel of the aircraft to provide deviation information which indicates how the front wheel of the aircraft deviates from the guidance line;

for a case where there is a distance remaining from the front wheel of the aircraft to a stopping line;

determining whether there is an abnormal fluctuation in the second position of the front wheel of the aircraft;

if there is not such abnormal fluctuation, obtaining the second position of the front wheel of the aircraft as the distance remaining from the front wheel of the aircraft to the stopping line;

if there is such abnormal fluctuation, obtaining an average difference value of N previous difference values between positions of the front wheel of the aircraft obtained by the machine vision subsystem and the laser scanning subsystem, and calculating a total of a position of the front wheel of the aircraft obtained by the machine vision subsystem this time and the average difference value as the distance remaining from the front wheel of the aircraft to the stopping line.

2. The method according to claim 1, wherein the step of obtaining images further comprises:

performing an aircraft docking scene setting step to divide a monitoring scene into different information process functional areas;

performing an image preprocess step to perform preprocess on the captured images;

performing an aircraft capture step to recognize engines and the front wheel of the aircraft from the images so as to conform that an aircraft appears in the images;

performing an aircraft tracking step to continuously track the images of the engines and the front wheel of the aircraft captured in the aircraft capture step and update the images of the engines and the front wheel of the aircraft in real time;

performing an aircraft locating step to locate the aircraft in real time and accurately determine a deviation degree of the aircraft with respect to a guidance line and how far the aircraft is from a stopping line;

wherein the image preprocess step further comprises;

determining whether the images belong to a low illumination image, a high illumination image or a normal illumination image according to average gray levels of the images, performing a low illumination image process on low illumination images, and performing a high illumination image process on high illumination images;

determining whether each of the normal illumination images is a normal image according to a variance of the image; and

for images which are not normal images, determining whether each of the images is an image of rain or snow or an image of fog, performing a rain-or-snow-image process step on the image of rain or snow, and performing a fog-image process step on the image of fog.

3. The method according to claim 2, wherein the low illumination image process comprises:

g(x,y)=f(x,y)+af(255−

f (x,y))

wherein f (x, y) represents an original image, (x, y) are coordinates of individual pixel points in the image, g (x, y) represents the processed image, and a is a low illumination image process parameter;

or wherein the rain-or-snow-image process step comprises;

finding pixels to be processed which are corrupted by rain or snow using a photometric model;

for pixels to be processed in a current image, extracting brightness values of corresponding pixels in preceding and succeeding images which are adjacent to the current image, determining whether the corresponding pixels in the preceding and succeeding images which are adjacent to the current image are pixels to be processed according to the brightness values;

if the corresponding pixels in the preceding and succeeding images which are adjacent to the current image are pixels to be processed, obtaining an average value of brightness values of all pixels which are adjacent to the pixels to be processed in the current image, and replacing the brightness values of the pixels to be processed in the current image with the average value; and

if the corresponding pixels in the preceding and succeeding images which are adjacent to the current image are not pixels to be processed, replacing the brightness values of the pixels to be processed in the current image with a minimum value among the brightness values of the corresponding pixels in the preceding and succeeding images which are adjacent to the current image or an average value of two minimum values among the brightness values of the corresponding pixels in the preceding and succeeding images which are adjacent to the current image;

or wherein the fog-image process step is performed by homomorphic filtering.

4. The method according to claim 2, wherein the aircraft capture step further comprises:

performing a background removal step to simulate a dynamic distribution of background of the filed using a single-Gaussian background model to establish a background model, and removing the background using a difference between current images and the background model to obtain a foreground area;

performing a shadow removal step to calculate gray levels of the foreground area to identify a maximum gray level g max and a minimum gray level g min, and perform a shadow removal on an area which has a gray level smaller than T=g min+(g max−

g min)*0.5;

performing an area classification step, establishing a standard front aircraft area template, extracting a target area by change detection, calculating a vertical projection curve of the area, calculating a correlation coefficient between the vertical projection curve and a vertical projection curve of the standard front aircraft area template, and if the correlation coefficient is greater than or equals to a classification threshold, determining that the target area corresponds to an aircraft; and

performing a feature verification step to further verify whether the target area corresponds to the aircraft by detecting the engines and the front wheel of the captured aircraft.

5. The method according to claim 4, wherein the feature verification step further comprises:

(a) performing an extremely black area extraction step to obtain a statistical gray level histogram for the target area of the current image, identify a maximum gray level and a minimum gray level within a gray level range of 1%˜

99%, and extract the most black portion of the image based on a preset extremely black judgment threshold, the maximum gray level and the minimum gray level to obtain an extremely black area;

(b) performing quasi-circle detection to extract all outer boundaries of the extremely black area, and for each of the boundaries, calculate barycentric coordinates of the boundary using moment of the boundary, wherein the ji-th moment of the boundary is defined as;

6. The method according to claim 5, wherein in step (c), for M detected quasi-circular areas, a similarity Similarity_ij, between a i-th quasi-circular area and a j-th quasi-circular area is:

Similarity_ij=|Height_i−

Height_j|*|Radius_i−

Radius_j|

wherein height represents a barycenter height, Radius represent a radius, and when the similarity Similarity_ij is smaller than a preset similarity threshold, it is determined that the i-th quasi-circular area and the j-th quasi-circular area correspond to engines of the aircrafts.

7. The method according to claim 6, wherein in step (c), if no aircraft engine is detected, iterative detection is performed to perform steps (a) to (c) by increasing the extremely black judgment threshold, the circle judgment threshold, and the similarity threshold;

if there still is no aircraft engine detected, a 7*7 circular template is applied on all the extremely black areas to perform an opening operation and then steps (b) to (c) are performed;

if there still is no aircraft engine detected, the iterative detection is performed for twice; and

if there still is no aircraft engine detected, it is determined that there is no engine existing in the image.

8. The method according to claim 7, wherein the extremely black judgment threshold, the circle judgment threshold, and the similarity threshold are increased by 0.05, 0.5and 20, respectively.

9. The method according to claim 5, wherein step (d) further comprises:

in an searching area of the image, quantizing 256 gray levels into 64 gray levels, searching a first peak and a first valley in the quantized 64 gray level histogram, wherein a best peak position BestPeak and a best valley position BestValley in the original 256 gray level histogram are defined as;

10. The method according to claim 9, wherein the aircraft front wheel deviation degree calculation step further comprises:

according to the first position (x₀,y₀) of the front wheel of the aircraft obtained in the step of detecting the front wheel of the aircraft, using a relationship between coordinates of the position and the guidance and stopping lines, obtaining a straight line equation of the guidance line as y₁=k₁x₁+b₁), and a straight line equation of the stopping line as y₂=k₂x₂+b₂, wherein a distance from the coordinates of the position to a straight line is;

11. The method according to claim 10, wherein the aircraft front wheel deviation degree calculation step further comprises:

determining whether |d₁|>

width/2, wherein width is a threshold equal to a width of the front wheel of the detected aircraft;

if |d₁|>

width/2, determining that the aircraft deviates from the guidance line.

12. The method according to claim 9, wherein the aircraft front wheel actual distance calculation step further comprises:

establishing a correspondence between image coordinates and geodetic coordinates;

obtaining image coordinates of calibration points used during the filed setting in the aircraft docking scene setting step, conducting quadratic curve fitting using the image coordinates by a least square method to obtain a curve line equation y=ax²+bx+c, wherein x is a distance on an image, y is an actual distance;

projecting the position of the front wheel of the aircraft on the image onto the guidance line along a direction of the stopping line, calculating an Euclidean distance between a projection point and a stopping point as x, and obtaining an actual distance from the front wheel of the aircraft to the stopping line using y=ax²+bx+c.

13. The method according to claim 2, wherein the method further comprises an aircraft identification and identity verification step performed after the aircraft capture step and which comprises:

performing parameter verification;

extracting aircraft parameters in the images, and comparing the aircraft parameters with type data which is pre-stored in a database to obtain a type similarity parameter;

performing template matching;

comparing the images with type templates which are pre-stored in the database to obtain a template similarity parameter; and

performing a synthetic judgment;

when the type similarity parameter and the template similarity are greater than or equal to a verification threshold, determining that identity verification is passed.

14. The method according to claim 13, wherein the parameter verification step further comprises:

(a) extracting an aircraft engine parameter in the images, and comparing the aircraft engine parameter with an aircraft engine parameter of a corresponding type which is pre-stored in the database to obtain a first ratio;

(b) extracting an aircraft wing parameter in the images, and comparing the aircraft wining parameter with an aircraft wing parameter of the corresponding type which is pre-stored in the database to obtain a second ratio;

(c) extracting an aircraft head parameter in the images, and comparing the aircraft head parameter with an aircraft head parameter of the corresponding type which is pre-stored in the database to obtain a third ratio;

(d) extracting an aircraft tail fin parameter in the images, and comparing the aircraft tail fin parameter with an aircraft tail fin parameter of the corresponding type which is pre-stored in the database to obtain a fourth ratio; and

(e) identifying the maximum one and the minimum one among the first ratio, the second ratio, the third ratio and the fourth ratio, and calculating a ratio of the minimum one to the maximum one as the type similarity parameter;

wherein the template matching step further comprises;

performing global template matching;

with a whole image as an image to be searched and a standard aircraft image as a template, calculating a global template similarity parameter; and

performing local template matching;

with images of an aircraft engine, an aircraft wing, an aircraft head and an aircraft tail fin extracted according to steps (a) to (d) as images to be searched, and an aircraft engine, an aircraft wing, an aircraft head and an aircraft tail fin in the standard aircraft image as templates, calculating four similarities between the images to be searched and the templates, removing the minimum one among the four similarities and calculating an average value of the remaining three similarities as a local template similarity parameter;

wherein the synthetic judgment step further comprises;

if at least two of the type similarity parameter, the global template similarity parameter and the local template similarity parameter are greater than or equal to a first verification threshold, determining that the identity verification is passed;

or

if all of the type similarity parameter, the global template similarity parameter and the local template similarity parameter are greater than a second verification threshold, determining that the identity verification is passed.

15. The method according to claim 1, wherein the step of obtaining a position of an aircraft nose of the aircraft further comprises:

in a capture step, conducting laser scanning to horizontally scan a position where an aircraft head of an aircraft is expected to appear, obtaining pieces of echo data based on the laser scanning, and according to a judgment condition, determining whether the aircraft appears based on the echo data;

in a locating step, after the aircraft appears, conducting the laser scanning to horizontally scan the aircraft head, obtaining the echo data based on the laser scanning, and determining a position of an aircraft nose of the aircraft according to the echo data; and

in a tracking and guidance step, while the aircraft is traveling, tracking the position of the aircraft nose by adjusting a vertical scanning angle of the laser scanning.

16. The method according to claim 15, wherein prior to the capture step, the method further comprises a zero point calibration step which comprises a device mounting zero point calibration step performed when the laser scanning system is first installed, wherein the device mounting zero point calibration step comprises:

a horizontal zero point measurement step which comprises;

horizontally scanning, by the scanning system, a calibration area on the ground with a vertical scanning angle fixed, wherein the calibration area is divided equally into N small regions;

measuring distances from a boundary point of each of the small regions, and finding a minimum value among the obtained distance values;

determining an extended area with a boundary point corresponding to the minimum value as a center; and

identifying a point corresponding to a minimum distance value as a horizontal zero point; and

a vertical zero point measurement step which comprises;

conducting distance measurement at a vertical scanning angle β

₁ by the scanning system to obtain a first distance value L₁, conducting the distance measurement again by adjusting the vertical scanning angle β

₁ by an angle λ

to obtain a second distance value L₂, and calculating β

₁ using the following equation;

L₁*sin β

₁=L₂*sin(β

₁−

λ

)

wherein a straight line where L1 is located is taken as a hypotenuse, a projection line of the hypotenuse on the ground is taken as a leg, and a vertex of a right-angled triangle formed by the hypotenuse and the leg, an included angle between which is β

₁, is taken as a vertical zero point;

wherein the zero point calibration step further comprises a zero point correction step which comprises;

after the device mounting zero point calibration step is finished, adjusting a scanning angle of the scanning system from a zero point step by step until a preset reference object is found, and recording a scanning angle at this time as a correction angle; and

when the scanning system is restarted, adjusting the scanning angle backwards by the correction angle to find the zero point after the preset reference object is found by the scanning system.

17. The method according to claim 15, wherein the judgment condition further comprises:

determining whether a ratio of the number of points of interest, which are continuous and one of which corresponds to a minimum distance value, to the total number of the pieces of echo data is greater than a threshold;

calculating an aircraft width based on the total number of the echo data, and determining whether the aircraft width is not less than a width threshold; and

calculating an aircraft height and determining whether the calculated aircraft height is within a predetermined range;

wherein the distance values of the points of interest are within a specified range at two sides of a prediction length;

or wherein in the capture step, when the echo data is obtained, median filtering is performed on the echo data and then whether the aircraft appears is determined;

or wherein the locating step further comprises;

extracting from the echo data target points falling on the aircraft head, and converting the target points from polar coordinate data into three-dimensional spatial data;

identifying from the target points a point having a minimum Y-direction value as a closest point;

conducting curve fitting to fit the target points to a curve and identify a vertex of the curve; and

determining one of the closest point and the vertex as the position of the aircraft nose according to a difference between an X-value of the closest point and an X-value of the vertex;

or wherein the tracking and guidance step further comprises;

scanning the position of the aircraft nose at a predetermined vertical scanning angle while the aircraft is travelling;

calculating a current aircraft nose point according to the echo data; and

vertically scanning the current aircraft nose point, and calculating a variation amount of the vertical scanning angle according to a vertex of a obtained parabola according to the scanning, and changing the vertical scanning angle according to the variation amount, or calculating the variation amount of the vertical scanning angle according to the current aircraft nose point and changing the vertical scanning angle according to the variation amount;

or wherein the method further comprises an aircraft type identification step;

wherein the aircraft type identification step comprises one or more of aircraft nose height verification, aircraft cabin width verification, verification of a head aerial view profile, verification of a head side view profile and aircraft engine verification;

the aircraft nose height verification comprises;

if a difference between the aircraft nose height and a preset aircraft nose height is within a preset range, determining that the aircraft nose height verification is passed;

the aircraft cabin width verification comprises;

finding two points from the echo data, a difference between X coordinates of which is the maximum, with a straight line distance between the two points as an aircraft cabin width, if the aircraft cabin width is greater than a product of a theoretic aircraft cabin width and a width coefficient, determining that the aircraft cabin width verification is passed;

the verification of the head aerial view profile comprises;

bringing the echo data (x_i, y_i,z_i) obtained by horizontal laser scanning into a preset side view profile equation y=f″

(x), calculating a horizontal fit point error Δ

d_i^h=|f″

(x_i)−

y_i|, identifying a maximum horizontal fit error D_H=max (Δ

d_i^h), and if D_H <

Δ

D_H,determining that the verification of the head aerial view profile is passed, wherein Δ

D_H is a threshold for the verification of the head aerial view profile, i is the serial number of individual one of target points in the echo data, and n is the number of fittings of the aerial view profile equation;

the verification of the head side view profile comprises;

bringing the echo data (x_i,y_i,z) obtained by vertical laser scanning into a preset side view profile equation z=g^m(y), calculating a vertical fit point error Δ

d_j^v=|g^m(y_i)−

z_i|, identifying a maximum vertical fit error D_v=max(Δ

d_j^v), and if D_v <

Δ

D_v, determining that the verification of the head side view profile is passed, wherein Δ

D_v is a threshold for the verification of the head side view profile, i is the serial number of individual one of target points in the echo data, and m is the number of fittings of the side view profile equation;

the aircraft engine verification comprises;

calculating positions of engines according to the position of the aircraft nose, and conducting laser scanning to horizontally and vertically scan the positions of the engines;

converting the echo data obtained by the horizontal scanning into x-y-z three dimensional coordinates, finding a closest coordinate point which is closest to a theoretic engine center, and finding points contiguous to the closest coordinate point to obtain a point set, and calculating a distance between a leftmost point and a rightmost point in the point set as an engine width and a middle point between the leftmost point and the rightmost point as a horizontal coordinate of an engine center;

converting the echo data obtained by the vertical scanning into x-y-z three dimensional coordinates, finding a closest coordinate point which is closest to the theoretic engine center, and finding points contiguous to the closest coordinate point to obtain a point set, and calculating a distance between a uppermost point and a lowermost point in the point set as an engine height, and a middle point between the uppermost point and the lowermost point as a height of the engine center from ground;

determining whether the number of the engines is consistent with a preset number, and if the number of the engines is not consistent with the preset number, determining that the aircraft engine verification is not passed;

determining whether a difference between the horizontal coordinate of the engine center and a standard value exceeds a threshold or whether a difference between the height of the engine center from ground and a standard value exceeds a threshold, and if the difference between the horizontal coordinate of the engine center and a standard value exceeds a threshold or the difference between the height of the engine center from ground and a standard value exceeds a threshold, determining that the aircraft engine verification is not passed;

determining whether a difference between the engine width and a standard value exceeds a threshold, or whether a difference between the engine height and a standard value exceeds a threshold, and if the difference between the engine width and a standard value exceeds a threshold, or the difference between the engine height and a standard value exceeds a threshold, determining that the aircraft engine verification is not passed.

18. The method according to claim 1, wherein the fusion rule further comprises:

establishing a not-empty frame of discernment D={Ø

,h,h,H}using a fusion method based on D-S theory, wherein h represents that the aircraft type is right, h represents that the aircraft type is wrong, H ={h}represents that the aircraft type might be right or wrong and cannot be determined temporarily, and Ø

represents an impossible event;

establishing a mass function m₁ (⋅

) of the machine vision subsystem and a mass function m₂(⋅

) of the laser scanning subsystem, and the following condition is met;

m₁(Ø

)=0, m₂(Ø

)=0,Σ

m₁(⋅

)=1,Σ

m₂ (⋅

)=1,

19. An aircraft docking guidance and type identification device, comprising:

a machine vision subsystem configured to obtain images, through image capturing and obtain a first position of a front wheel of an aircraft by calculation based on the images;

a laser scanning subsystem configured to obtain a position of an aircraft nose of the aircraft by means of laser scanning, and obtain a second position of the front wheel of the aircraft by calculation; and

a fusion device configured to perform fusion with respect to the first position and the second position of the front wheel of the aircraft according to a fusion rule to identify deviation of the front wheel of the aircraft,

wherein the fusion rule comprises;

for a case where the front wheel of the aircraft deviates from a guidance line;

determining whether a distance between x coordinates of the first position and the second position of the front wheel of the aircraft is smaller than or equal to a threshold;

if the distance is smaller than or equal to the threshold, obtaining the deviation degree of the front wheel of the aircraft calculated by the machine vision subsystem;

if the distance is greater than the threshold, performing smooth filtering on the first position and the second position of the front wheel of the aircraft to provide deviation information which indicates how the front wheel of the aircraft deviates from the guidance line;

for a case where there is a distance remaining from the front wheel of the aircraft to a stopping line;

determining whether there is an abnormal fluctuation in the second position of the front wheel of the aircraft;

if there is not such abnormal fluctuation, obtaining the second position of the front wheel of the aircraft as the distance remaining from the front wheel of the aircraft to the stopping line;

if there is such abnormal fluctuation, obtaining an average difference value of N previous difference values between positions of the front wheel of the aircraft obtained by the machine vision subsystem and the laser scanning subsystem, and calculating a total of a position of the front wheel of the aircraft obtained by the machine vision subsystem this time and the average difference value as the distance remaining from the front wheel of the aircraft to the stopping line.