Runway surveillance system and method

• US 9,483,952 B2
• Filed: 08/25/2008
• Issued: 11/01/2016
• Est. Priority Date: 08/24/2007
• Status: Active Grant

• Current Assignee: Changi Airport Group Pte Limited (Government of Singapore)
• Original Assignee: Stratech Systems Limited (In Liquidation) (The Stratech Group Ltd.)
• Inventors: Chew, Khien Meow David
• Primary Examiner(s): TODD, GREGORY G
• Application Number:
  US12/674,966
• Publication Number:
  US 20110063445A1
• Time in Patent Office: 2,990 Days
• Field of Search: 348/143, 348/144, 348/148, 348/159
• US Class Current: 1/1
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/30184 Infrastructure
• CPC Class Codes: G06T 2207/30236 Traffic on road, railway or crossing
• CPC Class Codes: G06T 7/0008 checking presence/absence
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G08B 13/19652 Systems using zones in a single scene defined for different treatment, e.g. outer zone gives pre-alarm, inner zone gives alarm
• CPC Class Codes: G08B 13/19691 Signalling events for better perception by user, e.g. indicating alarms by making display brighter, adding text, creating a sound
• CPC Class Codes: G08G 5/0026 located on the ground
• CPC Class Codes: G08G 5/04 Anti-collision systems
• CPC Class Codes: G08G 5/06 for control when on the ground
• CPC Class Codes: G08G 5/065 Navigation or guidance aids, e.g. for taxiing or rolling

Abstract

A surveillance system and method for detecting a foreign object, debris, or damage (FOD) on a runway. The system comprises one or more cameras for capturing images of the runway; and an image processing system for detecting the FOD on the runway based on adaptive image processing of the images captured by the cameras; wherein the surveillance system is adaptively operable for FOD detection under both day and night ambient light conditions without assisted illumination such as infrared or laser illuminators.

Claims

1. A surveillance system for detecting a foreign object, debris, or damage (FOD) on a runway, comprising:

one or more cameras for capturing electronic images of the runway; and

an image processing system for detecting the FOD on the runway based on adaptive image processing of the electronic images captured by the cameras and for applying image enhancement to eliminate areas of the captured electronic images having gradual grayscale change in a direction parallel to the runway, to eliminate runway white lines and to enhance features having high grayscale gradient in the direction parallel to the runway in the captured electronic images,

wherein the image processing system is configured to subject the captured electronic images to background learning after the image enhancement is applied, and generate a composite background edge map during the background learning, the composite background edge map comprising an adaptive background edge map, a previously learned and saved day or night background edge map, and a seasonal marking map generated for a particular season or weather condition,

wherein the surveillance system is adaptively operable for FOD detection under both day and night ambient light conditions without assisted illumination including infrared or laser illuminators,

wherein the image processing system is configured to generate a robust edge map comprising selected pixels from the captured electronic images,

wherein the image processing system is configured to compare the composite background edge map and the robust edge map, and remove background edges to extract a suspected edge map of FOD.

2. The surveillance system according to claim 1, wherein the captured electronic images are enhanced by using a high pass filter, a Sobel X from left_to_right filter and a Sobel X from right_to_left filter, or a Scharr X filter.

3. The surveillance system according to claim 1, wherein the image processing system is configured to determines if an instant of processing is a day-time or night-time;

and detect an abnormal light condition from the captured electronic images during night-time, wherein the abnormal light condition is due to aircraft landing or aircraft taking off or ground vehicle movement.

4. The surveillance system according to claim 3, wherein detecting of the abnormal light condition comprises global histogram and statistical analysis to compare each captured image with one or more preceding images and identifies the abnormal light condition based on a change in intensity with reference to a threshold value.

5. The surveillance system according to claim 3, wherein captured electronic images for which the abnormal light condition are detected are ignored from further processing.

6. The surveillance system according to claim 1, wherein the image processing system is configured to apply temporal filtering to a stack of pixel level edge maps to obtain the robust edge map which consists only of pixels that have accumulated to pass a threshold.

7. The surveillance system according to claim 6, wherein the image processing system is configured to subject the robust edge map to adaptive background learning, the adaptive background learning comprising:

comparing background edge images obtained at previous instants with current image;

identifying slow-change features on the runway; and

updating the background edge image with the slow changing features.

8. The surveillance system according to claim 1, wherein the edge filtering filters unwanted edges related to the environmental conditions from the suspected edge map, and computes edge parameters of FOD from the suspected edge map.

9. The surveillance system according to claim 8, wherein the environmental conditions include day to night transition, or night to day transition, weather conditions, rain, smoke or cloud.

10. The surveillance system according to claim 1, wherein the image processing system is configured to overlay an FOD graphic on a suspected region of the runway on a video display to alarm an operator at a control tower or control room of FOD detection.

11. The surveillance system according to claim 10, wherein one or more of the cameras, or one or more additional cameras are arranged for zooming on to the suspected region for visual verification.

12. The surveillance system according to claim 1, wherein the image processing system is configured to classify the FOD.

13. The surveillance system according to claim 1, wherein the one or more cameras are one or more static cameras, one or more non-static cameras or a combination of both static and non static cameras.

14. The surveillance system according to claim 1, wherein the one or more cameras are placed on one side of the runway.

15. The surveillance system according to claim 1, wherein the one or more cameras are placed on either sides of the runway in a staggered manner.

16. The surveillance system according to claim 1, wherein on a condition that one or more cameras fail to function, respective adjacent ones of the cameras are operable to cover areas covered by the failed cameras.

17. The surveillance system according to claim 1, wherein the one or more cameras are one or more monochrome cameras, one or more color cameras or both.

18. The surveillance system according to claim 1, further comprising one or more night vision cameras.

19. The surveillance system according to claim 1, wherein a runway surface is divided into a plurality of segments, and one or more non-static cameras sequentially scan the runway surface segment-by-segment for FOD detection.

20. The surveillance system according to claim 1, wherein a static camera detects respective locations of aircraft take off and landing on the runway such that a non-static camera is directed to first scan runway segments in the respective locations of aircraft landing or take off to reduce FOD detection time.

21. The surveillance system according to claim 1, wherein the image processing system is configured to apply temporal filtering to filter out rain clutter in runway scene images by recognising characteristics of rain motion clutter and based on the motion clutter due to rain occurring across the entire runway.

22. The surveillance system according to claim 1, wherein the image processing system is configured to apply temporal filtering to filter out snow clutter in runway scene images by recognising characteristics of snow motion clutter and based on the motion clutter due to snow occurring across the entire runway.

23. The surveillance system according to claim 1, wherein the image processing system is configured to make use of markers or runway edge lights located along a longitudinal (horizontal) direction on the runway and on a same vertical distance from a side of the runway for runway scene calibration to map pixels on the images of the runway to precise co-ordinates on a real-world co-ordinate frame (including WGS84 or Airport Grid).

24. The surveillance system according to claim 1, wherein the image processing system is configured to make use of two parallel horizontal runway lines on each side of a runway middle line and the runway middle line to derive two vertical pixel mapping ratios for runway scene calibration to map pixels on the images on the runway to precise co-ordinates on a real-world co-ordinate frame (including WGS84 or Airport Grid).

25. The surveillance system according to claim 1, wherein the image processing system is configured to make use of monoscopic vision and a calibrated runway scene image captured by a monoscopic camera to determine position and range of the FOD on the runway.

26. The surveillance system according to claim 1, wherein the image processing system is configured to make use of a position and range of the FOD determined by a static camera and a calibrated runway scene image to automatically control a non-static camera (including a pan tilt zoom camera) to pan and/or tilt and/or zoom and/or focus onto a FOD to obtain telephoto images of the FOD with sufficient details to enable a verification of detected FOD or to filter a false alarm.

27. The surveillance system according to claim 1, wherein the image processing system is configured to make use of stereo vision using a pair of surveillance cameras to cover a same segment of the runway so that FOD range and position can be computed from a difference image obtained by comparing images as captured by the pair of surveillance cameras covering the same segment of surveillance (field of view) on the runway.

28. The surveillance system according to claim 1, wherein the image processing system is configured to adaptively estimate one or more threshold values for optimal FOD edge extraction for different environmental conditions;

and generate a pixel level edge map using a statistical method based on progressively learned background image edge map to determine a grayscale lookup table (LUT) to be used to generate pixel level threshold map.

29. The surveillance system according to claim 1, wherein the image processing system is configured to perform edge filtering on the suspected edge map to locate and filter out light reflection on the runway resulting from environmental conditions.

30. The surveillance system according to claim 1, wherein the image processing system is configured to detect motion in the captured electronic images and to subject captured electronic images devoid of motion to the background learning.