Moving object detection using a mobile infrared camera

• US 8,446,468 B1
• Filed: 06/19/2008
• Issued: 05/21/2013
• Est. Priority Date: 06/19/2007
• Status: Active Grant

• Current Assignee: University of Southern California
• Original Assignee: University of Southern California
• Inventors: Medioni, Gerard, Lin, Yuping, Pai, Cheng-Hua
• Primary Examiner(s): Feild, Lynn
• Assistant Examiner(s): ELFERVIG, TAYLOR A
• Application Number:
  US12/142,442
• Time in Patent Office: 1,797 Days
• Field of Search: 348/144
• US Class Current: 348/144
• CPC Class Codes: B64D 45/08 optical
• CPC Class Codes: G06T 2207/10048 Infrared image
• CPC Class Codes: G06T 2207/20076 Probabilistic image processing
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G06T 7/254 involving subtraction of images
• CPC Class Codes: G06T 7/337 involving reference images or patches
• CPC Class Codes: G06V 20/13 Satellite images
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: G08G 5/02 Automatic approach or landing aids, i.e. systems in which flight data of incoming planes are processed to provide landing data landing aids fitted in or to aircraft B64D45/04; visual or acoustic landing aids B64F1/18
• CPC Class Codes: H04N 23/6811 based on the image signal
• CPC Class Codes: H04N 23/683 performed by a processor, e.g. controlling the readout of an image memory
• CPC Class Codes: H04N 5/33 Transforming infrared radiation cameras or camera modules for generating image signals from infrared radiation H04N23/20; circuitry of SSIS for transforming infrared radiation into image signals H04N25/20

Abstract

Systems and methods for moving object detection using a mobile infrared camera are described. The methods include receiving multiple frames, each frame including an image of at least a portion of a planar surface, stabilizing two consecutive frames of the plurality of frames, the stabilizing comprising determining a transformation mapping a succeeding frame of the two consecutive frames to a preceding frame of the two consecutive frames and based on the transformation, warping the two consecutive frames to a reference frame of the plurality of frames, and detecting a movement of an object in the two consecutive frames, the movement based on a change in positions of the object in the preceding frame and the successive frame.

Claims

1. A computer-implemented method comprising:

receiving a plurality of frames representing images including at least portions of a planar surface, each image captured at a separate location;

for any two consecutive frames in the plurality of frames, determining a transformation mapping a portion of the planar surface included in a first frame of the two consecutive frames to a corresponding portion of the planar surface included in a second frame of the two consecutive frames;

updating from a first reference frame from the plurality of frames to a second reference frame from the plurality frames in accordance with a measure;

transforming the plurality of frames to the respective first and second reference frames based on the determined transformations to stabilize the plurality of frames; and

providing the stabilized frames as an output;

wherein the measure comprises a ratio of an edge length for a frame from the plurality of frames before and after transformation to the first reference frame.

2. The method of claim 1, wherein the ratio comprises a ratio of an edge length of a portion of the planar surface.

3. The method of claim 1, wherein determining the transformation comprises identifying feature points in the two consecutive frames.

4. The method of claim 3, wherein the feature points are determined using Scale Invariant Feature Transform.

5. The method of claim 1, wherein the planar surface is a polygon.

6. The method of claim 5, wherein the polygon consists of four sides.

7. The method of claim 1, wherein the plurality of frames are captured by an infrared camera.

8. The method of claim 7, wherein the infrared camera is mounted on an airplane.

9. The method of claim 8, wherein the planar surface is a runway on which the airplane is to land.

10. The method of claim 1, further comprising:

performing motion detection using background modeling on the stabilized frames, wherein the background modeling identifies an image in the first reference frame;

identifying an image in a stabilized frame that corresponds to the image in the first reference frame by comparing images in the stabilized frame with images in the first reference frame; and

upon identifying that a position of the corresponding image in the stabilized frame is different from the image in the first reference frame, determining that the corresponding image represents a moving object.

11. A system comprising:

a data processing apparatus; and

a non-transitory medium configured and arranged to cause the data processing apparatus to perform operations comprising;

receiving a plurality of frames representing images including at least portions of a planar surface, each image captured at a separate location;

for any two consecutive frames in the plurality of frames, determining a transformation mapping a portion of the planar surface included in a first frame of the two consecutive frames to a corresponding portion of the planar surface included in a second frame of the two consecutive frames;

updating from a first reference frame from the plurality of frames to a second reference frame from the plurality frames in accordance with a measure;

transforming the plurality of frames to the respective first and second reference frames based on the determined transformations to stabilize the plurality of frames; and

providing the stabilized frames as an output;

wherein the measure comprises a ratio of an edge length for a frame from the plurality of frames before and after transformation to the first reference frame.

12. The system of claim 11, wherein the ratio comprises a ratio of an edge length of a portion of the planar surface.

13. The system of claim 11, wherein determining the transformation comprises identifying feature points in the two consecutive frames.

14. The system of claim 13, wherein the feature points are determined using Scale Invariant Feature Transform.

15. The system of claim 11, wherein the planar surface is a polygon.

16. The system of claim 15, wherein the polygon consists of four sides.

17. The system of claim 11, further comprising an infrared camera to capture the plurality of frames.

18. The system of claim 17, wherein the infrared camera is mounted on an airplane.

19. The system of claim 18, wherein the planar surface is a runway on which the airplane is to land.

20. The system of claim 11, the operations further comprising:

performing motion detection using background modeling on the stabilized frames, wherein the background modeling identifies an image in the first reference frame;

identifying an image in a stabilized frame that corresponds to the image in the first reference frame by comparing images in the stabilized frame with images in the first reference frame; and

upon identifying that a position of the corresponding image in the stabilized frame is different from the image in the first reference frame, determining that the corresponding image represents a moving object.

21. A computer-implemented method comprising:

receiving a plurality of frames, each frame including an image of at least a portion of a planar surface;

stabilizing two consecutive frames of the plurality of frames, the stabilizing comprising determining a transformation mapping a succeeding frame of the two consecutive frames to a preceding frame of the two consecutive frames and based on the transformation, warping the two consecutive frames to a reference frame of the plurality of frames; and

detecting a movement of an object in the two consecutive frames, the movement based on a change in positions of the object in the preceding frame and the successive frame;

wherein the detecting further comprises determining a transformation to compensate for a variation in intensity gain between the two consecutive frames; and

wherein the reference frame is updated based on a ratio of an edge length for frames before and after their respective warping.

22. The method of claim 21, further comprising capturing the plurality of frames using an infrared camera.

23. The method of claim 21, further comprising identifying the planar surface in at least one frame of the plurality of frames.

24. The method of claim 21, wherein the stabilizing further comprises determining a transformation mapping a frame in the plurality of frames to the reference frame.

25. The method of claim 21, wherein the detecting further comprises filtering the plurality of frames to retain only the at least a portion of the planar surface.

26. The method of claim 21, wherein the movement of the object in the two consecutive frames is detected using background modeling.

27. A computer-implemented method to detect moving objects on a runway from data captured by an infrared camera included in an aircraft approaching the runway, the method comprising:

receiving a sequence of images, each of the images including a representation of the runway captured by the infrared camera from a different location as the aircraft approaches the runway;

estimating, for each of at least two separate portions of the sequence of images, perspective transformations between a selected region in a reference image and in additional images of the portion of the sequence, the selected region in each image including the representation of the runway but not including all image data of the image;

updating the reference image from a first reference image to a second reference image responsive to an indication of a titling angle passing a threshold, wherein the first reference image is used for a first of the at least two separate portions of the sequence of images, the second reference image is used for a second of the at least two separate portions of the sequence of images, and the titling angle is associated with a current projective transformation estimated with respect to the first reference image;

creating a background model from the sequence of images; and

identifying moving objects by comparing the sequence of images with the background model.

28. The computer-implemented method of claim 27, comprising identifying the selected region in each of the images of the sequence.

29. The computer-implemented method of claim 28, wherein identifying the selected region comprises using data from one or more devices on the aircraft, including a receiver for a satellite-based positioning system, to calculated locations of vertices for the selected region.

30. The computer-implemented method of claim 27, wherein the estimating comprises:

identifying feature points in the selected region for each of the images using Scale Invariant Feature Transform; and

calculating a projective transformation between two images using identified feature points matched between the two images.

31. The computer-implemented method of claim 27, comprising determining the indication of the titling angle by calculating a ratio of a lower edge length of an image before and after its projective transformation.

32. The computer-implemented method of claim 27, comprising normalizing each of the at least two separate portions of the sequence of images by correcting global intensity for each of the additional images with respect to the reference image in each respective portion of the sequence.

33. The computer-implemented method of claim 32, wherein creating the background model comprises updating the background model for the transition between the first portion and the second portion of the sequence of images, the background model updating comprising adjusting mean intensity of the background model against the second reference image.