Bio-inspired method of ground object cueing in airborne motion imagery

• US 9,008,366 B1
• Filed: 07/09/2013
• Issued: 04/14/2015
• Est. Priority Date: 01/23/2012
• Status: Active Grant

• Current Assignee: HRL Laboratories LLC (The Boeing Co.)
• Original Assignee: HRL Laboratories LLC (The Boeing Co.)
• Inventors: Kim, Kyungnam, Jeong, Changsoo S., Khosla, Deepak, Chen, Yang, Cheng, Shinko Y., Zhang, Lei, Honda, Alexander L.
• Primary Examiner(s): Fitzpatrick, Atiba O
• Application Number:
  US13/938,196
• Time in Patent Office: 644 Days
• Field of Search: None
• US Class Current: 382/103
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10032 Satellite or aerial image; Remote sensing
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 2207/30232 Surveillance
• CPC Class Codes: G06T 2207/30241 Trajectory
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments

Abstract

Described is method for object cueing in motion imagery. Key points and features are extracted from motion imagery, and features between consecutive image frames of the motion imagery are compared to identify similar image frames. A candidate set of matching keypoints is generated by matching keypoints between the similar image frames. A ground plane homography model that fits the candidate set of matching keypoints is determined to generate a set of correct matching keypoints. Each image frame of a set of image frames within a selected time window is registered into a reference frame'"'"'s coordinate system using the homography transformation. A difference image is obtained between the reference frame and each registered image frame, resulting in multiple difference images. The difference images are then accumulated to calculate a detection image which is used for detection of salient regions. Object cues for surveillance use are produced based on the detected salient regions.

Claims

1. A system for object cueing in motion imagery, the system comprising:

one or more processors and a non-transitory memory having instructions encoded thereon such that when the instructions are executed, the one or more processors perform operations of;

extracting key points and bio-inspired features from input motion imagery, wherein the input motion imagery comprises a sequence of image frames obtained with a sensor;

comparing bio-inspired features between consecutive image frames in the sequence of image frames to identify a set of image frames having similar bio-inspired features;

matching keypoints between the set of image frames to generate a candidate set of matching keypoints;

determining a ground plane homography model that fits the candidate set of matching keypoints to generate a set of correct matching keypoints and an accurate homography transformation between the set of image frames;

registering each image within a selected time window into a reference frame'"'"'s coordinate system using the homography transformation;

obtaining a difference image between the reference frame and each registered image frame within the time window, resulting in a plurality of difference images;

accumulating the plurality of difference images to calculate a detection image;

detecting salient regions from the detection image by blob extraction; and

producing object cues for surveillance use based on the detected salient regions.

2. The system as set forth in claim 1, wherein the one or more processors further perform an operation of thresholding the detection image by a predefined threshold value.

3. The system as set forth in claim 2, wherein the one or more processors further perform an operation of automatically selecting parameters for object cueing based on information obtained from the sensor, wherein the parameters comprise at least one of:

time window size, minimum pixel area of a target of interest, maximum pixel area of a target of interest, and the predefined threshold value.

4. The system as set forth in claim 3, wherein the one or more processors further perform an operation of determining a size of a time window using information obtained from the sensor according to the following:

5. The system as set forth in claim 4, wherein the one or more processors further perform an operation of calculating the detection image according to the following:

D=Th{Σ

_i=0^2kabs(F_i−

F_k)} th=k*m*σ

,

where D denotes the detection image, Σ

denotes a summation, abs denotes absolute value, F_i is the ith image frame within a time window of size 2k+1, F_k is a current image frame, Th represents a thresholding function, th denotes the predefined threshold value, σ

is an average background noise level, and m is a scaler for σ

.

6. The system as set forth in claim 1, wherein the one or more processors further perform an operation of using random sample consensus to determine the ground plane homography model.

7. A computer-implemented method for object cueing in motion imagery, comprising:

an act of causing a data processor to execute instructions stored on a non-transitory memory such that upon execution, the data processor performs operations of;

extracting key points and bio-inspired features from input motion imagery, wherein the input motion imagery comprises a sequence of image frames obtained with a sensor;

comparing bio-inspired features between consecutive image frames in the sequence of image frames to identify a set of image frames having similar bio-inspired features;

matching keypoints between the set of image frames to generate a candidate set of matching keypoints;

determining a ground plane homography model that fits the candidate set of matching keypoints to generate a set of correct matching keypoints and an accurate homography transformation between the set of image frames;

registering each image frame within a selected time window into a reference frame'"'"'s coordinate system using the homography transformation;

obtaining a difference image between the reference frame and each registered image frame within the time window, resulting in a plurality of difference images;

accumulating the plurality of difference images to calculate a detection image;

detecting salient regions from the detection image by blob extraction; and

producing object cues for surveillance use based on the detected salient regions.

8. The method as set forth in claim 7, wherein the data processor further performs an operation of thresholding the detection image by a predefined threshold value.

9. The method as set forth in claim 8, wherein the data processor further performs an operation of automatically selecting parameters for object cueing based on information obtained from the sensor, wherein the parameters comprise at least one of:

time window size, minimum pixel area of a target of interest, maximum pixel area of a target of interest, and the predefined threshold value.

10. The method as set forth in claim 9, wherein the data processor further performs an operation of determining a size of a time window using information obtained from the sensor according to the following:

11. The method as set forth in claim 10, wherein the data processor further performs an operation of calculating the detection image according to the following:

D=Th{Σ

_i=0^2kabs(F_i−

F_k)} th=k*m*σ

,

where D denotes the detection image, Σ

denotes a summation, abs denotes absolute value, F_i is the ith image frame within a time window of size 2k+1, F_k is a current image frame, Th represents a thresholding function, th denotes the predefined threshold value, σ

is an average background noise level, and m is a scaler for σ

.

12. The method as set forth in claim 7, wherein the data processor further performs an operation of using random sample consensus to determine the ground plane homography model.

13. A computer program product for object cueing in motion imagery, the computer program product comprising computer-readable instructions stored on a non-transitory computer-readable medium that are executable by a computer having a processor for causing the processor to perform operations of:

extracting key points and bio-inspired features from input motion imagery, wherein the input motion imagery comprises a sequence of image frames obtained with a sensor;

comparing bio-inspired features between consecutive image frames in the sequence of image frames to identify a set of image frames having similar bio-inspired features;

matching keypoints between the set of image frames to generate a candidate set of matching keypoints;

determining a ground plane homography model that fits the candidate set of matching keypoints to generate a set of correct matching keypoints and an accurate homography transformation between the set of image frames;

registering each image frame within a selected time window into a reference frame'"'"'s coordinate system using the homography transformation;

obtaining a difference image between the reference frame and each registered image frame within the time window, resulting in a plurality of difference images;

accumulating the plurality of difference images to calculate a detection image;

detecting salient regions from the detection image by blob extraction; and

producing object cues for surveillance use based on the detected salient regions.

14. The computer program product as set forth in claim 13, further comprising instructions for causing the processor to perform an operation of thresholding the detection image by a predefined threshold value.

15. The computer program product as set forth in claim 14, further comprising instructions for causing the processor to perform an operation of automatically selecting parameters for object cueing based on information obtained from the sensor, wherein the parameters comprise at least one of:

time window size, minimum pixel area of a target of interest, maximum pixel area of a target of interest, and the predefined threshold value.

16. The computer program product as set forth in claim 15, further comprising instructions for causing the processor to perform an operation of determining a size of a time window using information obtained from the sensor according to the following:

17. The computer program product as set forth in claim 16, further comprising instructions for causing the processor to perform an operation of calculating the detection image according to the following:

D=Th{Σ

_i=0^2kabs(F_i−

F_k)} th=k*m*σ

,

where D denotes the detection image, Σ

denotes a summation, abs denotes absolute value, F_i is the ith image frame within a time window of size 2k+1, F_k is a current image frame, Th represents a thresholding function, th denotes the predefined threshold value, σ

is an average background noise level, and m is a scaler for σ

.

18. The computer program product as set forth in claim 13, further comprising instructions for causing the processor to perform an operation of using random sample consensus to determine the ground plane homography model.