Method and system for fusion of fast surprise and motion-based saliency for finding objects of interest in dynamic scenes

• US 9,177,228 B1
• Filed: 01/17/2013
• Issued: 11/03/2015
• Est. Priority Date: 10/04/2007
• Status: Active Grant

• Current Assignee: HRL Laboratories LLC (The Boeing Co.)
• Original Assignee: HRL Laboratories LLC (The Boeing Co.)
• Inventors: Khosla, Deepak, Huber, David J., Kim, Kyungnam
• Primary Examiner(s): DRENNAN, BARRY T
• Application Number:
  US13/743,742
• Time in Patent Office: 1,020 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 18/253 of extracted features
• CPC Class Codes: G06V 10/451 with interaction between the filter responses, e.g. cortical complex cells
• CPC Class Codes: G06V 10/806 of extracted features
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69

Abstract

Described is a system for object detection from dynamic visual imagery. Dynamic visual input obtained from a stationary sensor is processed by a surprise-based module. The surprise-based module detects a stationary object in a scene to generate surprise scores. The dynamic visual input is also processed by a motion-based saliency module which detects foreground in the scene to generate motion scores. The surprise scores and motion scores are fused into a single score, and the single score is used to determine the presence of an object of interest.

Claims

1. A system for object detection from dynamic visual imagery, the system comprising:

one or more processors and a non-transitory memory having instructions encoded thereon such that when the instructions are executed, the one or more processors perform operations of;

receiving dynamic visual input obtained from a stationary sensor;

processing the dynamic visual input by a surprise-based module, wherein the surprise-based module detects at least one stationary object in a scene to generate a plurality of surprise scores;

processing the dynamic visual input by a motion-based saliency module, wherein the motion-based saliency module detects foreground in the scene to generate a plurality of motion scores; and

fusing at least one of the plurality of surprise scores with at least one of the plurality of motion scores into a single score, wherein the single score is used to determine the presence of at least one object of interest in the dynamic visual input.

2. The system as set forth in claim 1, wherein the one or more processors further perform an operation of processing the plurality of motion scores through a normalization function that constrains a score domain for each motion score to between zero and one.

3. The system as set forth in claim 2, wherein the one or more processors further perform an operation of fusing at least one score from a set of N scores selected from the plurality of surprise scores with at least one score from a set of N scores selected from the plurality of motion scores, wherein each set of N scores represents scores from a set of predetermined locations of interest in the scene.

4. The system as set forth in claim 2, wherein the one or more processors further perform an operation of determining a maximum number of scores to analyze for a given period of time, wherein a fixed number of scores are fused from each of the plurality of surprise scores and the plurality of motion scores.

5. The system as set forth in claim 2, wherein the one of more processors further perform an operation of using all of the scores from one of the plurality of surprise scores or the plurality of motion scores for fusion.

6. The system as set forth in claim 2, wherein the one or more processors further perform an operation of fusing at least one surprise score of the plurality of surprise scores with at least one motion score of the plurality of motion scores, wherein the at least one surprise score and the at least one motion score comprise at least one target in the same spatial location in the scene.

7. A computer-implemented method for object detection from dynamic visual imagery, comprising an act of causing a data processor to execute instructions stored on a non-transitory computer-readable medium such that upon execution, the data processor performs operations of:

receiving dynamic visual input obtained from a stationary sensor;

processing the dynamic visual input by a surprise-based module, wherein the surprise-based module detects at least one stationary object in a scene to generate a plurality of surprise scores;

processing the dynamic visual input by a motion-based saliency module, wherein the motion-based saliency module detects foreground in the scene to generate a plurality of motion scores; and

fusing at least one of the plurality of surprise scores with at least one of the plurality of motion scores into a single score, wherein the single score is used to determine the presence of at least one object of interest in the dynamic visual input.

8. The method as set forth in claim 7, further comprising an act of causing a data processor to perform an operation of processing the plurality of motion scores through a normalization function that constrains a score domain for each motion score to between zero and one.

9. The method as set forth in claim 8, further comprising an act of causing a data processor to perform an operation of fusing at least one score from a set of N scores selected from the plurality of surprise scores with at least one score from a set of N scores selected from the plurality of motion scores, wherein each set of N scores represents scores from a set of predetermined locations of interest in the scene.

10. The method as set forth in claim 8, further comprising an act of causing a data processor to perform an operation of determining a maximum number of scores to analyze for a given period of time, wherein a fixed number of scores are fused from each of the plurality of surprise scores and the plurality of motion scores.

11. The method as set forth in claim 8, further comprising an act of causing a data processor to perform an operation of using all of the scores from one of the plurality of surprise scores or the plurality of motion scores for fusion.

12. The method as set forth in claim 8, further comprising an act of causing a data processor to perform an operations of fusing at least one surprise score of the plurality of surprise scores with at least one motion score of the plurality of motion scores, wherein the at least one surprise score and the at least one motion score comprise at least one target in the same spatial location in the scene.

13. A computer program product for object detection from dynamic visual imagery, the computer program product comprising:

computer-readable instructions stored on a non-transitory computer-readable medium that are executable by a computer having a processor for causing the processor to perform operations of;

receiving dynamic visual input obtained from a stationary sensor;

processing the dynamic visual input by a surprise-based module, wherein the surprise-based module detects at least one stationary object in a scene to generate a plurality of surprise scores;

processing the dynamic visual input by a motion-based saliency module, wherein the motion-based saliency module detects foreground in the scene to generate a plurality of motion scores; and

fusing at least one of the plurality of surprise scores with at least one of the plurality of motion scores into a single score, wherein the single score is used to determine the presence of at least one object of interest in the dynamic visual input.

14. The computer program product as set forth in claim 13, further comprising instructions for causing a processor to perform an operation of processing the plurality of motion scores through a normalization function that constrains a score domain for each motion score to between zero and one.

15. The computer program product as set forth in claim 14, further comprising instructions for causing a processor to perform an operation of fusing at least one score from a set of N scores selected from the plurality of surprise scores with at least one score from a set of N scores selected from the plurality of motion scores, wherein each set of N scores represents scores from a set of predetermined locations of interest in the scene.

16. The computer program product as set forth in claim 14, further comprising instructions for causing a processor to perform an operation of determining a maximum number of scores to analyze for a given period of time, wherein a fixed number of scores are fused from each of the plurality of surprise scores and the plurality of motion scores.

17. The computer program product as set forth in claim 14, further comprising instructions for causing a processor to perform an operation of using all of the scores from one of the plurality of surprise scores or the plurality of motion scores for fusion.

18. The computer program product as set forth in claim 14, further comprising instructions for causing a processor to perform an operation of fusing at least one surprise score of the plurality of surprise scores with at least one motion score of the plurality of motion scores, wherein the at least one surprise score and the at least one motion score comprise at least one target in the same spatial location in the scene.