Motion-seeded object based attention for dynamic visual imagery

• US 9,196,053 B1
• Filed: 11/05/2012
• Issued: 11/24/2015
• Est. Priority Date: 10/04/2007
• Status: Active Grant

• Current Assignee: HRL Laboratories LLC (The Boeing Co.)
• Original Assignee: HRL Laboratories LLC (The Boeing Co.)
• Inventors: Huber, David J., Khosla, Deepak, Kim, Kyungnam
• Primary Examiner(s): DRENNAN, BARRY T
• Application Number:
  US13/669,269
• Time in Patent Office: 1,114 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06T 7/187 involving region growing; involving region merging; involving connected component labelling
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G06T 7/215 Motion-based segmentation
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06V 10/255 Detecting or recognising potential candidate objects based on visual cues, e.g. shapes
• CPC Class Codes: G06V 10/451 with interaction between the filter responses, e.g. cortical complex cells
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69

Abstract

Described is a system for object detection from dynamic visual imagery. Video imagery is received as input, and the system processes each frame to detect a motion region exhibiting unexpected motion representing a moving object. Object-based feature extraction is applied to each frame containing a detection motion region. Each frame is then divided into feature-consistent regions. A motion seeded region (MSR) is identified in each frame containing a detected motion region, where the MSR corresponds to a spatial location of the detected motion region. Similar feature-consistent regions that are adjacent to and include the MSR are joined to compute a boundary around the plurality of similar feature-consistent regions. Finally, the system outputs a computed boundary surrounding an object in the detected motion region.

Claims

1. A system for object detection from dynamic visual imagery, the system comprising:

one or more processors and a memory having instructions such that when the instructions are executed, the one or more processors perform operations of;

receiving, as input, video imagery comprising a plurality of frames;

processing each frame to detect at least one motion region exhibiting unexpected motion, wherein the at least one motion region represents a moving object in the video imagery;

applying object-based feature extraction to each frame containing a detected motion region;

dividing each frame containing a detected motion region into a plurality of feature-consistent regions;

identifying, in each frame containing a detected motion region, at least one motion seeded region (MSR) in the plurality of feature-consistent regions, wherein the MSR corresponds to a spatial location of the detected motion region in the frame;

joining a plurality of similar feature-consistent regions that are adjacent to and include the MSR to compute a boundary around the plurality of similar feature-consistent regions; and

outputting, for each frame containing a detected motion region, a computed boundary surrounding an object in the detected motion region.

2. The system as set forth in claim 1, wherein the one or more processors further perform an operation of identifying a motion seeded feature (MSF) in each feature-consistent region, wherein the MSF is the feature that is consistent in the plurality of feature-consistent regions.

3. The system as set forth in claim 2, wherein the one or more processors further perform operations of:

defining a list X as a list of all detected motion regions;

defining the MSR as a first region in the list X for a first iteration;

defining a list N comprising all feature-consistent regions adjacent to the MSR;

computing a relative difference value across the MSF between the MSR in the list X and a candidate feature-consistent region in the list N;

wherein if the relative difference value is less than a predetermined threshold, then the candidate feature-consistent region is accepted as part of a detected motion region and added to the list X, and if the relative difference value is more than a predetermined threshold, then the candidate feature-consistent region is rejected.

4. The system as set forth in claim 3, wherein the one or more processors further perform an operation of computing, for a plurality of iterations, a relative difference value between each detection motion region in the list X and each feature-consistent region in the list N.

5. The system as set forth in claim 1, wherein the video imagery comprises red, green, and blue color channels, and wherein the one or more processors further perform operations of:

calculating intensity channels, comprising light intensity channels and dark intensity channels, from the video imagery by averaging the red, green, and blue color channels; and

creating four fully-saturated color channels, comprising red, blue, green, and yellow color channels, that yield zero-response to white;

wherein effects of the color channels and the intensity channels are separated.

6. The system as set forth in claim 5, wherein the one or more processors further perform operations of:

computing a set of feature maps from the four fully-saturated color channels and the intensity channels;

computing gradient information for each feature map in the set of feature maps to generate a set of edge maps;

computing a master edge map comprising feature contour areas by combining the set of edge maps;

processing the master edge map to complete a boundary around the detected motion region and close the feature contour areas in the master edge map; and

inputting the master edge map into a watershed process, wherein the watershed process converts the feature contour areas into the feature-consistent regions.

7. A computer-implemented method for object detection from dynamic visual imagery, comprising an act of causing a data processor to execute instructions stored on a non-transitory memory such that upon execution, the data processor performs operations of:

receiving, as input, video imagery comprising a plurality of frames;

processing each frame to detect at least one motion region exhibiting unexpected motion, wherein the at least one motion region represents a moving object in the video imagery;

applying object-based feature extraction to each frame containing a detected motion region;

dividing each frame containing a detected motion region into a plurality of feature-consistent regions;

identifying, in each frame containing a detected motion region, at least one motion seeded region (MSR) in the plurality of feature-consistent regions, wherein the MSR corresponds to a spatial location of the detected motion region in the frame;

joining a plurality of similar feature-consistent regions that are adjacent to and include the MSR to compute a boundary around the plurality of similar feature-consistent regions; and

outputting, for each frame containing a detected motion region, a computed boundary surrounding an object in the detected motion region.

8. The method as set forth in claim 7, further comprising an act of causing a data processor to perform an operation of identifying a motion seeded feature (MSF) in each feature-consistent region, wherein the MSF is the feature that is consistent in the plurality of feature-consistent regions.

9. The method as set forth in claim 8, further comprising an act of causing a data processor to perform operations of:

defining a list X as a list of all detected motion regions;

defining the MSR as a first region in the list X for a first iteration;

defining a list N comprising all feature-consistent regions adjacent to the MSR;

computing a relative difference value across the MSF between the MSR in the list X and a candidate feature-consistent region in the list N;

wherein if the relative difference value is less than a predetermined threshold, then the candidate feature-consistent region is accepted as part of a detected motion region and added to the list X, and if the relative difference value is more than a predetermined threshold, then the candidate feature-consistent region is rejected.

10. The method as set forth in claim 9, further comprising an act of causing a data processor to perform an operation of computing, for a plurality of iterations, a relative difference value between each detection motion region in the list X and each feature-consistent region in the list N.

11. The method as set forth in claim 7, further comprising an act of causing a data processor to perform operations of:

calculating intensity channels, comprising light intensity channels and dark intensity channels, from the video imagery by averaging the red, green, and blue color channels; and

creating four fully-saturated color channels, comprising red, blue, green, and yellow color channels, that yield zero-response to white;

wherein effects of the color channels and the intensity channels are separated.

12. The method as set forth in claim 11, further comprising an act of causing a data processor to perform operations of:

computing a set of feature maps from the four fully-saturated color channels and the intensity channels;

computing gradient information for each feature map in the set of feature maps to generate a set of edge maps;

computing a master edge map comprising feature contour areas by combining the set of edge maps;

processing the master edge map to complete a boundary around the detected motion region and close the feature contour areas in the master edge map; and

inputting the master edge map into a watershed process, wherein the watershed process converts the feature contour areas into the feature-consistent regions.

13. A computer program product for object detection from dynamic visual imagery, the computer program product comprising:

computer-readable instructions stored on a non-transitory computer-readable medium that are executable by a computer having a processor for causing the processor to perform operations of;

receiving, as input, video imagery comprising a plurality of frames;

processing each frame to detect at least one motion region exhibiting unexpected motion, wherein the at least one motion region represents a moving object in the video imagery;

applying object-based feature extraction to each frame containing a detected motion region;

dividing each frame containing a detected motion region into a plurality of feature-consistent regions;

identifying, in each frame containing a detected motion region, at least one motion seeded region (MSR) in the plurality of feature-consistent regions, wherein the MSR corresponds to a spatial location of the detected motion region in the frame;

joining a plurality of similar feature-consistent regions that are adjacent to and include the MSR to compute a boundary around the plurality of similar feature-consistent regions; and

outputting, for each frame containing a detected motion region, a computed boundary surrounding an object in the detected motion region.

14. The computer program product as set forth in claim 13, further comprising instructions for causing a processor to perform an operation of identifying a motion seeded feature (MSF) in each feature-consistent region, wherein the MSF is the feature that is consistent in the plurality of feature-consistent regions.

15. The computer program product as set forth in claim 14, further comprising instructions for causing a processor to perform operations of:

defining a list X as a list of all detected motion regions;

defining the MSR as a first region in the list X for a first iteration;

defining a list N comprising all feature-consistent regions adjacent to the MSR;

computing a relative difference value across the MSF between the MSR in the list X and a candidate feature-consistent region in the list N;

wherein if the relative difference value is less than a predetermined threshold, then the candidate feature-consistent region is accepted as part of a detected motion region and added to the list X, and if the relative difference value is more than a predetermined threshold, then the candidate feature-consistent region is rejected.

16. The computer program product as set forth in claim 15, further comprising instructions for causing a processor to perform an operation of computing, for a plurality of iterations, a relative difference value between each detection motion region in the list X and each feature-consistent region in the list N.

17. The computer program product as set forth in claim 13, further comprising instructions for causing a processor to perform operations of:

calculating intensity channels, comprising light intensity channels and dark intensity channels, from the video imagery by averaging the red, green, and blue color channels; and

creating four fully-saturated color channels, comprising red, blue, green, and yellow color channels, that yield zero-response to white;

wherein effects of the color channels and the intensity channels are separated.

18. The computer program product as set forth in claim 17, further comprising instructions for causing a processor to perform operations of:

computing a set of feature maps from the four fully-saturated color channels and the intensity channels;

computing gradient information for each feature map in the set of feature maps to generate a set of edge maps;

computing a master edge map comprising feature contour areas by combining the set of edge maps;

processing the master edge map to complete a boundary around the detected motion region and close the feature contour areas in the master edge map; and

inputting the master edge map into a watershed process, wherein the watershed process converts the feature contour areas into the feature-consistent regions.