System for identifying regions of interest in visual imagery

• US 8,774,517 B1
• Filed: 12/30/2010
• Issued: 07/08/2014
• Est. Priority Date: 06/14/2007
• Status: Active Grant

• Current Assignee: HRL Laboratories LLC (The Boeing Co.)
• Original Assignee: HRL Laboratories LLC (The Boeing Co.)
• Inventors: Khosla, Deepak, Bhattacharyya, Rajan, Huber, David J., Mundhenk, Terrell N.
• Primary Examiner(s): DRENNAN, BARRY T
• Application Number:
  US12/982,713
• Time in Patent Office: 1,286 Days
• Field of Search: None
• US Class Current: 382/190
• CPC Class Codes: G06F 18/253 of extracted features
• CPC Class Codes: G06V 10/451 with interaction between the filter responses, e.g. cortical complex cells
• CPC Class Codes: G06V 10/806 of extracted features
• CPC Class Codes: G06V 10/96 Management of image or video recognition tasks
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69

Abstract

The present invention relates to a system for identifying regions of interest in visual imagery. The system is configured to receive a series of consecutive frames representing a scene as captured from N sensors. The frames include at least a current frame and a previous frame. A surprise map can be generated based on features found in the current frame and the previous frame. The surprise map having a plurality of values corresponding to spatial locations within the scene. Based on the values, a surprise in the scene can be identified if a value in the surprise map exceeds a predetermined threshold.

Claims

1. A system for identifying regions of interest in visual imagery, the system comprising:

a processor;

a memory, the memory including executable instructions encoded thereon that, when executed by the processor, causes the processor to perform operations of;

receiving a series of consecutive frames representing a scene as captured from N sensors, the frames having at least a current frame and a previous frame;

generating a surprise map based on features found in the current frame and the previous frame, the surprise map having a plurality of values corresponding to spatial locations within the scene; and

determining a surprise in the scene based on a value in the surprise map exceeding a predetermined threshold

wherein the processor uses a number B, less than N, of the N sensors to capture the series of consecutive frames, such that the B sensors are active during a burst period and are deactivated during a wait period during which at least one of the remaining sensors is active, and

wherein the processor determines, for the B sensors, whether to store over the wait period at least one of a prior map and the previous frame.

2. The system as set forth in claim 1, wherein the operation of generating the surprise map further comprises operations of:

splitting the current frame into four fully saturated color channels, an R-channel (R), a G-channel (G), a B-channel (B), and a Y-channel (Y);

generating a light intensity channel (L) and a dark intensity channel (D) from the current frame;

generating a series of color feature maps from the four color channels for the current frame using center-surround differencing between color channels, each color feature map representing a color feature;

generating a series of intensity feature maps from the two intensity channels for the current frame using center-surround differencing, each intensity feature map representing a color/intensity feature;

comparing the light intensity channel (L) from the current frame with an light intensity channel (L) from the previous frame to generate five motion feature maps, each motion feature map representing a motion feature;

generating a prior map (P) for each of the color and motion features by integrating the motion feature maps and the color feature maps over time;

updating the prior map as new current frames are received from the sensors;

generating a surprise feature map, for each feature, by determining a rectified difference between the feature map from the newest frame and its corresponding prior map;

generating a surprise color conspicuity map by combining and normalizing the RG and BY surprise feature maps;

generating an intensity surprise conspicuity map by normalizing the LD surprise feature map;

generating a motion surprise conspicuity map by combining and normalizing the five surprise feature maps for motion;

generating the surprise map by combining and normalizing the surprise conspicuity maps, the surprise map having a plurality of values corresponding to spatial locations within the scene.

3. The system as set forth in claim 2, wherein the processor is further configured to perform an operation of stabilizing the prior map through receiving new current frames and updating the prior map until a feature variance is less than a predetermined threshold.

4. The system as set forth in claim 3, wherein in receiving a series of consecutive frames representing a scene as captured from N sensors, N is a plurality of fixed sensors and each sensor has a memory containing its own prior map (P) and the previous frame.

5. The system as set forth in claim 4, wherein the processor is configured to perform an operation of memory management through selectively using a subset of the plurality of fixed the sensors to capture the series of consecutive frames.

6. The system as set forth in claim 5, wherein in selectively using the subset, the subset of sensors is selected such that the subset is active and used during a burst period, and deactivated during a wait period.

7. The system as set forth in claim 6, wherein in selectively using the subset, the processor is configured to perform a memory management function selected from a group consisting of store all, store none, store motion, and store priors, as follows:

wherein in store all, the prior map (P) and the previous frame are maintained over the wait period;

wherein in store none, both the prior map and previous frame are deleted during the wait period;

wherein in store motion, the prior map is deleted during the wait period and the previous frame is maintained over the wait period; and

wherein in store priors, the previous frame is deleted during the wait period and the prior map is maintained over the wait period.

8. The system as set forth in claim 7, wherein the processor is further configured to allow a user to selectively remove a sensor from the plurality of fixed sensors, leaving remaining sensors, which causes the system to adjust the memory management function by decreasing the wait period assigned to the remaining sensors.

9. The system as set forth in claim 8, wherein in receiving a series of consecutive frames, if the frames are in black and white, the frames are converted into an red/green/blue (rgb) format of three channels, an red-channel (r), a g-channel (g), and a b-channel (b), where all three channels have the same value.

10. The system as set forth in claim 9, wherein in splitting the current frame into four fully saturated channels, the channels are split as follows:

11. The system as set forth in claim 10, wherein when generating a light intensity channel (L) and a dark intensity channel (D) from each frame, they are generated by averaging the r, g, and b-channels, as follows:

L=(r+g+b)/3

D=(maximum value of a color channel)−

L.

12. The system as set forth in claim 11, wherein when generating five motion feature maps, the motion feature maps are generated based on motion across four cardinal directions and a glimmer.

13. The system as set forth in claim 12, wherein the system is configured to weight the surprise feature maps to modulate the system'"'"'s surprise response to specific features.

14. A computer program product for identifying regions of interest in visual imagery, the computer program product comprising computer-readable instructions stored on a non-transitory computer-readable medium that are executable by a computer having a processor for causing the processor to perform operations of:

receiving a series of consecutive frames representing a scene as captured from N sensors, the frames having at least a current frame and a previous frame;

generating a surprise map based on features found in the current frame and the previous frame, the surprise map having a plurality of values corresponding to spatial locations within the scene; and

determining a surprise in the scene based on a value in the surprise map exceeding a predetermined threshold

wherein the processor uses a number B, less than N, of the N sensors to capture the series of consecutive frames, such that the B sensors are active during a burst period and are deactivated during a wait period during which at least one of the remaining sensors is active, and

wherein the processor determines, for the B sensors, whether to store over the wait period at least one of a prior map and the previous frame.

15. The computer program product as set forth in claim 14, wherein the operation of generating the surprise map further comprises operations of:

splitting the current frame into four fully saturated color channels, an R-channel (R), a G-channel (G), a B-channel (B), and a Y-channel (Y);

generating a light intensity channel (L) and a dark intensity channel (D) from the current frame;

generating a series of color feature maps from the four color channels for the current frame using center-surround differencing between color channels, each color feature map representing a color feature;

generating a series of intensity feature maps from the two intensity channels for the current frame using center-surround differencing, each intensity feature map representing a color/intensity feature;

comparing the light intensity channel (L) from the current frame with an light intensity channel (L) from the previous frame to generate five motion feature maps, each motion feature map representing a motion feature;

generating a prior map (P) for each of the color and motion features by integrating the motion feature maps and the color feature maps over time;

updating the prior map as new current frames are received from the sensors;

generating a surprise feature map, for each feature, by determining a rectified difference between the feature map from the newest frame and its corresponding prior map;

generating a surprise color conspicuity map by combining and normalizing the RG and BY surprise feature maps;

generating an intensity surprise conspicuity map by normalizing the LD surprise feature map;

generating a motion surprise conspicuity map by combining and normalizing the five surprise feature maps for motion;

generating the surprise map by combining and normalizing the surprise conspicuity maps, the surprise map having a plurality of values corresponding to spatial locations within the scene.

16. The computer program product as set forth in claim 15, further comprising instructions to cause a processor to perform an operation of stabilizing the prior map through receiving new current frames and updating the prior map until a feature variance is less than a predetermined threshold.

17. The computer program product as set forth in claim 16, wherein in receiving a series of consecutive frames representing a scene as captured from N sensors, N is a plurality of fixed sensors and each sensor has a memory containing its own prior map (P) and the previous frame.

18. The computer program product as set forth in claim 17, further comprising instructions to cause a processor to perform an operation of memory management through selectively using a subset of the plurality of fixed the sensors to capture the series of consecutive frames.

19. The computer program product as set forth in claim 18, wherein in selectively using the subset, the subset of sensors is selected such that the subset is active and used during a burst period, and deactivated during a wait period.

20. The computer program product as set forth in claim 19, further comprising instructions to cause a processor to perform a memory management function selected from a group consisting of store all, store none, store motion, and store priors, as follows:

wherein in store all, the prior map (P) and the previous frame are maintained over the wait period;

wherein in store none, both the prior map and previous frame are deleted during the wait period;

wherein in store motion, the prior map is deleted during the wait period and the previous frame is maintained over the wait period; and

wherein in store priors, the previous frame is deleted during the wait period and the prior map is maintained over the wait period.

21. The computer program product as set forth in claim 20, further comprising instructions to cause a processor to allow a user to selectively remove a sensor from the plurality of fixed sensors, leaving remaining sensors, which causes the processor to adjust the memory management function by decreasing the wait period assigned to the remaining sensors.

22. The computer program product as set forth in claim 21, wherein in receiving a series of consecutive frames, if the frames are in black and white, the frames are converted into an red/green/blue (rgb) format of three channels, an red-channel (r), a g-channel (g), and a b-channel (b), where all three channels have the same value.

23. The computer program product as set forth in claim 22, wherein in splitting the current frame into four fully saturated channels, the channels are split as follows:

24. The computer program product as set forth in claim 23, wherein when generating a light intensity channel (L) and a dark intensity channel (D) from each frame, they are generated by averaging the r, g, and b-channels, as follows:

L=(r+g+b)/3

D=(maximum value of a color channel)−

L.

25. The computer program product as set forth in claim 24, wherein when generating five motion feature maps, the motion feature maps are generated based on motion across four cardinal directions and a glimmer.

26. The computer program product as set forth in claim 25, further comprising instructions to cause a processor to weight the surprise feature maps to modulate the processor'"'"'s surprise response to specific features.

27. A method for identifying regions of interest in visual imagery, the method comprising an act of:

causing a processor to execute instructions encoded on a memory, such that upon execution, the processor performs operations of;

receiving a series of consecutive frames representing a scene as captured from N sensors, the frames having at least a current frame and a previous frame;

generating a surprise map based on features found in the current frame and the previous frame, the surprise map having a plurality of values corresponding to spatial locations within the scene; and

determining a surprise in the scene based on a value in the surprise map exceeding a predetermined threshold

wherein the processor uses a number B, less than N, of the N sensors to capture the series of consecutive frames, such that the B sensors are active during a burst period and are deactivated during a wait period during which at least one of the remaining sensors is active, and

wherein the processor determines, for the B sensors, whether to store over the wait period at least one of a prior map and the previous frame.

28. The method as set forth in claim 27, wherein the operation of generating the surprise map further comprises operations of:

splitting the current frame into four fully saturated color channels, an R-channel (R), a G-channel (G), a B-channel (B), and a Y-channel (Y);

generating a light intensity channel (L) and a dark intensity channel (D) from the current frame;

generating a series of color feature maps from the four color channels for the current frame using center-surround differencing between color channels, each color feature map representing a color feature;

generating a series of intensity feature maps from the two intensity channels for the current frame using center-surround differencing, each intensity feature map representing a color/intensity feature;

comparing the light intensity channel (L) from the current frame with an light intensity channel (L) from the previous frame to generate five motion feature maps, each motion feature map representing a motion feature;

generating a prior map (P) for each of the color and motion features by integrating the motion feature maps and the color feature maps over time;

updating the prior map as new current frames are received from the sensors;

generating a surprise feature map, for each feature, by determining a rectified difference between the feature map from the newest frame and its corresponding prior map;

generating a surprise color conspicuity map by combining and normalizing the RG and BY surprise feature maps;

generating an intensity surprise conspicuity map by normalizing the LD surprise feature map;

generating a motion surprise conspicuity map by combining and normalizing the five surprise feature maps for motion;

generating the surprise map by combining and normalizing the surprise conspicuity maps, the surprise map having a plurality of values corresponding to spatial locations within the scene.

29. The method as set forth in claim 28, further comprising an act of stabilizing the prior map through receiving new current frames and updating the prior map until a feature variance is less than a predetermined threshold.

30. The method as set forth in claim 29, wherein in receiving a series of consecutive frames representing a scene as captured from N sensors, N is a plurality of fixed sensors and each sensor has a memory containing its own prior map (P) and the previous frame.

31. The method as set forth in claim 30, further comprising an act of memory management through selectively using a subset of the plurality of fixed the sensors to capture the series of consecutive frames.

32. The method as set forth in claim 31, wherein in selectively using the subset, the subset of sensors is selected such that the subset is active and used during a burst period, and deactivated during a wait period.

33. The method as set forth in claim 32, further comprising an act of performing a memory management function selected from a group consisting of store all, store none, store motion, and store priors, as follows:

wherein in store all, the prior map (P) and the previous frame are maintained over the wait period;

wherein in store none, both the prior map and previous frame are deleted during the wait period;

wherein in store motion, the prior map is deleted during the wait period and the previous frame is maintained over the wait period; and

wherein in store priors, the previous frame is deleted during the wait period and the prior map is maintained over the wait period.

34. The method as set forth in claim 33, further comprising an act of allowing a user to selectively remove a sensor from the plurality of fixed sensors, leaving remaining sensors, which causes the system to adjust the memory management function by decreasing the wait period assigned to the remaining sensors.

35. The method as set forth in claim 34, wherein in receiving a series of consecutive frames, if the frames are in black and white, the frames are converted into an red/green/blue (rgb) format of three channels, an red-channel (r), a g-channel (g), and a b-channel (b), where all three channels have the same value.

36. The method as set forth in claim 35, wherein in splitting the current frame into four fully saturated channels, the channels are split as follows:

37. The method as set forth in claim 36, wherein when generating a light intensity channel (L) and a dark intensity channel (D) from each frame, they are generated by averaging the r, g, and b-channels, as follows:

L=(r+g+b)/3

D=(maximum value of a color channel)−

L.

38. The method as set forth in claim 37, wherein when generating five motion feature maps, the motion feature maps are generated based on motion across four cardinal directions and a glimmer.

39. The method as set forth in claim 38, further comprising an act of weighting the surprise feature maps to modulate the system'"'"'s surprise response to specific features.