Graphical display and user-interface for high-speed triage of potential items of interest in imagery

• US 10,621,461 B1
• Filed: 03/10/2014
• Issued: 04/14/2020
• Est. Priority Date: 03/13/2013
• Status: Active Grant

• Current Assignee: HRL Laboratories LLC (The Boeing Co.)
• Original Assignee: HRL Laboratories LLC (The Boeing Co.)
• Inventors: Huber, David J., Khosla, Deepak, Martin, Kevin R., Chen, Yang
• Primary Examiner(s): Rush, Eric
• Application Number:
  US14/203,256
• Time in Patent Office: 2,227 Days
• Field of Search: 382100, 382103, 382159, 382173, 382181, 382224, 382282, 382291, 382307, 382308, 382328
• CPC Class Codes: G06F 18/2113 by ranking or filtering the set of features, e.g. using a measure of variance or of feature cross-correlation
• CPC Class Codes: G06F 18/2178 based on feedback of a supervisor
• CPC Class Codes: G06F 2218/00 Aspects of pattern recognition specially adapted for signal processing
• CPC Class Codes: G06V 10/25 Determination of region of interest [ROI] or a volume of interest [VOI]
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/7784 based on feedback from supervisors
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 20/176 Urban or other man-made structures
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: H04N 7/18 Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
• CPC Class Codes: H04N 7/188 Capturing isolated or intermittent images triggered by the occurrence of a predetermined event, e.g. an object reaching a predetermined position signal generation from motion picture films H04N5/253

Abstract

The present invention relates to a surveillance system and, more particularly, to a graphical display and user interface system that provides high-speed triage of potential items of interest in imagery. The system receives at least one image of a scene from a sensor. The image is pre-processed to identify a plurality of potential objects of interest (OI) in the image. The potential OI are presented to the user as a series of chips on a threat chip display (TCD), where each chip is a region extracted from the image that corresponds to a potential OI. Finally, the system allows the user to designate, via the TCD, any one of the chips as an actual OI.

Claims

1. A computer program product for processing imagery, the computer program product comprising:

a non-transitory computer-readable medium having executable instructions encoded thereon, such that upon execution of the instructions by one or more processors, the one or more processors perform operations of;

receiving at least one image of a scene in a visible spectrum from a sensor;

processing the at least one image to identify a plurality of potential objects of interest (OI) in the at least one image based on saliency or motion of the plurality of potential OI;

assigning a confidence score to each of the plurality of potential OI by displaying the plurality of potential OI to a user at a high seed using raid serial visual presentation (RSVP) while measuring the user'"'"'s electroencephalographic (EEG) response to the RSVP, with the confidence scores being based on the user'"'"'s EEG response to the RSVP;

altering, based on user selection, a minimum confidence score threshold for the plurality of potential OI;

presenting a full frame of the at least one image of the scene at a reduced resolution, the full frame having a location of the plurality of potential OI superimposed over the full frame such that the plurality of potential OI remain at the reduced resolution within the full frame;

presenting a plurality of potential OI that exceed the minimum confidence score as a series of chips on a threat chip display (TCD), the chips being presented at full resolution, where each chip is a region extracted from the at least one image that corresponds to a potential OI; and

designating, via the TCD, any one of the chips as an actual OI.

2. The computer program product as set forth in claim 1, wherein the chips are arranged on the TCD according to their confidence score.

3. The computer program product as set forth in claim 2, wherein the chips are tiled on the TCD such that the chip with the highest confidence score is located in an upper left pane.

4. The computer program product as set forth in claim 3, wherein if there are more potential OI chips than can be displayed on a single TCD page, then the chips are arranged into pages that a user can switch between such that the chip with the highest confidence score is located in an upper-left pane of a first TCD page, and chips with lower scores are found on subsequent TCD pages.

5. The computer program product as set forth in claim 4, wherein in processing the at least one image to identify a plurality of potential OI in the at least one image, a front-end image processing algorithm is used.

6. The computer program product as set forth in claim 5, further comprising instructions for causing the one or more processors to perform an operation of receiving an inhibit command from a user regarding a selected chip, the inhibit command adjusting the front-end image processing algorithm to prevent the front-end algorithm from designating the selected chip as a potential OI.

7. The computer program product as set forth in claim 6, wherein each chip is a static image.

8. The computer program product as set forth in claim 6, wherein each chip is a short video clip obtained from either the same location over several frames or the output of a tracking algorithm applied to the potential OI over a series of frames.

9. The computer program product as set forth in claim 1, further comprising instructions for causing the one or more processors to perform an operation of switching a chip, based on user selection, between a recorded video and a live feed.

10. The computer program product as set forth in claim 1, further comprising instructions for causing the one or more processors to perform an operation of switching, based on user selection, between the full frame with the reduced resolution and the TCD having chips presented at full resolution.

11. The computer program product as set forth in claim 1, further comprising instructions for causing the one or more processors to perform an operation of altering, based on user selection, a threat class threshold to increase or decrease a number of chips being displayed in the TCD.

12. The computer program product as set forth in claim 1, wherein in processing the at least one image to identify a plurality of potential OI in the at least one image, a front-end image processing algorithm is used, and further comprising instructions for causing the one or more processors to perform an operation of receiving an ignore command from a user regarding a potential OI that corresponds to a foreground object, the ignore command causing the front-end image processing algorithm to avoid processing a spatial location in the at least one image associated with the foreground object such that future TCDs exclude instances of the spatial location.

13. A system for processing imagery, the system comprising:

one or more processors and a memory, the memory being a non-transitory computer-readable medium having executable instructions encoded thereon, such that upon execution of the instructions, the one or more processors perform operations of;

receiving at least one image of a scene in a visible spectrum from a sensor;

processing the at least one image to identify a plurality of potential objects of interest (OI) in the at least one image based on saliency or motion of the plurality of potential OI;

assigning a confidence score to each of the plurality of potential OI by displaying the plurality of potential OI to a user at a high seed using rapid serial visual presentation (RSVP) while measuring the user'"'"'s a electroencephalographic (EEG) response to the RSVP, with the confidence scores being based on the user'"'"'s EEG response to the RSVP;

a altering, based on user selection, a minimum confidence score threshold for the plurality of potential OI;

presenting a full frame of the at least one image of the scene at a reduced resolution, the full frame having a location of the plurality of potential OI superimposed over the full frame such that the plurality of potential OI remain at the reduced resolution within the full frame;

presenting a plurality of potential OI that exceed the minimum confidence score as a series of chips on a threat chip display (TCD), the chips being presented at full resolution, where each chip is a region extracted from the at least one image that corresponds to a potential OI; and

designating, via the TCD, any one of the chips as an actual OI.

14. The system as set forth in claim 13, wherein the chips arranged on the TCD according to their confidence score.

15. The system as set forth in claim 14, wherein the chips are tiled on the TCD such that the chip with the highest confidence score is located in an upper left pane.

16. The system as set forth in claim 15, wherein if there are more potential OI chips than can be displayed on a single TCD page, then the chips are arranged into pages that a user can switch between such that the chip with the highest confidence score is located in an upper-left pane of a first TCD page, and chips with lower scores are found on subsequent TCD pages.

17. The system as set forth in claim 16, wherein in processing the at least one image to identify a plurality of potential OI in the at least one image, a front-end image processing algorithm is used.

18. The system as set forth in claim 17, further comprising instructions for causing the one or more processors to perform an operation of receiving an inhibit command from a user regarding a selected chip, the inhibit command adjusting the front-end image processing algorithm to prevent the front-end algorithm from designating the selected chip as a potential OI.

19. The system as set forth in claim 18, wherein each chip is a static image.

20. The system as set forth in claim 18, wherein each chip is a short video clip obtained from either the same location over several frames or the output of a tracking algorithm applied to the potential OI over a series of frames.

21. The system as set forth in claim 14, wherein presenting the series of chips on a threat chip display replaces the full frame of the at least one image of the scene at a reduced resolution.

22. The system as set forth in claim 14, further comprising an operation of switching a chip between a live feed of the location and a recorded video of the location.

23. The system as set forth in claim 13, wherein in processing the at least one image to identify a plurality of potential OI in the at least one image, a front-end image processing algorithm is used, and further comprising instructions for causing the one or more processors to perform an operation of receiving an ignore command from a user regarding a potential OI that corresponds to a foreground object, the ignore command causing the front-end image processing algorithm to avoid processing a spatial location in the at least one image associated with the foreground object such that future TCDs exclude instances of the spatial location.

24. A computer implemented method for processing imagery, the method comprising an act of:

causing one or more processers to execute instructions encoded on a non-transitory computer-readable medium, such that upon execution, the one or more processors perform operations of;

receiving at least one image of a scene in a visible spectrum from a sensor;

processing the at least one image to identify a plurality of potential objects of interest (OI) in the at least one image based on saliency or motion of the plurality of potential OI;

assigning a confidence score to each of the plurality of potential OI by displaying the plurality of potential OI to a user at a high speed using rapid serial visual presentation (RSVP) while measuring the user'"'"'s electroencephalographic (EEG) response to the RSVP, with the confidence scores being based on the user'"'"'s EEG response to the RSVP;

altering, based on user selection, a minimum confidence score threshold for the plurality of potential OI;

presenting a full frame of the at least one image of the scene at a reduced resolution, the full frame having a location of the plurality of potential OI superimposed over the full frame such that the plurality of potential OI remain at the reduced resolution within the full frame;

presenting a plurality of potential OI that exceed the minimum confidence score as a series of chips on a threat chip display (TCD), the chips being presented at full resolution, where each chip is a region extracted from the at least one image that corresponds to a potential OI; and

designating, via the TCD, any one of the chips as an actual OI.

25. The method as set forth in claim 24, wherein the chip are arranged on the TCD according to their confidence score.

26. The method as set forth in claim 25, wherein the chips are tiled on the TCD such that the chip with the highest confidence score is located in an upper left pane.

27. The method as set forth in claim 26, wherein if there are more potential OI chips than can be displayed on a single TCD page, then the chips are arranged into pages that a user can switch between such that the chip with the highest confidence score is located in an upper-left pane of a first TCD page, and chips with lower scores are found on subsequent TCD pages.

28. The method as set forth in claim 27, wherein in processing the at least one image to identify a plurality of potential OI in the at least one image, a front-end image processing algorithm is used.

29. The method as set forth in claim 28, further comprising operations of receiving an inhibit command from a user regarding a selected chip, the inhibit command adjusting the front-end image processing algorithm to prevent the front-end algorithm from designating the selected chip as a potential OI.

30. The method as set forth in claim 29, wherein each chip is a static image.

31. The method as set forth in claim 29, wherein each chip is a short video clip obtained from either the same location over several frames or the output of a tracking algorithm applied to the potential OI over a series of frames.

32. The method as set forth in claim 24, wherein in processing the at least one image to identify a plurality of potential OI in the at least one image, a front-end image processing algorithm is used, and further comprising an operation of causing the one or more processors to perform an operation of receiving an ignore command from a user regarding a potential OI that corresponds to a foreground object, the ignore command causing the front-end image processing algorithm to avoid processing a spatial location in the at least one image associated with the foreground object such that future TCDs exclude instances of the spatial location.