ATTRIBUTE-BASED ALERT RANKING FOR ALERT ADJUDICATION

• US 20140211988A1
• Filed: 01/31/2013
• Published: 07/31/2014
• Est. Priority Date: 01/31/2013
• Status: Active Grant

• Current Assignee: Kyndryl Incorporated
• Original Assignee: International Business Machines Corporation
• Inventors: Fan, Quanfu, Pankanti, Sharathchandra U.
• Granted Patent: US 9,020,190 B2
• Time in Patent Office: Days
• US Class Current: 382/103
• CPC Class Codes: G06F 18/2415 based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30232 Surveillance
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/248 involving reference images or patches
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G06V 20/46 Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69

Abstract

Alerts to object behaviors are prioritized for adjudication as a function of relative values of abandonment, foregroundness and staticness attributes. The attributes are determined from feature data extracted from video frame image data. The abandonment attribute indicates a level of likelihood of abandonment of an object. The foregroundness attribute quantifies a level of separation of foreground image data of the object from a background model of the image scene. The staticness attribute quantifies a level of stability of dimensions of a bounding box of the object over time. Alerts are also prioritized according to an importance or relevance value that is learned and generated from the relative abandonment, foregroundness and staticness attribute strengths.

Claims

1. A method for prioritizing the adjudication of object alerts as a function of relative visual attribute values, the method comprising:

in response to detecting an object that is discernible and static within an image scene of a video data input, a processing unit;

generating an alert;

tracking the detected object over a tracking period of time;

extracting low-level image features from the video data input that are relevant to the object over the tracking time period;

learning and ranking relative strengths of each of a plurality of attributes from the extracted features, wherein the plurality of attributes comprises an abandonment attribute that indicates a level of likelihood of abandonment of the object, a foregroundness attribute that quantifies a level of separation of foreground image data of the object from a background model of the image scene, and a staticness attribute that quantifies a level of stability of dimensions of a bounding box of the object over time; and

prioritizing the alert relative to other alerts according to a learned importance or relevance value generated from the relative abandonment, foregroundness and staticness attribute strengths.

2. The method of claim 1, wherein the step of prioritizing the alert relative to other alerts according to the learned importance or relevance value generated from the relative abandonment, foregroundness and staticness attribute strengths, comprises:

using a probabilistic binary support vector machine classifier that generates a probabilistic output from the abandonment, foregroundness and staticness attribute strengths.

3. The method of claim 1, wherein the step of prioritizing the alert relative to other alerts according to the learned importance or relevance value generated from the relative abandonment, foregroundness and staticness attribute strengths, further comprises:

sorting the alerts in a progressively decreasing order of their likelihood of being an image of the abandoned object of interest, then of the people, then lastly of the artifact that is not an image of the abandoned object of interest or of the people, as a function their foregroundness, staticness, and abandonment attribute values in view of a human knowledge policy input that indicates a preference of people over the artifact that is not an image of the abandoned object of interest or of the people.

4. The method of claim 3, wherein the step of sorting the alerts comprises:

assigning a highest ranking value to an alert only if each of the abandonment attribute value, the foregroundness attribute value and the staticness attribute value of the alert are highest values of their respective attribute values.

5. The method of claim 1, wherein the extracted features comprise two or more of:

an amount of a change in size over time of a bounding box defining a foreground object blob distinguished from a background model of the image data and identified as a possible abandoned object;

an optical flow feature comprising motion flow data that is variable over time and associated with movement of the foreground object blob within its bounding box dimensions; and

a border contrast feature that is relevant to visually distinguishing the foreground blob from the background model, wherein the border contrast feature comprises sharpness, definition, or granularity attributes.

6. The method of claim 5, wherein the processing unit determines the abandonment attribute value by:

determining an amount of difference in motion activity before and after commencement of a period of static motion values associated with the object;

determining an amount of change in a size of the bounding box relative to a commencement of the period of static motion values; and

assigning the abandonment attribute value in proportion to the determined amount of difference in motion activity and the determined amount of change in the size of the bounding box.

7. The method of claim 6, wherein the step of the extracting low-level image features from the video data input that are relevant to the object over the time period comprises tracking movement of a location of the bounding box over the time period, and a size of the bounding box over the time period;

and

wherein the step of determining the abandonment attribute value from the extracted features comprises;

searching for another blob bounding box that overlaps with the bounding box at a point in time before a beginning of the tracking time period; and

determining a location distance between the bounding box and the another bounding box over the tracking time period if the searching finds the another bounding box that overlaps with the blob bounding box at the point in time before the beginning of the tracking time period.

8. The method of claim 7, further comprising determining the value of the abandonment attribute of the object from the extracted features as a function of:

determining a first ratio of an area value of the bounding box at the beginning of the tracking time period relative to an area value of the bounding box at an end of the tracking time period;

determining a second ratio of a height value of the bounding box at the beginning of the tracking time period relative to a height value of the bounding box at the end of the tracking time period;

determining a third ratio of the area value of the bounding box relative to an area value of the another bounding box over the tracking time period; and

determining a fourth ratio of the height value of the bounding box relative to a height value of the another bounding box over the tracking time period.

9. The method of claim 8, wherein the step of determining the staticness attribute value of the object comprises:

determining an average amount of movement of the bounding box over the tracking time period via a finite state machine;

determining a maximum amount of movement of the bounding box over the tracking time period via the finite state machine;

determining the staticness attribute value as a function of a ratio of a match of an amount of movement of the bounding box at the beginning of the tracking time period to at least one the determined average amount and the determined maximum amount of movement of the bounding box over the tracking time period.

10. The method of claim 1, further comprising:

integrating computer-readable program code into a computer system comprising the processing unit, a computer readable memory and a computer readable tangible storage medium;

wherein the computer readable program code is embodied on the computer readable tangible storage medium and comprises instructions that, when executed by the processing unit via the computer readable memory, cause the processing unit to automatically, in response to detecting the discernible and static object, generate the alert, track the detected object, extract the image features relevant to the object over the tracking time period, learn and rank the relative abandonment, foregroundness and staticness attribute strengths from the extracted features, determine the true event or false event label for the alert, determine the probabilistic relevance value of the detected object, and prioritize the alert relative to the other alerts according to the learned importance or relevance value generated from the relative abandonment, foregroundness and staticness attribute strengths.

11. A system that automatically prioritizes the adjudication of object alerts as a function of relative visual attribute values, the system comprising:

a processing unit in communication with a computer readable memory and a tangible computer-readable storage medium;

wherein the processing unit, when executing program instructions stored on the tangible computer-readable storage medium via the computer readable memory, in response to an indication that an object is discernible and static within an image scene of a video data input;

tracks the detected object over a tracking period of time;

extracts low-level image features from the video data input that are relevant to the object over the tracking time period;

learns and ranks relative strengths of each of a plurality of attributes from the extracted features, wherein the plurality of attributes comprises an abandonment attribute that indicates a level of likelihood of abandonment of the object, a foregroundness attribute that quantifies a level of separation of foreground image data of the object from a background model of the image scene, and a staticness attribute that quantifies a level of stability of dimensions of a bounding box of the object over time; and

prioritizes the alert relative to other alerts according to a learned importance or relevance value generated from the relative abandonment, foregroundness and staticness attribute strengths.

12. The system of claim 11, wherein the extracted features comprise two or more of:

an amount of a change in size over time of a bounding box defining a foreground object blob distinguished from a background model of the image data and identified as a possible abandoned object;

an optical flow feature comprising motion flow data that is variable over time and associated with movement of the foreground object blob within its bounding box dimensions; and

a border contrast feature that is relevant to visually distinguishing the foreground blob from the background model, wherein the border contrast feature comprises sharpness, definition, or granularity attributes.

13. The system of claim 12, wherein the processing unit, when executing the program instructions stored on the computer-readable storage medium via the computer readable memory, further determines the value of the abandonment attribute by:

determining an amount of difference in motion activity before and after commencement of a period of static motion values associated with the object;

determining an amount of change in a size of a bounding box associated with the object relative to a commencement of the period of static motion values; and

assigning the abandonment attribute value in proportion to the determined amount of difference in motion activity and the determined amount of change in the size of the bounding box.

14. The system of claim 13, wherein the processing unit, when executing the program instructions stored on the computer-readable storage medium via the computer readable memory, prioritizes the alert relative to other alerts according to the learned importance or relevance value generated from the relative abandonment, foregroundness and staticness attribute strengths by using a probabilistic binary support vector machine classifier that generates a probabilistic output from the abandonment, foregroundness and staticness attribute strengths.

15. The system of claim 13, wherein the processing unit, when executing the program instructions stored on the computer-readable storage medium via the computer readable memory, prioritizes the alert relative to other alerts according to the learned importance or relevance value generated from the relative abandonment, foregroundness and staticness attribute strengths by:

sorting the alerts in a progressively decreasing order of their likelihood of being an image of the abandoned object of interest, then of the people, then lastly of the artifact that is not an image of the abandoned object of interest or of the people, as a function their foregroundness, staticness, and abandonment attribute values in view of a human knowledge policy input that indicates a preference of people over the artifact that is not an image of the abandoned object of interest or of the people.

16. A computer program product for automatically prioritizing the adjudication of object alerts as a function of relative visual attribute values, the computer program product comprising:

a computer readable tangible storage medium having computer readable program code embodied therewith, the computer readable program code comprising instructions that, when executed by a computer processing unit, cause the computer processing unit to;

track a detected object over a tracking period of time in response to an indication that the object is discernible and static within an image scene of a video data input;

extract low-level image features from the video data input that are relevant to the object over the tracking time period;

learn and rank relative strengths of each of plurality of attributes from the extracted features, wherein the plurality of attributes comprises an abandonment attribute that indicates a level of likelihood of abandonment of the object. a foregroundness attribute that quantifies a level of separation of foreground image data of the object from a background model of the image scene, and a staticness attribute that quantifies a level of stability of dimensions of a bounding box of the object over time; and

prioritize the alert relative to other alerts according to a learned importance or relevance value generated from the relative abandonment, foregroundness and staticness attribute strengths.

17. The computer program product of claim 16, wherein the computer readable program code instructions, when executed by the computer processing unit, further cause the computer processing unit to determine the value of the abandonment attribute by:

determining an amount of difference in motion activity before and after commencement of a period of static motion values associated with the object;

determining an amount of change in a size of a bounding box associated with the object relative to a commencement of the period of static motion values; and

assigning the abandonment attribute value in proportion to the determined amount of difference in motion activity and the determined amount of change in the size of the bounding box; and

wherein the extracted features comprise two or more of;

an amount of a change in size over time of a bounding box defining a foreground object blob distinguished from a background model of the image data and identified as a possible abandoned object;

an optical flow feature comprising motion flow data that is variable over time and associated with movement of the foreground object blob within its bounding box dimensions; and

a border contrast feature that is relevant to visually distinguishing the foreground blob from the background model, wherein the border contrast feature comprises sharpness, definition, or granularity attributes.

18. The computer program product of claim 17, wherein the computer readable program code instructions, when executed by the computer processing unit, further cause the computer processing unit to prioritize the alert relative to other alerts according to the learned importance or relevance value generated from the relative abandonment, foregroundness and staticness attribute strengths by using a probabilistic binary support vector machine classifier that generates a probabilistic output from the abandonment, foregroundness and staticness attribute strengths.

19. The computer program product of claim 17, wherein the computer readable program code instructions, when executed by the computer processing unit, further cause the computer processing unit to prioritize the alert relative to other alerts according to the learned importance or relevance value generated from the relative abandonment, foregroundness and staticness attribute strengths by:

sorting the alerts in a progressively decreasing order of their likelihood of being an image of the abandoned object of interest, then of the people, then lastly of the artifact that is not an image of the abandoned object of interest or of the people, as a function their foregroundness, staticness, and abandonment attribute values in view of a human knowledge policy input that indicates a preference of people over the artifact that is not an image of the abandoned object of interest or of the people.

20. The computer program product of claim 17, wherein the computer readable program code instructions, when executed by the computer processing unit, further cause the computer processing unit to:

extract the image features from the video data input by tracking movement of a location of a bounding box of a motion blob of the detected object over the time period, and a size of the bounding box over the time period;

search for another blob bounding box that overlaps with the object blob bounding box at a point in time before a beginning of the tracking time period; and

determine a location distance between the object blob bounding box and the another blob bounding box over the tracking time period if the searching finds the another blob bounding box that overlaps with the object blob bounding box at the point in time before the beginning of the tracking time period.

21. The computer program product of claim 20, wherein the computer readable program code instructions, when executed by the computer processing unit, further cause the computer processing unit to determine the value of the abandonment attribute of the object from the extracted features as a function of:

determining a first ratio of an area value of the object blob bounding box at the beginning of the tracking time period relative to an area value of the object blob bounding box at an end of the tracking time period;

determining a second ratio of a height value of the object blob bounding box at the beginning of the tracking time period relative to a height value of the object blob bounding box at the end of the tracking time period;

determining a third ratio of the area value of the object blob bounding box relative to an area value of the another blob bounding box over the tracking time period; and

determining a fourth ratio of the height value of the object blob bounding box relative to a height value of the another blob bounding box over the tracking time period.