Attribute-based alert ranking for alert adjudication

• US 9,230,174 B2
• Filed: 03/18/2015
• Issued: 01/05/2016
• Est. Priority Date: 01/31/2013
• Status: Active Grant

• Current Assignee: Kyndryl Incorporated
• Original Assignee: International Business Machines Corporation
• Inventors: Fan, Quanfu, Pankanti, Sharathchandra U.
• Primary Examiner(s): Lu, Tom Y
• Application Number:
  US14/661,071
• Publication Number:
  US 20150193661A1
• Time in Patent Office: 293 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 18/2415 based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30232 Surveillance
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/248 involving reference images or patches
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G06V 20/46 Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69

Abstract

Alerts to object behaviors are prioritized for adjudication as a function of relative values of abandonment, foregroundness and staticness attributes. The attributes are determined from feature data extracted from video frame image data. The abandonment attribute indicates a level of likelihood of abandonment of an object. The foregroundness attribute quantifies a level of separation of foreground image data of the object from a background model of the image scene. The staticness attribute quantifies a level of stability of dimensions of a bounding box of the object over time. Alerts are also prioritized according to an importance or relevance value that is learned and generated from the relative abandonment, foregroundness and staticness attribute strengths.

Claims

1. A method for prioritizing the adjudication of object alerts as a function of relative visual attribute values, the method comprising:

in response to detecting an object that is discernible and static within an image scene of a video data input, a processing unit;

generating an alert;

tracking the detected object over a tracking period of time;

extracting image features from the video data input over the tracking time period;

learning and ranking relative strengths of each of a plurality of attributes from the extracted features, wherein the plurality of attributes comprises a foregroundness attribute and a staticness attribute; and

prioritizing the alert relative to other alerts according to a learned importance or relevance value generated from the relative foregroundness and staticness attribute strengths.

2. The method of claim 1, wherein the foregroundness attribute quantifies a level of separation of foreground image data of the object from a background model of the image scene;

and

wherein the staticness attribute quantifies a level of stability of dimensions of a bounding box of the object over time.

3. The method of claim 2, wherein the plurality of attributes further comprises an abandonment attribute that indicates a level of likelihood of abandonment of the object;

and

wherein the step of prioritizing the alert relative to the other alerts is according to a learned importance or relevance value generated from the relative abandonment, foregroundness and staticness attribute strengths.

4. The method of claim 3, wherein the step of prioritizing the alert relative to other alerts according to the learned importance or relevance value generated from the relative abandonment, foregroundness and staticness attribute strengths, comprises:

using a probabilistic binary support vector machine classifier that generates a probabilistic output from the abandonment, foregroundness and staticness attribute strengths.

5. The method of claim 3, wherein the step of prioritizing the alert relative to other alerts according to the learned importance or relevance value generated from the relative abandonment, foregroundness and staticness attribute strengths, further comprises:

sorting the alerts in a progressively decreasing order of their likelihood of being an image of the abandoned object of interest, then of the people, then lastly of the artifact that is not an image of the abandoned object of interest or of the people, as a function their foregroundness, staticness, and abandonment attribute values in view of a policy input that indicates a preference of people over the artifact that is not an image of the abandoned object of interest or of the people.

6. The method of claim 5, wherein the step of sorting the alerts comprises:

assigning a highest ranking value to an alert only in response to determining that each of the abandonment attribute value, the foregroundness attribute value and the staticness attribute value of the alert are highest values of their respective attribute values.

7. The method of claim 3, wherein the image features extracted from the video data input over the tracking time period are low-level image features that are relevant to the object over the tracking time period and comprise two or more of:

an amount of a change in size over time of a bounding box defining a foreground object blob distinguished from a background model of the image data and identified as a possible abandoned object;

an optical flow feature comprising motion flow data that is variable over time and associated with movement of the foreground object blob within its bounding box dimensions; and

a border contrast feature that is relevant to visually distinguishing the foreground blob from the background model, wherein the border contrast feature comprises sharpness, definition, or granularity attributes.

8. The method of claim 7, wherein the processing unit determines the abandonment attribute value by:

determining an amount of difference in motion activity before and after commencement of a period of static motion values associated with the object;

determining an amount of change in a size of the bounding box relative to a commencement of the period of static motion values; and

assigning the abandonment attribute value in proportion to the determined amount of difference in motion activity and the determined amount of change in the size of the bounding box.

9. The method of claim 8, wherein the step of the extracting low-level image features from the video data input that are relevant to the object over the time period comprises tracking movement of a location of the bounding box over the time period, and a size of the bounding box over the time period;

and

wherein the step of determining the abandonment attribute value from the extracted features comprises;

searching for another blob bounding box that overlaps with the bounding box at a point in time before a beginning of the tracking time period; and

determining a location distance between the bounding box and the another bounding box over the tracking time period in response to the searching finding the another bounding box that overlaps with the blob bounding box at the point in time before the beginning of the tracking time period.

10. The method of claim 9, further comprising determining the value of the abandonment attribute of the object from the extracted features as a function of:

determining a first ratio of an area value of the bounding box at the beginning of the tracking time period relative to an area value of the bounding box at an end of the tracking time period;

determining a second ratio of a height value of the bounding box at the beginning of the tracking time period relative to a height value of the bounding box at the end of the tracking time period;

determining a third ratio of the area value of the bounding box relative to an area value of the another bounding box over the tracking time period; and

determining a fourth ratio of the height value of the bounding box relative to a height value of the another bounding box over the tracking time period.

11. The method of claim 10, wherein the step of determining the staticness attribute value of the object comprises:

determining an average amount of movement of the bounding box over the tracking time period via a finite state machine;

determining a maximum amount of movement of the bounding box over the tracking time period via the finite state machine;

determining the staticness attribute value as a function of a ratio of a match of an amount of movement of the bounding box at the beginning of the tracking time period to at least one the determined average amount and the determined maximum amount of movement of the bounding box over the tracking time period.

12. The method of claim 1, further comprising:

integrating computer-readable program code into a computer system comprising the processing unit, a computer readable memory and a computer readable tangible storage medium;

wherein the computer readable program code is embodied on the computer readable tangible storage medium and comprises instructions for execution by the processing unit via the computer readable memory that cause the processing unit to automatically, in response to detecting the discernible and static object, generate the alert, track the detected object, extract the image features, learn and rank the relative foregroundness and staticness attribute strengths from the extracted features, determine the true event or false event label for the alert, determine the probabilistic relevance value of the detected object, and prioritize the alert relative to the other alerts according to the learned importance or relevance value generated from the relative foregroundness and staticness attribute strengths.

13. A system that automatically prioritizes the adjudication of object alerts as a function of relative visual attribute values, the system comprising:

a processing unit;

a computer readable memory in communication with the processing unit; and

a computer-readable storage medium in communication with the processing unit;

wherein the processing unit executes program instructions stored on the computer-readable storage medium via the computer readable memory and, in response to an indication that an object is discernible and static within an image scene of a video data input;

generates an alert;

tracks the detected object over a tracking period of time;

extracts image features from the video data input over the tracking time period;

learns and ranks relative strengths of each of a plurality of attributes from the extracted features, wherein the plurality of attributes comprises a foregroundness attribute that quantifies a level of separation of foreground image data of the object from a background model of the image scene, and a staticness attribute that quantifies a level of stability of dimensions of a bounding box of the object over time; and

prioritizes the alert relative to other alerts according to a learned importance or relevance value generated from the relative foregroundness and staticness attribute strengths.

14. The system of claim 13, wherein the foregroundness attribute quantifies a level of separation of foreground image data of the object from a background model of the image scene;

wherein the staticness attribute quantifies a level of stability of dimensions of a bounding box of the object over time;

wherein the plurality of attributes further comprises an abandonment attribute that indicates a level of likelihood of abandonment of the object; and

wherein the processing unit executes the program instructions stored on the computer-readable storage medium via the computer readable memory, and thereby prioritizes the alert relative to the other alerts according to a learned importance or relevance value generated from the relative abandonment, foregroundness and staticness attribute strengths.

15. The system of claim 14, wherein the extracted features comprise two or more of:

an amount of a change in size over time of a bounding box defining a foreground object blob distinguished from a background model of the image data and identified as a possible abandoned object;

an optical flow feature comprising motion flow data that is variable over time and associated with movement of the foreground object blob within its bounding box dimensions; and

a border contrast feature that is relevant to visually distinguishing the foreground blob from the background model, wherein the border contrast feature comprises sharpness, definition, or granularity attributes.

16. The system of claim 15, wherein the processing unit executes the program instructions stored on the computer-readable storage medium via the computer readable memory and thereby further determines the value of the abandonment attribute by:

determining an amount of difference in motion activity before and after commencement of a period of static motion values associated with the object;

determining an amount of change in a size of a bounding box associated with the object relative to a commencement of the period of static motion values; and

assigning the abandonment attribute value in proportion to the determined amount of difference in motion activity and the determined amount of change in the size of the bounding box.

17. A computer program product for automatically prioritizing the adjudication of object alerts as a function of relative visual attribute values, the computer program product comprising:

a computer readable hardware storage device having computer readable program code embodied therewith, the computer readable program code comprising instructions for execution by a computer processing unit that cause the computer processing unit to;

generate an alert;

track the detected object over a tracking period of time;

extract image features from the video data input over the tracking time period;

learn and rank relative strengths of each of a plurality of attributes from the extracted features, wherein the plurality of attributes comprises a foregroundness attribute that quantifies a level of separation of foreground image data of the object from a background model of the image scene, and a staticness attribute that quantifies a level of stability of dimensions of a bounding box of the object over time; and

prioritize the alert relative to other alerts according to a learned importance or relevance value generated from the relative foregroundness and staticness attribute strengths.

18. The computer program product of claim 17, wherein the foregroundness attribute quantifies a level of separation of foreground image data of the object from a background model of the image scene;

wherein the staticness attribute quantifies a level of stability of dimensions of a bounding box of the object over time;

wherein the plurality of attributes further comprises an abandonment attribute that indicates a level of likelihood of abandonment of the object; and

wherein the computer readable program code instructions for execution by the computer processing unit further cause the computer processing unit to prioritize the alert relative to the other alerts according to a learned importance or relevance value generated from the relative abandonment, foregroundness and staticness attribute strengths.

19. The computer program product of claim 17, the computer readable program code instructions for execution by the computer processing unit further cause the computer processing unit to sort the alerts in a progressively decreasing order of their likelihood of being an image of the abandoned object of interest, then of the people, then lastly of the artifact that is not an image of the abandoned object of interest or of the people, as a function their foregroundness, staticness, and abandonment attribute values in view of a policy input that indicates a preference of people over the artifact that is not an image of the abandoned object of interest or of the people.

20. The computer program product of claim 19, wherein the computer readable program code instructions for execution by the computer processing unit further cause the computer processing unit to determine the value of the abandonment attribute by:

determining an amount of difference in motion activity before and after commencement of a period of static motion values associated with the object;

determining an amount of change in a size of a bounding box associated with the object relative to a commencement of the period of static motion values; and

assigning the abandonment attribute value in proportion to the determined amount of difference in motion activity and the determined amount of change in the size of the bounding box; and

wherein the extracted features comprise two or more of;

an amount of a change in size over time of a bounding box defining a foreground object blob distinguished from a background model of the image data and identified as a possible abandoned object;

an optical flow feature comprising motion flow data that is variable over time and associated with movement of the foreground object blob within its bounding box dimensions; and

a border contrast feature that is relevant to visually distinguishing the foreground blob from the background model, wherein the border contrast feature comprises sharpness, definition, or granularity attributes.