Attribute-based alert ranking for alert adjudication

• US 9,330,314 B2
• Filed: 11/23/2015
• Issued: 05/03/2016
• Est. Priority Date: 01/31/2013
• Status: Expired due to Fees

• Current Assignee: Kyndryl Incorporated (Kyndryl Holdings, Inc.)
• Original Assignee: International Business Machines Corporation
• Inventors: Fan, Quanfu, Pankanti, Sharathchandra U.
• Primary Examiner(s): Lu, Tom Y
• Application Number:
  US14/948,458
• Publication Number:
  US 20160078295A1
• Time in Patent Office: 162 Days
• Field of Search: 382/103, 382/104, 382/107
• US Class Current: 1/1
• CPC Class Codes: G06F 18/2415 based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30232 Surveillance
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/248 involving reference images or patches
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G06V 20/46 Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69

Abstract

Alerts to object behaviors are prioritized for adjudication as a function of relative values of abandonment, foregroundness and staticness attributes. The attributes are determined from feature data extracted from video frame image data. The abandonment attribute indicates a level of likelihood of abandonment of an object. The foregroundness attribute quantifies a level of separation of foreground image data of the object from a background model of the image scene. The staticness attribute quantifies a level of stability of dimensions of a bounding box of the object over time. Alerts are also prioritized according to an importance or relevance value that is learned and generated from the relative abandonment, foregroundness and staticness attribute strengths.

Claims

1. A method for prioritizing the adjudication of object alerts as a function of relative visual attribute values, the method comprising:

in response to detecting an object that is discernible and static within an image scene of a video data input, a processing unit;

generating an alert;

tracking the detected object over a tracking period of time;

extracting image features from the video data input over the tracking time period;

learning and ranking relative strengths of each of a plurality of attributes from the extracted features, wherein the plurality of attributes comprises a foregroundness attribute and a staticness attribute;

identifying the detected object as one of a bag object, a people object and a visual artifact as a function of the learned and ranked relative strengths of the plurality of extracted feature attributes, wherein the bag object, the people object and the visual artifact are each associated with different combinations of values of learned and ranked relative strengths of the plurality of extracted feature attributes; and

prioritizing the alert relative to other alerts by ranking alerts generated from detected bag objects over alerts generated from detected people objects, and alerts generated from detected people object over alerts generated from detected visual artifacts.

2. The method of claim 1, wherein the foregroundness attribute quantifies a level of separation of foreground image data of the object from a background model of the image scene;

and

wherein the staticness attribute quantifies a level of stability of dimensions of a bounding box of the object over time.

3. The method of claim 2, wherein the plurality of attributes further comprises an abandonment attribute that indicates a level of likelihood of abandonment of the object.

4. The method of claim 3, wherein the step of identifying the detected object as one of the bag object, the people object and the visual artifact as the function of the learned and ranked relative strengths of the plurality of extracted feature attributes comprises:

using a probabilistic binary support vector machine classifier that generates a probabilistic output from the abandonment, foregroundness and staticness attribute strengths.

5. The method of claim 3, wherein the detected visual artifact is identified as one of a detected lighting change artifact, a detected shadow artifact and a detected ghost artifact, as a function of different combinations of values of the learned and ranked relative strengths of the plurality of extracted feature attributes that are each associated with the lighting change artifact, the detected shadow artifact and the detected ghost artifact.

6. The method of claim 3, wherein the step of sorting the alerts comprises:

assigning a highest ranking value to an alert in response to determining that each of the abandonment attribute value, the foregroundness attribute value and the staticness attribute value of the alert are highest values of their respective attribute values.

7. The method of claim 3, wherein the image features extracted from the video data input over the tracking time period are low-level image features that are relevant to the object over the tracking time period and comprise two or more of:

an amount of a change in size over time of a bounding box defining a foreground object blob distinguished from a background model of the image data and identified as a possible abandoned object;

an optical flow feature comprising motion flow data that is variable over time and associated with movement of the foreground object blob within its bounding box dimensions; and

a border contrast feature that is relevant to visually distinguishing the foreground blob from the background model, wherein the border contrast feature comprises sharpness, definition, or granularity attributes.

8. The method of claim 7, wherein the processing unit determines the abandonment attribute value by:

determining an amount of difference in motion activity before and after commencement of a period of static motion values associated with the object;

determining an amount of change in a size of the bounding box relative to a commencement of the period of static motion values; and

assigning the abandonment attribute value in proportion to the determined amount of difference in motion activity and the determined amount of change in the size of the bounding box.

9. The method of claim 8, wherein the step of the extracting low-level image features from the video data input that are relevant to the object over the time period comprises tracking movement of a location of the bounding box over the time period, and a size of the bounding box over the time period;

and

wherein the step of determining the abandonment attribute value from the extracted features comprises;

searching for another blob bounding box that overlaps with the bounding box at a point in time before a beginning of the tracking time period; and

determining a location distance between the bounding box and the another bounding box over the tracking time period in response to the searching finding the another bounding box that overlaps with the blob bounding box at the point in time before the beginning of the tracking time period.

10. The method of claim 9, further comprising determining the value of the abandonment attribute of the object from the extracted features as a function of:

determining a first ratio of an area value of the bounding box at the beginning of the tracking time period relative to an area value of the bounding box at an end of the tracking time period;

determining a second ratio of a height value of the bounding box at the beginning of the tracking time period relative to a height value of the bounding box at the end of the tracking time period;

determining a third ratio of the area value of the bounding box relative to an area value of the another bounding box over the tracking time period; and

determining a fourth ratio of the height value of the bounding box relative to a height value of the another bounding box over the tracking time period.

11. The method of claim 10, wherein the step of determining the staticness attribute value of the object comprises:

determining an average amount of movement of the bounding box over the tracking time period via a finite state machine;

determining a maximum amount of movement of the bounding box over the tracking time period via the finite state machine;

determining the staticness attribute value as a function of a ratio of a match of an amount of movement of the bounding box at the beginning of the tracking time period to at least one the determined average amount and the determined maximum amount of movement of the bounding box over the tracking time period.

12. The method of claim 1, further comprising:

integrating computer-readable program code into a computer system comprising the processing unit, a computer readable memory and a computer readable tangible storage medium;

wherein the computer readable program code is embodied on the computer readable tangible storage medium and comprises instructions for execution by the processing unit via the computer readable memory that cause the processing unit to automatically, in response to detecting the discernible and static object, generate the alert, track the detected object, extract the image features, learn and rank the relative foregroundness and staticness attribute strengths from the extracted features, identify the detected object as one of the bag object, the people object and the visual artifact as the function of the learned and ranked relative strengths of the plurality of extracted feature attributes, and prioritize the alert relative to other alerts by ranking alerts generated from detected bag objects over alerts generated from detected people objects, and alerts generated from detected people object over alerts generated from detected visual artifacts.

13. A system that automatically prioritizes the adjudication of object alerts as a function of relative visual attribute values, the system comprising:

a processing unit;

a computer readable memory in communication with the processing unit; and

a computer-readable storage medium in communication with the processing unit;

wherein the processing unit executes program instructions stored on the computer-readable storage medium via the computer readable memory and, in response to an indication that an object is discernible and static within an image scene of a video data input;

generates an alert;

tracks the detected object over a tracking period of time;

extracts image features from the video data input over the tracking time period;

learns and ranks relative strengths of each of a plurality of attributes from the extracted features, wherein the plurality of attributes comprises a foregroundness attribute that quantifies a level of separation of foreground image data of the object from a background model of the image scene, and a staticness attribute that quantifies a level of stability of dimensions of a bounding box of the object over time;

identify the detected object as one of a bag object, a people object and a visual artifact as a function of the learned and ranked relative strengths of the plurality of extracted feature attributes, wherein the bag object, the people object and the visual artifact are each associated with different combinations of values of learned and ranked relative strengths of the plurality of extracted feature attributes; and

prioritize the alert relative to other alerts by ranking alerts generated from detected bag objects over alerts generated from detected people objects, and alerts generated from detected people object over alerts generated from detected visual artifacts.

14. The system of claim 13, wherein the plurality of attributes further comprises an abandonment attribute that indicates a level of likelihood of abandonment of the object.

15. The system of claim 14, wherein the processing unit executes program instructions stored on the computer-readable storage medium via the computer readable memory and thereby identifies the detected object as one of the bag object, the people object and the visual artifact as the function of the learned and ranked relative strengths of the plurality of extracted feature attributes by using a probabilistic binary support vector machine classifier that generates a probabilistic output from the abandonment, foregroundness and staticness attribute strengths.

16. The system of claim 14, wherein the processing unit executes the program instructions stored on the computer-readable storage medium via the computer readable memory and thereby identifies the detected visual artifact as one of a detected lighting change artifact, a detected shadow artifact and a detected ghost artifact, as a function of different combinations of values of the learned and ranked relative strengths of the plurality of extracted feature attributes that are each associated with the lighting change artifact, the detected shadow artifact and the detected ghost artifact.

17. A computer program product for automatically prioritizing the adjudication of object alerts as a function of relative visual attribute values, the computer program product comprising:

a computer readable hardware storage device having computer readable program code embodied therewith, the computer readable program code comprising instructions for execution by a computer processing unit that cause the computer processing unit to;

generate an alert;

track the detected object over a tracking period of time;

extract image features from the video data input over the tracking time period;

learn and rank relative strengths of each of a plurality of attributes from the extracted features, wherein the plurality of attributes comprises a foregroundness attribute that quantifies a level of separation of foreground image data of the object from a background model of the image scene, and a staticness attribute that quantifies a level of stability of dimensions of a bounding box of the object over time;

identify the detected object as one of a bag object, a people object and a visual artifact as a function of the learned and ranked relative strengths of the plurality of extracted feature attributes, wherein the bag object, the people object and the visual artifact are each associated with different combinations of values of learned and ranked relative strengths of the plurality of extracted feature attributes; and

prioritize the alert relative to other alerts by ranking alerts generated from detected bag objects over alerts generated from detected people objects, and alerts generated from detected people object over alerts generated from detected visual artifacts.

18. The computer program product of claim 17, wherein the foregroundness attribute quantifies a level of separation of foreground image data of the object from a background model of the image scene;

wherein the staticness attribute quantifies a level of stability of dimensions of a bounding box of the object over time; and

wherein the plurality of attributes further comprises an abandonment attribute that indicates a level of likelihood of abandonment of the object.

19. The computer program product of claim 18, wherein the computer readable program code instructions for execution by the computer processing unit further cause the computer processing unit to identify the detected object as one of the bag object, the people object and the visual artifact as the function of the learned and ranked relative strengths of the plurality of extracted feature attributes by using a probabilistic binary support vector machine classifier that generates a probabilistic output from the abandonment, foregroundness and staticness attribute strengths.

20. The computer program product of claim 18, wherein the computer readable program code instructions for execution by the computer processing unit further cause the computer processing unit to identify the detected visual artifact as one of a detected lighting change artifact, a detected shadow artifact and a detected ghost artifact, as a function of different combinations of values of the learned and ranked relative strengths of the plurality of extracted feature attributes that are each associated with the lighting change artifact, the detected shadow artifact and the detected ghost artifact.