Efficient retrieval of anomalous events with priority learning

• US 9,928,423 B2
• Filed: 09/04/2015
• Issued: 03/27/2018
• Est. Priority Date: 05/18/2011
• Status: Active Grant

• Current Assignee: Kyndryl Incorporated (Kyndryl Holdings, Inc.)
• Original Assignee: International Business Machines Corporation
• Inventors: Datta, Ankur, Paluri, Balamanohar, Pankanti, Sharathchandra U., Zhai, Yun
• Primary Examiner(s): Moyer, Andrew
• Assistant Examiner(s): Rosario, Dennis
• Application Number:
  US14/845,438
• Publication Number:
  US 20150379357A1
• Time in Patent Office: 935 Days
• CPC Class Codes: G06F 18/2433 Single-class perspective, e.g. one-against-all classification; Novelty detection; Outlier detection
• CPC Class Codes: G06T 7/251 involving models
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G08B 13/19606 Discriminating between target movement or movement in an area of interest and other non-signicative movements, e.g. target movements induced by camera shake or movements of pets, falling leaves, rotating fan
• CPC Class Codes: G16B 40/00 ICT specially adapted for biostatistics; ICT specially adapted for bioinformatics-related machine learning or data mining, e.g. knowledge discovery or pattern finding

Abstract

Local models learned from anomaly detection are used to rank detected anomalies. The local model patterns are defined from image feature values extracted from an image field of video image data with respect to different predefined spatial and temporal local units, wherein anomaly results are determined by fitting extracted image features to the local model patterns. Image features values extracted from the image field local units associated with anomaly results are normalized, and image feature values extracted from the image field local units are clustered. Weights for anomaly results are learned as a function of the relations of the normalized extracted image feature values to the clustered image feature values. The normalized values are multiplied by the learned weights to generate ranking values to rank the anomalies.

Claims

1. A computer implemented method for detecting anomalies, the method comprising executing on a processor:

tracking movement of an object through an image field of a camera, wherein the object is detected within a video data input from the camera, wherein the image field is partitioned into a matrix comprising a grid of a plurality of different local units, and wherein the tracking generates a trajectory of the object'"'"'s motion that passes through a subset of the local units that is less than a totality of the plurality of the different local units;

extracting image features from the video data with respect to only each of the subset of the local units of the plurality of local units;

generating a plurality of anomaly confidence decision values for the extracted image features as a function of fitting the extracted image features to normal patterns of a plurality of local motion pattern models that are defined by dominant distributions of the extracted image features, and to anomaly patterns of the plurality of local motion pattern models that are defined by rare distributions of the extracted image features;

normalizing values of the image features extracted from the image field and associated with each of the plurality of anomaly confidence decision values;

clustering the image feature values extracted from the image field and associated with each of the plurality of anomaly confidence decision values;

determining spatial locations within the field of view of the clustered extracted image feature values of the anomaly confidence decision values that are correlated to features of interest of a real-world scene represented within the field of view;

assigning a first weighting to a first anomaly of the anomaly confidence decision values that is higher than a second weighting assigned to a second anomaly of the anomaly confidence decision values in response to the determined spatial location of the clustered extracted image feature values of said first anomaly confidence decision value being within a portion of the field of view of the input video that is correlated with a cordoned off area of the real-world scene, wherein the determined spatial location of the clustered extracted image feature values of the second anomaly confidence decision value is outside the portion;

ranking the plurality of anomaly confidence decision values as a function of their respective assigned weightings;

extracting trajectory features from the video data input relative to the trajectory of the tracked object;

generating a global anomaly confidence decision value for the object trajectory as a function of fitting the extracted trajectory features to a normal learned motion trajectory model, wherein the global anomaly confidence decision value indicates a likelihood that the object trajectory is normal or anomalous;

determining whether anomalies have occurred as a function of combining the generated global anomaly confidence decision value with the anomaly confidence decision values; and

prioritizing anomaly determinations as a function of the ranking of the anomaly confidence decision values.

2. The method of claim 1, further comprising:

integrating computer-readable program code into a computer system comprising the processor, a computer readable memory in circuit communication with the processor, and a computer readable storage medium in circuit communication with the processor; and

wherein the processor executes program code instructions stored on the computer-readable storage medium via the computer readable memory and thereby performs the tracking the movement of the object through the image field of the camera, the extracting the image features from the video data with respect to each of the subset of the local units of the plurality of local units by using said trajectory, the generating the plurality of anomaly confidence decision values, the normalizing the values of the image features extracted from the image field and associated with each of the plurality of anomaly confidence decision values, the clustering the image feature values extracted from the image field and associated with each of the plurality of anomaly confidence decision values, the determining the spatial locations within the field of view of the clustered extracted image feature values of the anomaly confidence decision values that are correlated to features of interest of a real-world scene represented within the field of view, the assigning the first weighting to the first anomaly of the anomaly confidence decision values that is higher than the second weighting assigned to a second anomaly of the anomaly confidence decision values, the ranking the plurality of anomaly confidence decision values as the function of their respective assigned weightings, the extracting the trajectory features from the video data input relative to the trajectory of the tracked object, the generating the global anomaly confidence decision value for the object trajectory, the determining whether anomalies have occurred and the prioritizing the anomaly determinations.

3. The method of claim 2, further comprising:

learning the normal patterns and the anomaly patterns of the plurality of local motion pattern models, one for each of the subset of the local units, wherein the normal patterns are defined by finding dominant distributions of the extracted image features within respective ones of the subset of the local units, and the anomaly patterns are defined by rare distributions of the extracted image features within the respective ones of the subset of the local units;

generating the plurality of anomaly confidence decision values as a function of fitting the image features extracted for each of the subset of the local units from the video data input of the tracked object to the plurality of the learned local motion pattern models of the respective subset of the local units by determining whether the features extracted relevant to the object'"'"'s motion within the video data indicate that the object'"'"'s motion within each particular local unit is one of said normal patterns or anomaly patterns in view of the plurality of the learned local motion pattern models for the local units; and

wherein the step of normalizing the values of the image features extracted from the image field and associated with each of the plurality of anomaly confidence decision values comprises normalizing values of the image features that are extracted from the image field'"'"'s subset of the local units that are associated with each of the plurality of anomaly confidence decision values; and

wherein the step of clustering the image feature values extracted from the image field and associated with each of the plurality of anomaly confidence decision values comprises clustering the image feature values extracted from the image field'"'"'s subset of the local units that are associated with the each of the plurality of anomaly confidence decision values.

4. The method of claim 3, wherein the normalized values of the extracted features of the anomaly confidence decision values of each of the subset of the local units of the first and the second anomaly confidence decision values are outliers from and have the same distance to a center of a cluster of extracted features of a one of the learned motion pattern local models.

5. The method of claim 4, wherein the step of ranking the plurality of anomaly confidence decision values as a function of their respective assigned weightings comprises:

multiplying the normalized values of the extracted features of the anomaly confidence decision values of the subset of the local units by their respective learned weights to generate respective ranking values; and

ranking the plurality of anomaly confidence decision values by their generated respective ranking values.

6. The method of claim 5, further comprising:

updating the weighting of one of the first and the second anomaly confidence decision values by increasing or decreasing the updated weighting relative to the weighting of an other of the first and the second anomaly confidence decision values as a function of a difference in relative distances of the respective normalized extracted features of the subset local units of the first and the second anomaly confidence decision values from a center of a cluster of extracted features of a same associated learned motion pattern local model.

7. The method of claim 6, further comprising:

updating the weighting of one of the first and the second anomaly confidence decision values by increasing or decreasing the updated weighting relative to the weighting of an other of the first and the second anomaly confidence decision values as a function of a difference in temporal data of the extracted data of the subset local units of the respective first and second anomaly confidence decision values.

8. The method of claim 7, wherein the difference in the temporal data is a longer duration time of the tracked object movement.

9. The method of claim 7, wherein the difference in the temporal data indicates a difference in a time of day of occurrences of the respective first and second anomaly confidence decision values.

10. A system, comprising:

a processor;

computer readable memory in circuit communication with the processor; and

a computer readable storage medium in circuit communication with the processor and the computer readable memory; and

wherein the processor executes program instructions stored on the computer-readable storage medium via the computer readable memory and thereby;

tracks movement of an object through an image field of a camera, wherein the object is detected within a video data input from the camera, wherein the image field is partitioned into a matrix comprising a grid of a plurality of different local units, and wherein the tracking generates a trajectory of the object'"'"'s motion that passes through a subset of the local units that is less than a totality of the plurality of the different local units;

extracts image features from the video data with respect to only each of the subset of the local units of the plurality of local units;

generates a plurality of anomaly confidence decision values for the extracted image features as a function of fitting the extracted image features to normal patterns of a plurality of local motion pattern models that are defined by dominant distributions of the extracted image features, and to anomaly patterns of the plurality of local motion pattern models that are defined by rare distributions of the extracted image features;

normalizes values of the image features extracted from the image field and associated with each of the plurality of anomaly confidence decision values;

clusters the image feature values extracted from the image field and associated with each of the plurality of anomaly confidence decision values;

determines spatial locations within the field of view of the clustered extracted image feature values of the anomaly confidence decision values that are correlated to features of interest of a real-world scene represented within the field of view;

assigns a first weighting to a first anomaly of the anomaly confidence decision values that is higher than a second weighting assigned to a second anomaly of the anomaly confidence decision values in response to the determined spatial location of the clustered extracted image feature values of said first anomaly confidence decision value being within a portion of the field of view of the input video that is correlated with a cordoned off area of the real-world scene, wherein the determined spatial location of the clustered extracted image feature values of the second anomaly confidence decision value is outside the portion;

ranks the plurality of anomaly confidence decision values as a function of their respective assigned weightings;

extracts trajectory features from the video data input relative to the trajectory of the tracked object;

generates a global anomaly confidence decision value for the object trajectory as a function of fitting the extracted trajectory features to a normal learned motion trajectory model, wherein the global anomaly confidence decision value indicates a likelihood that the object trajectory is normal or anomalous;

determines whether anomalies have occurred as a function of combining the generated global anomaly confidence decision value with the anomaly confidence decision values; and

prioritizes anomaly determinations as a function of the ranking of the anomaly confidence decision values.

11. The system of claim 10, wherein the processor executes the program instructions stored on the computer-readable storage medium via the computer readable memory and thereby further:

learns the normal patterns and the anomaly patterns of the plurality of local motion pattern models, one for each of the subset of the local units, wherein the normal patterns are defined by finding dominant distributions of the extracted image features within respective ones of the subset of the local units, and the anomaly patterns are defined by rare distributions of the extracted image features within the respective ones of the subset of the local units; and

generates the plurality of anomaly confidence decision values as a function of fitting the image features extracted for each of the subset of the local units from the video data input of the tracked object to the plurality of the learned local motion pattern models of the respective subset of the local units by determining whether the features extracted relevant to the object'"'"'s motion within the video data indicate that the object'"'"'s motion within each particular local unit is one of said normal patterns or anomaly patterns in view of the plurality of the learned local motion pattern models for the local units; and

wherein the processor normalizes the values of the image features extracted from the image field and associated with each of the plurality of anomaly confidence decision values by normalizing values of the image features that are extracted from the image field'"'"'s subset of the local units that are associated with each of the plurality of anomaly confidence decision values; and

wherein the processor clusters the image feature values extracted from the image field and associated with each of the plurality of anomaly confidence decision values by clustering the image feature values extracted from the image field'"'"'s subset of the local units that are associated with the each of the plurality of anomaly confidence decision values.

12. The system of claim 11, wherein the normalized values of the extracted features of the anomaly confidence decision values of each of the subset of the local units of the first and the second anomaly confidence decision values are outliers from and have the same distance to a center of a cluster of extracted features of a one of the learned motion pattern local models.

13. The system of claim 12, wherein the processor executes the program instructions stored on the computer-readable storage medium via the computer readable memory and thereby ranks the plurality of anomaly confidence decision values as a function of their respective assigned weightings by:

multiplying the normalized values of the extracted features of the anomaly confidence decision values of the subset of the local units by their respective learned weights to generate respective ranking values; and

ranking the plurality of anomaly confidence decision values by their generated respective ranking values.

14. The system of claim 13, wherein the processor executes the program instructions stored on the computer-readable storage medium via the computer readable memory and thereby:

updates the weighting of one of the first and the second anomaly confidence decision values by increasing or decreasing the updated weighting relative to the weighting of an other of the first and the second anomaly confidence decision values as a function of a difference in relative distances of the respective normalized extracted features of the subset local units of the first and the second anomaly confidence decision values from a center of a cluster of extracted features of a same associated learned motion pattern local model.

15. A computer program product, comprising:

a computer readable hardware storage medium device having computer readable program code embodied therewith, the computer readable program code comprising instructions for execution by a computer processor that cause the computer processor to;

track movement of an object through an image field of a camera, wherein the object is detected within a video data input from the camera, wherein the image field is partitioned into a matrix comprising a grid of a plurality of different local units, and wherein the tracking generates a trajectory of the object'"'"'s motion that passes through a subset of the local units that is less than a totality of the plurality of the different local units;

extract image features from the video data with respect to only each of the subset of the local units of the plurality of local units;

generate a plurality of anomaly confidence decision values for the extracted image features as a function of fitting the extracted image features to normal patterns of a plurality of local motion pattern models that are defined by dominant distributions of the extracted image features, and to anomaly patterns of the plurality of local motion pattern models that are defined by rare distributions of the extracted image features;

normalize values of the image features extracted from the image field and associated with each of the plurality of anomaly confidence decision values;

cluster the image feature values extracted from the image field and associated with each of the plurality of anomaly confidence decision values;

determine spatial locations within the field of view of the clustered extracted image feature values of the anomaly confidence decision values that are correlated to features of interest of a real-world scene represented within the field of view;

assign a first weighting to a first anomaly of the anomaly confidence decision values that is higher than a second weighting assigned to a second anomaly of the anomaly confidence decision values in response to the determined spatial location of the clustered extracted image feature values of said first anomaly confidence decision value being within a portion of the field of view of the input video that is correlated with a cordoned off area of the real-world scene, wherein the determined spatial location of the clustered extracted image feature values of the second anomaly confidence decision value is outside the portion;

rank the plurality of anomaly confidence decision values as a function of their respective assigned weightings;

extract trajectory features from the video data input relative to the trajectory of the tracked object;

generate a global anomaly confidence decision value for the object trajectory as a function of fitting the extracted trajectory features to a normal learned motion trajectory model, wherein the global anomaly confidence decision value indicates a likelihood that the object trajectory is normal or anomalous; and

determine whether anomalies have occurred as a function of combining the generated global anomaly confidence decision value with the anomaly confidence decision values; and

prioritize anomaly determinations as a function of the ranking of the anomaly confidence decision values.

16. The computer program product of claim 15, wherein the computer readable program code instructions for execution by the computer processor, further cause the computer processor to:

learn the normal patterns and the anomaly patterns of the plurality of local motion pattern models, one for each of the subset of the local units, wherein the normal patterns are defined by finding dominant distributions of the extracted image features within respective ones of the subset of the local units, and the anomaly patterns are defined by rare distributions of the extracted image features within the respective ones of the subset of the local units; and

generate the plurality of anomaly confidence decision values as a function of fitting the image features extracted for each of the subset of the local units from the video data input of the tracked object to the plurality of the learned local motion pattern models of the respective subset of the local units by determining whether the features extracted relevant to the object'"'"'s motion within the video data indicate that the object'"'"'s motion within each particular local unit is one of said normal patterns or anomaly patterns in view of the plurality of the learned local motion pattern models for the local units; and

wherein the processor is caused to normalize the values of the image features extracted from the image field and associated with each of the plurality of anomaly confidence decision values by normalizing values of the image features that are extracted from the image field'"'"'s subset of the local units that are associated with each of the plurality of anomaly confidence decision values; and

wherein the processor is caused to cluster the image feature values extracted from the image field and associated with each of the plurality of anomaly confidence decision values by clustering the image feature values extracted from the image field'"'"'s subset of the local units that are associated with the each of the plurality of anomaly confidence decision values.

17. The computer program product of claim 16, wherein the normalized values of the extracted features of the anomaly confidence decision values of each of the subset of the local units of the first and the second anomaly confidence decision values are outliers from and have the same distance to a center of a cluster of extracted features of a one of the learned motion pattern local models.

18. The computer program product of claim 17, wherein the computer readable program code instructions for execution by the computer processor, further cause the computer processor to rank the plurality of anomaly confidence decision values as a function of their respective assigned weightings by:

multiplying the normalized values of the extracted features of the anomaly confidence decision values of the subset of the local units by their respective learned weights to generate respective ranking values; and

ranking the plurality of anomaly confidence decision values by their generated respective ranking values.

19. The computer program product of claim 18, wherein the computer readable program code instructions for execution by the computer processor, further cause the computer processor to update the weighting of one of the first and the second anomaly confidence decision values by:

increasing or decreasing the updated weighting relative to the weighting of an other of the first and the second anomaly confidence decision values as a function of a difference in relative distances of the respective normalized extracted features of the subset local units of the first and the second anomaly confidence decision values from a center of a cluster of extracted features of a same associated learned motion pattern local model.