Efficient retrieval of anomalous events with priority learning

• US 9,158,976 B2
• Filed: 05/18/2011
• Issued: 10/13/2015
• Est. Priority Date: 05/18/2011
• Status: Expired due to Fees

• Current Assignee: Kyndryl Incorporated
• Original Assignee: International Business Machines Corporation
• Inventors: Datta, Ankur, Paluri, Balamanohar, Zhai, Yun, Pankanti, Sharathchandra U.
• Primary Examiner(s): Bella, Matthew
• Assistant Examiner(s): Rosario, Dennis
• Application Number:
  US13/110,331
• Publication Number:
  US 20120294511A1
• Time in Patent Office: 1,609 Days
• Field of Search: 382/190, 382/103
• US Class Current: 1/1
• CPC Class Codes: G06F 18/2433 Single-class perspective, e.g. one-against-all classification; Novelty detection; Outlier detection
• CPC Class Codes: G06T 7/251 involving models
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G08B 13/19606 Discriminating between target movement or movement in an area of interest and other non-signicative movements, e.g. target movements induced by camera shake or movements of pets, falling leaves, rotating fan
• CPC Class Codes: G16B 40/00 ICT specially adapted for biostatistics; ICT specially adapted for bioinformatics-related machine learning or data mining, e.g. knowledge discovery or pattern finding

Abstract

Local models learned from anomaly detection are used to rank detected anomalies. The local models include image feature values extracted from an image field of video image data with respect to different predefined spatial and temporal local units, wherein anomaly results are determined by failures to fit to applied anomaly detection module local models. Image features values extracted from the image field local units associated with anomaly results are normalized, and image feature values extracted from the image field local units are clustered. Weights for anomaly results are learned as a function of the relations of the normalized extracted image feature values to the clustered image feature values. The normalized values are multiplied by the learned weights to generate ranking values to rank the anomalies.

Claims

1. A method for using models learned from anomaly detection to rank detected anomalies, the method comprising:

tracking movement of an object that is detected within a video data input from a camera through an image field of the camera, wherein the image field is partitioned into a matrix comprising a grid of a plurality of different local units, and wherein the tracking generates a trajectory of the object'"'"'s motion that passes through a subset of the local units that is less than a totality of the plurality of the different local units;

extracting image features from the video data from the camera with respect to each of the subset of the local units of the plurality of local units by using said trajectory;

learning a plurality of local motion pattern models, one for each of the subset of the local units, wherein the plurality of the learned local motion pattern models comprise normal patterns that are defined by finding dominant distributions of the extracted image features within respective ones of the subset of the local units, and anomaly patterns that are defined by rare distributions of the extracted image features within the respective ones of the subset of the local units;

generating anomaly confidence decision values for the tracked object for each of said subset of the local units as a function of fitting the image features extracted for each of the subset of the local units from the video data input of the tracked object to the plurality of the learned local motion pattern models of the respective subset of the local units by determining whether the features extracted relevant to the object'"'"'s motion within the video data indicate that the object'"'"'s motion within each particular local unit is one of said normal patterns or anomaly patterns in view of the plurality of the learned local motion pattern models for the local units;

normalizing values of the image features that are extracted from the image field'"'"'s subset of the local units that are associated with each of the plurality of anomaly confidence decision values;

clustering the image feature values extracted from the image field'"'"'s subset of the local units that are associated with each of the plurality of anomaly confidence decision values;

learning weights for each of the anomaly confidence decision values as a function of a relation of their normalized values of the extracted image features to the clustered image feature values extracted from the each respective associated image field'"'"'s subset of the local units by

determining spatial locations of the clustered extracted image feature values of the subset local units of the anomaly confidence decision values within the field of view of the input video data as correlated to features of interest of a real-world scene represented within the field of view, and

assigning a first weighting to a first anomaly of the anomaly confidence decision values that is higher than a second weighting assigned to a second anomaly of the anomaly confidence decision values in response to

the determined spatial location of the clustered extracted image feature values of the subset local unit containing said first anomaly of the first anomaly confidence value being within a portion of the field of view of the input video that is correlated with a cordoned off area of the real-world scene and

the determined spatial location of the clustered extracted image feature values of the subset local unit containing said second anomaly of the second anomaly confidence decision value being outside the portion,

wherein the normalized extracted features of each of the subset local units of the first and the second anomaly confidence decision values are outliers from and have the same distance to a center of a cluster of extracted features of a one of the learned motion pattern local models;

multiplying the normalized values of the extracted features of the anomaly confidence decision values of the subset of the local units by their respective learned weights to generate respective ranking values; and

ranking the plurality of anomaly confidence decision values by their generated respective ranking values.

2. The method of claim 1, further comprising:

updating the weighting of one of the first and the second anomaly confidence decision values by increasing or decreasing the updated weighting relative to the weighting of an other of the first and the second anomaly confidence decision values as a function of a difference in relative distances of the respective normalized extracted features of the subset local units of the first and the second anomaly confidence decision values from a center of a cluster of extracted features of a same associated learned motion pattern local model.

3. The method of claim 1, further comprising:

updating the weighting of one of the first and the second anomaly confidence decision values by increasing or decreasing the updated weighting relative to the weighting of an other of the first and the second anomaly confidence decision values as a function of a difference in temporal data of the extracted data of the subset local units of the respective first and second anomaly confidence decision values.

4. The method of claim 3, wherein the difference in the temporal data is a longer duration time of the tracked object movement.

5. The method of claim 3, wherein the difference in the temporal data indicates a difference in a time of day of occurrences of the respective first and second anomaly confidence decision values.

6. The method of claim 3, further comprising:

extracting trajectory features from the video data input relative to the trajectory of the tracked object;

generating a global anomaly confidence decision value for the object trajectory as a function of fitting the extracted trajectory features to a normal learned motion trajectory model, wherein the global anomaly confidence decision value indicates a likelihood that the object trajectory is normal or anomalous; and

determining whether or not an anomaly has occurred as a function of the subset unit anomaly detection confidence decision values and the global anomaly detection confidence values.

7. A system, comprising:

a processor;

computer readable memory in circuit communication with the processor; and

a computer readable storage medium in circuit communication with the processor and the computer readable memory; and

wherein the processor executes program instructions stored on the computer-readable storage medium via the computer readable memory and thereby;

tracks movement of an object that is detected within a video data input from a camera through an image field of the camera,

wherein the image field is partitioned into a matrix comprising a grid of a plurality of different local units, and

wherein the tracking generates a trajectory of the object'"'"'s motion that passes through a subset of the local units that is less than a totality of the plurality of the different local units;

extracts image features from the video data from the camera with respect to each of the subset of the local units of the plurality of local units by using said trajectory;

learns a plurality of learned local motion pattern models, one for each of the subset of the local units,

wherein the plurality of learned local motion pattern models comprise

normal patterns that are defined by finding dominant distributions of the extracted image features within respective ones of the subset of the local units, and

anomaly patterns that are defined by rare distributions of the extracted image features within the respective ones of the subset of the local units;

generates anomaly confidence decision values for the tracked object for each of said subset of the local units as a function of fitting the image features extracted for each of the subset of the local units from the video data input of the tracked object to the plurality of the learned local motion pattern models of the respective subset of the local units, by

determining whether the features extracted relevant to the object'"'"'s motion within the video data indicate that the object'"'"'s motion within each particular local unit is one of said normal patterns or anomaly patterns in view of the plurality of the learned local motion pattern models for the local units;

normalizes values of the image features that are extracted from the image field'"'"'s subset of the local units that are associated with each of the plurality of anomaly confidence decision values;

clusters the image feature values extracted from the image field'"'"'s subset of the local units that are associated with each of the plurality of anomaly confidence decision values;

learns weights for each of the anomaly confidence decision values as a function of a relation of their normalized values of the extracted image features to the clustered image feature values extracted from the each respective associated image field'"'"'s subset of the local units by

determining spatial locations of the clustered extracted image feature values of the subset local units of the anomaly confidence decision values within the field of view of the input video data as correlated to features of interest of a real-world scene represented within the field of view, and

assigning a first weighting to a first anomaly of the anomaly confidence decision values that is higher than a second weighting assigned to a second anomaly of the anomaly confidence decision values in response to

the determined spatial location of the clustered extracted image feature values of the subset local unit containing said first anomaly of the first anomaly confidence decision value being within a portion of the field of view of the input video that is correlated with a cordoned off area of the real-world scene and

the determined spatial location of the clustered extracted image feature values of the subset local unit containing said second anomaly of the second anomaly confidence decision value being outside the portion,

wherein the normalized extracted features of each of the subset local units of the first and the second anomaly confidence decision values are outliers from and have the same distance to a center of a cluster of extracted features of a one of the learned motion pattern local models;

multiplies the normalized values of the extracted features of the anomaly confidence decision values of the subset of the local units by their respective learned weights to generate respective ranking values; and

ranks the plurality of anomaly confidence decision values by their generated respective ranking values.

8. The system of claim 7, wherein the processor executes the program instructions stored on the computer-readable storage medium via the computer readable memory and thereby updates the weighting of one of the first and the second anomaly confidence decision values by increasing or decreasing the updated weighting relative to the weighting of an other of the first and the second anomaly confidence decision values as a function of a difference in relative distances of the respective normalized extracted features of the subset of the local units of the first and the second anomaly confidence decision values from a center of a cluster of extracted features of a same associated learned motion pattern local model.

9. The system of claim 7, wherein the processor executes the program instructions stored on the computer-readable storage medium via the computer readable memory and thereby updates the weighting of one of the first and the second anomaly confidence decision values by increasing or decreasing the updated weighting relative to the weighting of an other of the first and the second anomaly confidence decision values as a function of a difference in temporal data of the extracted data of the subset of the local units of the respective first and second anomaly confidence decision values.

10. The system of claim 7, wherein the processor executes the program instructions stored on the computer-readable storage medium via the computer readable memory and thereby further:

extracts trajectory features from the video data input relative to the trajectory of the tracked object;

generates a global anomaly confidence decision value for the object trajectory as a function of fitting the extracted trajectory features to a normal learned motion trajectory model, wherein the global anomaly confidence decision value indicates a likelihood that the object trajectory is normal or anomalous; and

determines whether or not an anomaly has occurred as a function of the subset unit anomaly detection confidence decision values and the global anomaly detection confidence values.

11. A computer program product, comprising:

a computer readable hardware storage device having computer readable program code embodied therewith, the computer readable program code comprising instructions for execution by a computer processor that cause the computer processor to;

track movement of an object that is detected within a video data input from a camera through an image field of the camera, wherein the image field is partitioned into a matrix comprising a grid of a plurality of different local units, and wherein the tracking generates a trajectory of the object'"'"'s motion that passes through a subset of the local units that is less than a totality of the plurality of the different local units;

extract image features from the video data from the camera with respect to each of the subset of the local units of the plurality of local units by using said trajectory;

learn a plurality of learned local motion pattern models, one for each of the subset of the local units,

wherein the plurality of the learned local motion pattern models comprise normal patterns that are defined by finding dominant distributions of the extracted image features within respective ones of the subset of the local units, and anomaly patterns that are defined by rare distributions of the extracted image features within the respective ones of the subset of the local units;

generate anomaly confidence decision values for the tracked object for each of said subset of the local units as a function of fitting the image features extracted for each of the subset local units from the video data input of the tracked object to the plurality of the learned local motion pattern models of the respective subset of the local units by determining whether the features extracted relevant to the object'"'"'s motion within the video data indicate that the object'"'"'s motion within each particular local unit is one of said normal patterns or anomaly patterns in view of the plurality of the learned local motion pattern models for the local units;

normalize values of the image features that are extracted from the image field'"'"'s subset of the local units that are associated with each of the plurality of anomaly confidence decision values;

cluster the image feature values extracted from the image field'"'"'s subset of the local units that are associated with each of the plurality of anomaly confidence decision values;

learn weights for each of the anomaly confidence decision values as a function of a relation of their normalized values of the extracted image features to the clustered image feature values extracted from the each respective associated image field'"'"'s subset of the local units by

determining spatial locations of the clustered extracted image feature values of the subset local units of the anomaly confidence decision values within the field of view of the input video data as correlated to features of interest of a real-world scene represented within the field of view, and

assigning a first weighting to a first anomaly of the anomaly confidence decision values that is higher than a second weighting assigned to a second anomaly of the anomaly confidence decision values in response to

the determined spatial location of the clustered extracted image feature values of the subset local unit containing said first anomaly of the first anomaly confidence decision value being within a portion of the field of view of the input video that is correlated with a cordoned off area of the real-world scene and

the determined spatial location of the clustered extracted image feature values of the subset local unit containing said second anomaly of the second anomaly confidence decision value being outside the portion,

wherein the normalized extracted features of each of the subset local units of the first and the second anomaly confidence decision values are outliers from and have the same distance to a center of a cluster of extracted features of a one of the learned motion pattern local models;

multiply the normalized values of the extracted features of the anomaly confidence decision values of the subset of the local units by their respective learned weights to generate respective ranking values; and

rank the plurality of anomaly confidence decision values by their generated respective ranking values.

12. The computer program product of claim 11, wherein the computer readable program code instructions for execution by the computer processor, further cause the computer processor to update the weighting of one of the first and the second anomaly confidence decision values by increasing or decreasing the updated weighting relative to the weighting of an other of the first and the second anomaly confidence decision values as a function of a difference in relative distances of the respective normalized extracted features of the subset local units of the first and the second anomaly confidence decision values from a center of a cluster of extracted features of a same associated learned motion pattern local model.

13. The computer program product of claim 11, wherein the computer readable program code instructions for execution by the computer processor, further cause the computer processor to update the weighting of one of the first and the second anomaly confidence decision values by increasing or decreasing the updated weighting relative to the weighting of an other of the first and the second anomaly confidence decision values as a function of a difference in temporal data of the extracted data of the subset local units of the respective first and second anomaly confidence decision values.

14. The computer program product of claim 11, wherein the computer readable program code instructions for execution by the computer processor, further cause the computer processor to:

extract trajectory features from the video data input relative to the trajectory of the tracked object;

generate a global anomaly confidence decision value for the object trajectory as a function of fitting the extracted trajectory features to a normal learned motion trajectory model, wherein the global anomaly confidence decision value indicates a likelihood that the object trajectory is normal or anomalous; and

determine whether or not an anomaly has occurred as a function of the subset unit anomaly detection confidence decision values and the global anomaly detection confidence values.

15. A method for providing a service for using models learned from anomaly detection to rank detected anomalies, the method comprising:

providing a priority learning component that;

tracks movement of an object that is detected within a video data input from a camera through an image field of the camera,

wherein the image field is partitioned into a matrix comprising

a grid of a plurality of different local units, and

wherein the tracking generates a trajectory of the object'"'"'s motion that passes through a subset of the local units that is less than a totality of the plurality of the different local units;

extracts image features from the video data from the camera with respect to each of the subset of the local units of the plurality of local units by using said trajectory;

learns a plurality of learned local motion pattern models, one for each of the subset of the local units,

wherein the plurality of the learned local motion pattern models comprise

normal patterns that are defined by finding dominant distributions of the extracted image features within respective ones of the subset local units, and

anomaly patterns that are defined by rare distributions of the extracted image features within the respective ones of the subset of the local units;

generates anomaly confidence decision values for the tracked object for each of said subset of the local units as a function of fitting the image features extracted for each of the subset of the local units from the video data input of the tracked object to the plurality of the learned local motion pattern models of the respective subset local units by

determining whether the features extracted relevant to the object'"'"'s motion within the video data indicate that the object'"'"'s motion within each particular local unit is one of said normal patterns or anomaly patterns in view of the plurality of the learned local motion pattern models for the local units;

normalizes values of the image features that are extracted from the image field'"'"'s subset of the local units that are associated with each of the plurality of anomaly confidence decision values;

clusters the image feature values extracted from the image field'"'"'s subset of the local units that are associated with each of the plurality of anomaly confidence decision values; and

learns weights for each of the anomaly confidence decision values as a function of a relation of their normalized values of the extracted image features to the clustered image feature values extracted from the each respective associated image field'"'"'s subset of the local units by

determining spatial locations of the clustered extracted image feature values of the subset local units of the anomaly confidence decision values within the field of view of the input video data as correlated to features of interest of a real-world scene represented within the field of view, and

assigning a first weighting to a first anomaly of the anomaly confidence decision values that is higher than a second weighting assigned to a second anomaly of the anomaly confidence decision values in response to

the determined spatial location of the clustered extracted image feature values of the subset local unit containing said first anomaly of the first anomaly confidence decision value being within a portion of the field of view of the input video that is correlated with a cordoned off area of the real-world scene and

the determined spatial location of the clustered extracted image feature values of the subset local unit containing said second anomaly of the second anomaly confidence decision value being outside the portion,

wherein the normalized extracted features of each of the subset local units of the first and the second anomaly confidence decision values are outliers from and have the same distance to a center of a cluster of extracted features of a one of the learned motion pattern local models; and

providing a ranker that multiplies the normalized values of the extracted features of the anomaly confidence decision values of the subset of the local units by their respective learned weights to generate respective ranking values, and ranks the plurality of anomaly confidence decision values by their generated respective ranking values.

16. The method of claim 15, wherein the priority learning component updates the weighting of one of the first and the second anomaly confidence decision values by increasing or decreasing the updated weighting relative to the weighting of an other of the first and the second anomaly confidence decision values as a function of a difference in relative distances of the respective normalized extracted features of the subset local units of the first and the second anomaly confidence decision values from a center of a cluster of extracted features of a same associated learned motion pattern local model.

17. The method of claim 15, wherein the priority learning component updates the weighting of one of the first and the second anomaly confidence decision values by increasing or decreasing the updated weighting relative to the weighting of an other of the first and the second anomaly confidence decision values as a function of a difference in temporal data of the extracted data of the subset local units of the respective first and second anomaly confidence decision values.

18. The method of claim 15, further comprising:

providing a global feature extractor that extracts trajectory features from the video data input relative to the trajectory of the tracked object; and

providing a global anomaly detector that generates a global anomaly confidence decision value for the object trajectory as a function of fitting the extracted trajectory features to a normal learned motion trajectory model, wherein the global anomaly confidence decision value indicates a likelihood that the object trajectory is normal or anomalous, for determining whether or not an anomaly has occurred as a function of the subset unit anomaly detection confidence decision values and the global anomaly detection confidence values.