Scalable deep learning video analytics

• US 10,121,103 B2
• Filed: 12/09/2016
• Issued: 11/06/2018
• Est. Priority Date: 12/09/2016
• Status: Active Grant

• Current Assignee: Cisco Technology, Inc. (Cisco Systems, Inc.)
• Original Assignee: Cisco Technology, Inc. (Cisco Systems, Inc.)
• Inventors: Latapie, Hugo Mike, Fenoglio, Enzo, Friel, Joseph T., Surcouf, Andre, Thubert, Pascal
• Primary Examiner(s): STREGE, JOHN B
• Application Number:
  US15/374,571
• Publication Number:
  US 20180165576A1
• Time in Patent Office: 697 Days
• CPC Class Codes: G06F 18/213 Feature extraction, e.g. by transforming the feature space; Summarisation; Mappings, e.g. subspace methods
• CPC Class Codes: G06F 18/2414 Smoothing the distance, e.g. radial basis function networks [RBFN]
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/047 Probabilistic or stochastic networks
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06T 2207/10004 Still image; Photographic image
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/7715 Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69

Abstract

In one embodiment, a method includes receiving training data, the training data including training video data representing video of a location in a quiescent state, training a neural network using the training data to obtain a plurality of metrics, receiving current data, the current data including current video data representing video of the location at a current time period, generating a reconstruction error based on the plurality of metrics and the current video data in the embedded space, and generating, in response to determining that the reconstruction error is greater than a threshold, a notification indicative of the location being in a non-quiescent state.

Claims

1. A method comprising:

receiving training data, the training data including training video data representing video of a location in a quiescent state;

training a neural network using the training data to obtain a plurality of metrics;

receiving current data, the current data including current video data representing video of the location at a current time period;

generating a reconstruction error based on the plurality of metrics and the current video data; and

generating, in response to determining that the reconstruction error is greater than a threshold, a notification indicative of the location being in a non-quiescent state.

2. The method of claim 1, wherein training the neural network includes initializing the neural network based on a related plurality of metrics obtained by training a neural network using training data from a related location.

3. The method of claim 1, wherein the training data further includes training sensor data and the current data further includes current sensor data, wherein the reconstruction error is further based on the current sensor data.

4. The method of claim 3, wherein the current sensor data includes at least one of audio data, pollution data, WiFi data, or social media data.

5. The method of claim 1, further comprising reducing at least one of a frame rate or resolution of the current video data to generate dimensionality-reduced video data, wherein the reconstruction error is based on the dimensionality-reduced video data.

6. The method of claim 1, further comprising generating object tracking data based on the current video data, wherein the reconstruction error is based on the object tracking data.

7. The method of claim 6, wherein the object tracking data includes tracklet data representing a position of a respective one of a plurality of objects at a plurality of time instances.

8. The method of claim 1, further comprising setting the size of a buffer storing a recent portion of the current data based on the reconstruction error.

9. The method of claim 1, further comprising setting an amount of dimensionality reduction of the current data based on the reconstruction error.

10. The method of claim 1, further comprising selecting a computing resource to utilize based on the reconstruction error.

11. The method of claim 1, further comprising, performing, in response to determining that the reconstruction error is greater than a threshold, anomaly analytics upon the current data.

12. A system comprising:

one or more processors; and

a non-transitory memory comprising instructions that when executed cause the one or more processors to perform operations comprising;

receive training data, the training data including training video data representing video of a location in a quiescent state;

train a neural network using the training data to obtain a plurality of metrics;

receive current data, the current data including current video data representing video of the location at a current time period;

generate a reconstruction error based on the plurality of metrics and the current video data; and

generate, in response to determining that the reconstruction error is greater than a threshold, a notification indicative of the location being in a non-quiescent state.

13. The system of claim 12, wherein training data further includes training sensor data and the current data further includes current sensor data, wherein the reconstruction error is further based on the current sensor data.

14. The system of claim 12, wherein the operations further comprise at least one of setting the size of a buffer storing a recent portion of the current data based on the reconstruction error, setting an amount of dimensionality reduction of the current data based on the reconstruction error, or selecting a computing resource to utilize based on the reconstruction error.