Semantic representation module of a machine-learning engine in a video analysis system

• US 10,423,835 B2
• Filed: 12/19/2018
• Issued: 09/24/2019
• Est. Priority Date: 07/11/2007
• Status: Active Grant

• Current Assignee: Motorola Solutions, Inc.
• Original Assignee: Avigilon Patent Holding 1 Corporation
• Inventors: Eaton, John Eric, Cobb, Wesley Kenneth, Urech, Dennis G., Friedlander, David S., Xu, Gang, Seow, Ming-Jung, Risinger, Lon W., Solum, David M., Yang, Tao, Gottumukkal, Rajkiran K., Saitwal, Kishor Adinath
• Primary Examiner(s): Mariam, Daniel G
• Application Number:
  US16/226,496
• Publication Number:
  US 20190122048A1
• Time in Patent Office: 279 Days
• Field of Search: 382103, 382106-107, 382156, 382159, 382181, 382224
• CPC Class Codes: G06F 16/285 Clustering or classification
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 3/006 based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO]
• CPC Class Codes: G06N 3/042 Knowledge-based neural networks; Logical representations of neural networks
• CPC Class Codes: G06N 3/044 Recurrent networks, e.g. Hopfield networks
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G06V 20/44 Event detection
• CPC Class Codes: G06V 20/47 Detecting features for summarising video content
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

A machine-learning engine is disclosed that is configured to recognize and learn behaviors, as well as to identify and distinguish between normal and abnormal behavior within a scene, by analyzing movements and/or activities (or absence of such) over time. The machine-learning engine may be configured to evaluate a sequence of primitive events and associated kinematic data generated for an object depicted in a sequence of video frames and a related vector representation. The vector representation is generated from a primitive event symbol stream and a phase space symbol stream, and the streams describe actions of the objects depicted in the sequence of video frames.

Claims

1. A computer-implemented method for processing data related to a scene depicted in a sequence of video frames, the method comprising:

receiving input data describing an object detected in the scene, wherein the input data comprises at least a classification for the object;

identifying one or more primitive events, related to a behavior engaged in by the object detected in the scene and wherein each primitive event has an assigned primitive event symbol;

generating, for the object, a primitive event symbol stream comprising the primitive event symbols corresponding to the primitive events identified for the object;

forming a first vector representing the object; and

comparing the first vector to a stored vector to define a similarity measure.

2. The computer-implemented method of claim 1, further comprising, applying a singular value decomposition (SVD) to the first vector to generate a second vector from the first vector, wherein the second vector reduces the dimensionality of the first vector.

3. The computer-implemented method of claim 1, wherein the classification for the object specifies that the object depicts one of a vehicle object, a person object, or an unknown object.

4. The computer-implemented method of claim 3, wherein the object is classified as a person, and wherein the input data further includes a posture of the person as depicted in the scene.

5. The computer-implemented method of claim 3, wherein the object is classified as a vehicle, and wherein the input data further comprises behavior engaged in by the vehicle that includes one or more of appearing, moving, turning and stopping.

6. The computer-implemented method of claim 3, wherein the object is classified as a person, and wherein the input data further comprises behavior engaged in by the person that includes one or more of appearing, moving, turning and stopping.

7. The computer-implemented method of claim 3, wherein the object is classified as an unknown object, and wherein the input data further comprises behavior engaged in by the unknown object that includes one or more of appearing, moving, turning and stopping.

8. The computer-implemented method of claim 1, wherein comparing includes comparing the first vector to the stored vector with a machine learning engine.

9. A non-transitory computer-readable medium containing a program, which, when executed on a processor is configured to perform an operation for processing data related to a scene depicted in a sequence of video frames, comprising:

receiving input data describing an object detected in the scene, wherein the input data comprises at least a classification for the object;

identifying one or more primitive events, related to a behavior engaged in by the object detected in the scene and wherein each primitive event has an assigned primitive event symbol;

generating, for the object, a primitive event symbol stream which includes the primitive event symbols corresponding to the primitive events identified for the object;

forming a first vector representing the object; and

comparing the first vector to a stored vector to define a similarity measure.

10. The non-transitory computer-readable medium of claim 9, further comprising, applying a singular value decomposition (SVD) to the first vector to generate a second vector from the first vector, wherein the second vector reduces the dimensionality of the first vector.

11. The non-transitory computer-readable medium of claim 9, wherein the classification for the object specifies that the object depicted in the scene depicts one of a vehicle object, a person object, or an unknown object.

12. The non-transitory computer-readable medium of claim 11, wherein the object is classified as a person, and wherein the input data further includes a posture of the person as depicted in the scene.

13. The computer-implemented method of claim 9, wherein comparing includes comparing the first vector representation with a machine learning engine.

14. A system, comprising:

a video input source;

a processor; and

a memory storing computer instructions, which, when executed on the processor configure the processor to;

input data describing a scene depicted in a sequence of video frames, the input data comprises at least a classification for an object in a scene;

identify one or more primitive events, wherein each primitive event provides a semantic value describing a behavior engaged in by the object depicted in the scene and wherein each primitive event has an assigned primitive event symbol;

generate, for the object, a primitive event symbol stream which comprises the primitive event symbols corresponding to the primitive events identified for the object;

form a first vector representing the object; and

compare the first vector to a stored vector to define a similarity measure.

15. The system of claim 14, wherein the processor is further configured to apply a singular value decomposition (SVD) to the first vector to generate a second vector from the first vector, wherein the second vector reduces the dimensionality of the first vector representation.

16. The system of claim 14, wherein the classification for the object specifies that the object depicted in the scene depicts one of a vehicle object, a person object, or an unknown object.

17. The system of claim 16, wherein the object is classified as a person, and wherein the input data further includes a posture of the person as depicted in the scene.

18. The system of claim 16, wherein the object is classified as a vehicle, and wherein the behavior engaged in by the vehicle includes one or more of appearing, moving, turning and stopping.

19. The system of claim 16, wherein the object is classified as a person, and wherein the behavior engaged in by the person includes one or more of appearing, moving, turning and stopping.

20. The system of claim 14, wherein a machine learning engine is employed to compare the first vector to the stored vector.