Semantic representation module of a machine-learning engine in a video analysis system

• US 8,411,935 B2
• Filed: 07/09/2008
• Issued: 04/02/2013
• Est. Priority Date: 07/11/2007
• Status: Expired due to Fees

• Current Assignee: Avigilon Patent Holding 1 Corporation
• Original Assignee: Behavioral Recognition Systems Incorporated
• Inventors: Eaton, John Eric, Cobb, Wesley Kenneth, Urech, Dennis G., Friedlander, David S., Xu, Gang, Seow, Ming-Jung, Risinger, Lon W., Solum, David M., Yang, Tao, Gottumukkal, Rajkiran K., Saitwal, Kishor Adinath
• Primary Examiner(s): MARIAM, DANIEL G
• Application Number:
  US12/170,268
• Publication Number:
  US 20090016599A1
• Time in Patent Office: 1,728 Days
• Field of Search: 382106-107, 382156-159, 382/197, 382/224, 382/226, 382/228
• US Class Current: 382/159
• CPC Class Codes: G06F 16/285 Clustering or classification
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 3/006 based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO]
• CPC Class Codes: G06N 3/042 Knowledge-based neural networks; Logical representations of neural networks
• CPC Class Codes: G06N 3/044 Recurrent networks, e.g. Hopfield networks
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G06V 20/44 Event detection
• CPC Class Codes: G06V 20/47 Detecting features for summarising video content
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

A machine-learning engine is disclosed that is configured to recognize and learn behaviors, as well as to identify and distinguish between normal and abnormal behavior within a scene, by analyzing movements and/or activities (or absence of such) over time. The machine-learning engine may be configured to evaluate a sequence of primitive events and associated kinematic data generated for an object depicted in a sequence of video frames and a related vector representation. The vector representation is generated from a primitive event symbol stream and a phase space symbol stream, and the streams describe actions of the objects depicted in the sequence of video frames.

Claims

1. A computer-implemented method for processing data describing a scene depicted in a sequence of video frames, the method comprising:

receiving input data describing one or more objects detected in the scene, wherein the input data includes at least a classification for each of the one or more objects;

identifying one or more primitive events, wherein each primitive event provides a semantic value describing a behavior engaged in by a corresponding one of the objects depicted in the sequence of video frames, and wherein each primitive event has an assigned primitive event symbol;

generating, for one or more of the objects, a primitive event symbol stream which includes the primitive event symbols corresponding to the primitive events identified for a respective object;

generating, for one or more of the objects, a phase space symbol stream, wherein the phase space symbol stream describes a trajectory for a respective object through a phase space domain;

combining the primitive event symbol stream and the phase space symbol stream for each respective object to form a first vector representation of that object; and

passing the first vector representations to a machine learning engine configured to identify patterns of behavior for each object classification from the first vector representation.

2. The computer-implemented method of claim 1, further comprising, applying a singular value decomposition (SVD) to the first vector representations to generate a second vector representations from the first vector representations, wherein the second vector representations reduce the dimensionality of the first vector representations.

3. The computer-implemented method of claim 1, wherein the classification for an object specifies that the object depicted in the sequence of video frames depicts one of a vehicle object, a person object, or an unknown object.

4. The computer-implemented method of claim 3, wherein the object is classified as a person, and wherein the input data further includes a posture of the person as depicted in the sequence of video frames.

5. The computer-implemented method of claim 1, wherein the phase space domain specifies a three-dimensional position of the object as depicted within the scene.

6. The computer-implemented method of claim 1, wherein input data further describes trajectories of one or more of the objects depicted within the scene.

7. The computer-implemented method of claim 1, wherein the input data includes a trajectory of one or more of the objects depicted within the scene and includes a velocity determined for one or more of the objects depicted within the scene, and wherein the phase space symbol stream is generated from the trajectories and velocities of the one or more objects.

8. A non-transitory computer-readable medium containing a program, which, when executed on a processor is configured to perform an operation for processing data describing a scene depicted in a sequence of video frames, comprising:

receiving input data describing one or more objects detected in the scene, wherein the input data includes at least a classification for each of the one or more objects;

identifying one or more primitive events, wherein each primitive event provides a semantic value describing a behavior engaged in by a corresponding one of the objects depicted in the sequence of video frames, and wherein each primitive event has an assigned primitive event symbol;

generating, for one or more of the objects, a primitive event symbol stream which includes the primitive event symbols corresponding to the primitive events identified for a respective object;

generating, for one or more of the objects, a phase space symbol stream, wherein the phase space symbol stream describes a trajectory for a respective object through a phase space domain;

combining the primitive event symbol stream and the phase space symbol stream for each respective object to form a first vector representation of that object; and

passing the first vector representations to a machine learning engine configured to identify patterns of behavior for each object classification from the first vector representation.

9. The non-transitory computer-readable medium of claim 8, wherein the operation further comprises, applying a singular value decomposition (SVD) to the first vector representations to generate a second vector representation from each first vector representation, wherein the second vector representations reduce the dimensionality of the corresponding first vector representation.

10. The non-transitory computer-readable medium of claim 9, wherein the classification for an object specifies that the object depicted in the sequence of video frames depicts one of a vehicle object, a person object, or an unknown object.

11. The non-transitory computer-readable medium of claim 10, wherein the object is classified as a person, and wherein the input data further includes a posture of the person as depicted in the sequence of video frames.

12. The non-transitory computer-readable medium of claim 9, wherein the phase space domain specifies a three-dimensional position of the object as depicted within the scene.

13. The non-transitory computer-readable medium of claim 9, wherein input data further describes trajectories of one or more of the objects depicted within the scene.

14. The non-transitory computer-readable medium of claim 9, wherein the input data includes a trajectory of one or more of the objects depicted within the scene and includes a velocity determined for one or more of the objects depicted within the scene, and wherein the phase space symbol stream is generated from the trajectories and velocities of the one or more objects.

15. A system, comprising:

a video input source;

a processor; and

a memory storing a machine learning engine, wherein the machine learning engine is configured to;

receive input data describing one or more objects detected in the scene, wherein the input data includes at least a classification for each of the one or more objects;

identifying one or more primitive events, wherein each primitive event provides a semantic value describing a behavior engaged in by a corresponding one of the objects depicted in the sequence of video frames, and wherein each primitive event has an assigned primitive event symbol;

generate, for one or more of the objects, a primitive event symbol stream which includes the primitive event symbols corresponding to the primitive events identified for a respective object;

generate, for one or more of the objects, a phase space symbol stream, wherein the phase space symbol stream describes a trajectory for a respective object through a phase space domain;

combine the primitive event symbol stream and the phase space symbol stream for each respective object to form a first vector representation of that object; and

pass the first vector representations to a machine learning engine configured to identify patterns of behavior for each object classification from the first vector representation.

16. The system of claim 15, wherein the machine learning engine is further configured to apply a singular value decomposition (SVD) to the first vector representations to generate a second vector representations from the first vector representations, wherein the second vector representations reduce the dimensionality of the first vector representations.

17. The system of claim 15, wherein the classification for an object specifies that the object depicted in the sequence of video frames depicts one of a vehicle object, a person object, or an unknown object.

18. The system of claim 17, wherein the object is classified as a person, and wherein the input data further includes a posture of the person as depicted in the sequence of video frames.

19. The system of claim 15, wherein the phase space domain specifies a three-dimensional position of the object as depicted within the scene.

20. The system of claim 15, wherein input data further describes trajectories of one or more of the objects depicted within the scene.

21. The system of claim 15, wherein the input data includes a trajectory of one or more of the objects depicted within the scene and includes a velocity determined for one or more of the objects depicted within the scene, and wherein the phase space symbol stream is generated from the trajectories and velocities of the one or more objects.

22. A computer-implemented method for processing data describing a scene depicted in a sequence of video frames, the method comprising:

receiving input data describing one or more objects detected in the scene, wherein the input data includes at least a classification for each of the one or more objects;

identifying one or more primitive events, wherein each primitive event provides a semantic value describing a behavior engaged in by a first one of the objects depicted in the sequence of video frames and wherein each primitive event has an assigned primitive event symbol;

generating, for the first object, a primitive event symbol stream which includes the primitive event symbols corresponding to the primitive events identified for the first object; and

generating, for the first object, a phase space symbol stream, wherein the phase space symbol stream describes a trajectory for the first object through a phase space domain; and

outputting the primitive event symbol stream and the phase space symbol stream.

23. The computer-implemented method of claim 22, wherein the classification for the first object specifies that the first object depicted in the sequence of video frames depicts one of a vehicle object, a person object, and an unknown object.

24. The computer-implemented method of claim 23, wherein the first object is classified as a person, and wherein the input data further includes a posture of the person as depicted in the sequence of video frames.

25. The computer-implemented method of claim 22, wherein the phase space domain specifies a position of the first object as depicted within the scene.