Semantic representation module of a machine learning engine in a video analysis system

• US 8,923,609 B2
• Filed: 04/02/2013
• Issued: 12/30/2014
• Est. Priority Date: 07/11/2007
• Status: Active Grant

• Current Assignee: Motorola Solutions, Inc.
• Original Assignee: Behavioral Recognition Systems Incorporated
• Inventors: Eaton, John Eric, Cobb, Wesley Kenneth, Urech, Dennis G., Friedlander, David S., Xu, Gang, Seow, Ming-Jung, Risinger, Lon W., Solum, David M., Yang, Tao, Gottumukkal, Rajkiran K., Saitwal, Kishor Adinath
• Primary Examiner(s): MARIAM, DANIEL G
• Application Number:
  US13/855,332
• Publication Number:
  US 20140072206A1
• Time in Patent Office: 637 Days
• Field of Search: None
• US Class Current: 382/159
• CPC Class Codes: G06F 16/285 Clustering or classification
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 3/006 based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO]
• CPC Class Codes: G06N 3/042 Knowledge-based neural networks; Logical representations of neural networks
• CPC Class Codes: G06N 3/044 Recurrent networks, e.g. Hopfield networks
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G06V 20/44 Event detection
• CPC Class Codes: G06V 20/47 Detecting features for summarising video content
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

A machine-learning engine is disclosed that is configured to recognize and learn behaviors, as well as to identify and distinguish between normal and abnormal behavior within a scene, by analyzing movements and/or activities (or absence of such) over time. The machine-learning engine may be configured to evaluate a sequence of primitive events and associated kinematic data generated for an object depicted in a sequence of video frames and a related vector representation. The vector representation is generated from a primitive event symbol stream and a phase space symbol stream, and the streams describe actions of the objects depicted in the sequence of video frames.

Claims

1. A computer-implemented method for processing video image data, the method comprising:

detecting a plurality of objects in the video image data; and

for each object;

generating, via a processor, a primitive event symbol stream identifying one or more primitive events engaged in by the object, wherein each primitive event represents a behavior engaged in by the object,

generating, via the processor, a phase-space symbol stream representing quantitative characteristics of the object, wherein the phase-space symbol stream for the object indicates a trajectory of that object in the video image data over time, and

combining, via the processor, the primitive event symbol stream and the phase-space symbol stream to generate a vector representation of the behavior engaged in by the object as depicted in the video image data.

2. The method of claim 1, further comprising, identifying, by a machine learning engine, at least one pattern of behavior for one or more of the objects based on the vector representations of the plurality of objects.

3. The method of claim 2, wherein the pattern of behavior is identified by clustering the vector representations of the plurality of objects.

4. The method of claim 3, wherein each clustered vector representation represents a variation of a pattern.

5. The method of claim 1, wherein identifying the primitive events comprises:

analyzing kinematic data and posture data associated with one of the objects; and

generating the primitive event symbol stream based on the analyzed data.

6. The method of claim 1, wherein the primitive events are described using a formal language grammar.

7. A non-transitory computer-readable storage medium containing a program, which, when executed on a processor is configured to perform an operation for processing video image data, the operation comprising:

detecting a plurality of objects in the video image data; and

for each object;

generating a primitive event symbol stream identifying one or more primitive events engaged in by the object, wherein each primitive event represents a behavior engaged in by the object,

generating a phase-space symbol stream representing quantitative characteristics of the object, wherein the phase-space symbol stream for the object indicates a trajectory of that object in the video image data over time, and

combining the primitive event symbol stream and the phase-space symbol stream to generate a vector representation of the behavior engaged in by the object as depicted in the video image data.

8. The computer-readable storage medium of claim 7, wherein the operation further comprises, identifying, by a machine learning engine, at least one pattern of behavior for one or more of the objects based on the vector representations of the plurality of objects.

9. The computer-readable storage medium of claim 8, wherein the pattern of behavior is identified by clustering the vector representations of the plurality of objects.

10. The computer-readable storage medium of claim 9, wherein each clustered vector representation represents a variation of a pattern.

11. The computer-readable storage medium of claim 7, wherein identifying the primitive events comprises:

analyzing kinematic data and posture data associated with one of the objects; and

generating the primitive event symbol stream based on the analyzed data.

12. The computer-readable storage medium of claim 7, wherein the primitive events are described using a formal language grammar.

13. A system, comprising:

a video input source;

a processor; and

a memory storing a machine learning engine, wherein the machine learning engine is configured to perform an operation for processing video image data, the operation comprising;

detecting a plurality of objects in the video image data; and

for each object;

generating a primitive event symbol stream identifying one or more primitive events engaged in by the object, wherein each primitive event represents a behavior engaged in by the object,

generating a phase-space symbol stream representing quantitative characteristics of the object, wherein the phase-space symbol stream for the object indicates a trajectory of that object in the video image data over time, and

combining the primitive event symbol stream and the phase-space symbol stream to generate a vector representation of the behavior engaged in by the object as depicted in the video image data.

14. The system of claim 13, wherein the operation further comprises, identifying, by a machine learning engine, at least one pattern of behavior for one or more of the objects based on the vector representations of the plurality of objects.

15. The system of claim 14, wherein the pattern of behavior is identified by clustering the vector representations of the plurality of objects.

16. The system of claim 15, wherein each clustered vector representation represents a variation of a pattern.

17. The system of claim 13, wherein identifying the primitive events comprises:

analyzing kinematic data and posture data associated with one of the objects; and

generating the primitive event symbol stream based on the analyzed data.

18. The system of claim 13, wherein the primitive events are described using a formal language grammar.