Semantic representation module of a machine-learning engine in a video analysis system

• US 9,665,774 B2
• Filed: 10/28/2016
• Issued: 05/30/2017
• Est. Priority Date: 07/11/2007
• Status: Active Grant

• Current Assignee: Motorola Solutions, Inc.
• Original Assignee: Avigilon Patent Holding 1 Corporation
• Inventors: Xu, Gang, Seow, Ming-Jung, Risinger, Lon W., Solum, David M., Yang, Tao, Saitwal, Kishor Adinath, Eaton, John Eric, Gottumukkal, Rajkiran K., Cobb, Wesley Kenneth, Urech, Dennis G., Friedlander, David S.
• Primary Examiner(s): MARIAM, DANIEL G
• Application Number:
  US15/338,072
• Publication Number:
  US 20170046576A1
• Time in Patent Office: 214 Days
• Field of Search: 382106-107, 382156, 382159, 382181, 382224
• CPC Class Codes: G06F 16/285 Clustering or classification
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 3/006 based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO]
• CPC Class Codes: G06N 3/042 Knowledge-based neural networks; Logical representations of neural networks
• CPC Class Codes: G06N 3/044 Recurrent networks, e.g. Hopfield networks
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G06V 20/44 Event detection
• CPC Class Codes: G06V 20/47 Detecting features for summarising video content
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

A machine-learning engine is disclosed that is configured to recognize and learn behaviors, as well as to identify and distinguish between normal and abnormal behavior within a scene, by analyzing movements and/or activities (or absence of such) over time. The machine-learning engine may be configured to evaluate a sequence of primitive events and associated kinematic data generated for an object depicted in a sequence of video frames and a related vector representation. The vector representation is generated from a primitive event symbol stream and a phase space symbol stream, and the streams describe actions of the objects depicted in the sequence of video frames.

Claims

1. A system for processing data, the system comprising:

a processor; and

a memory communicatively coupled to the processor, the memory comprising computer-readable instructions that, when executed by the processor, cause the system to;

receive a scene depicted in a sequence of video frames from a camera;

identify one or more objects detected in the scene;

receive input data associated with the one or more identified objects;

evaluate the received input data to identify one or more primitive events, wherein for a first one of the primitive events, a semantic value is provided describing a behavior engaged in by a first one of the objects depicted in the sequence of video frames and wherein the first one of the primitive events has an assigned primitive event symbol;

generate, for the first object, a primitive event symbol stream which includes the primitive event symbol corresponding to the first one of the primitive events identified for the first object; and

output the primitive event symbol stream.

2. The system of claim 1, further comprising:

update, for the first object, the primitive event symbol stream as the first object moves about the scene.

3. The system of claim 2, wherein update the primitive event symbol stream further comprising:

identify, for the first object, a second one of the primitive events, wherein the second one of the primitive events has a second assigned primitive event symbol; and

add, for the first object, the second primitive event symbol to the primitive event symbol stream.

4. The system of claim 1, wherein the input data includes a classification for an object specifying that the object depicted in the sequence of video frames depicts at least one of:

a vehicle, a person, an unknown person, or an unknown object.

5. The system of claim 4, wherein the object is classified as the person or the unknown person, and wherein the input data further includes a posture of the person or the unknown person as depicted in the sequence of video frames.

6. The system of claim 4, wherein the object is classified as the vehicle, and wherein the behavior engaged in by the vehicle includes one or more of appearing, moving, turning and stopping.

7. The system of claim 4, wherein the object is classified as the person or the unknown person, and wherein the behavior engaged in by the person or the unknown person includes one or more of appearing, moving, turning and stopping.

8. The system of claim 4, wherein the object is classified as the unknown object, and wherein the behavior engaged in by the unknown object includes one or more of appearing, moving, turning and stopping.

9. The system of claim 1, further comprising:

form a first vector representation of the first object based on the primitive event symbol stream; and

analyze the first vector representations to identify a pattern of behavior for the first object from the first vector representation.

10. The system of claim 9, wherein analyze includes analyze the first vector representations with a machine learning engine configured to identify the pattern of behavior.

11. The system of claim 9, further comprising:

apply a singular value decomposition (SVD) to the first vector representation to generate a second vector representation from the first vector representation, wherein the second vector representation reduces the dimensionality of the first vector representation.

12. A method for processing data, the method comprising:

receiving a scene depicted in a sequence of video frames from a camera;

identifying one or more objects detected in the scene;

receiving input data associated with the one or more identified objects;

evaluating the received input data to identify one or more primitive events, wherein for a first one of the primitive events, a semantic value is provided describing a behavior engaged in by a first one of the objects depicted in the sequence of video frames and wherein the first one of the primitive events has an assigned primitive event symbol;

generating, for the first object, a primitive event symbol stream which includes the primitive event symbol corresponding to the first one of the primitive events identified for the first object; and

outputting the primitive event symbol stream.

13. The method of claim 12, further comprising:

updating, for the first object, the primitive event symbol stream as the first object moves about the scene.

14. The method of claim 13, wherein updating the primitive event symbol stream further comprising:

identifying, for the first object, a second one of the primitive events, wherein the second one of the primitive events has a second assigned primitive event symbol; and

adding, for the first object, the second primitive event symbol to the primitive event symbol stream.

15. The method of claim 12, wherein the input data includes a classification for an object specifying that the object depicted in the sequence of video frames depicts at least one of:

a vehicle, a person, an unknown person, or an unknown object.

16. The method of claim 15, wherein the object is classified as the person or the unknown person, and wherein the input data further includes a posture of the person or the unknown person as depicted in the sequence of video frames.

17. The method of claim 15, wherein the object is classified as the vehicle, and wherein the behavior engaged in by the vehicle includes one or more of appearing, moving, turning and stopping.

18. The method of claim 15, wherein the object is classified as the person, and wherein the behavior engaged in by the person includes one or more of appearing, moving, turning and stopping.

19. The method of claim 15, wherein the object is classified as an unknown object, and wherein the behavior engaged in by the unknown object includes one or more of appearing, moving, turning and stopping.

20. The method of claim 15, wherein the object is classified as the unknown person, and wherein the behavior engaged in by the unknown person includes one or more of appearing, moving, turning and stopping.