Systems and methods for a computer understanding of multi modal data streams

• US 9,563,843 B2
• Filed: 05/05/2016
• Issued: 02/07/2017
• Est. Priority Date: 05/10/2012
• Status: Active Grant

• Current Assignee: Yan M. Yufik
• Original Assignee: Yan M. Yufik
• Inventors: Yufik, Yan M.
• Primary Examiner(s): Chaki, Kakali
• Assistant Examiner(s): Seck, Ababacar
• Application Number:
  US15/147,004
• Publication Number:
  US 20160247066A1
• Time in Patent Office: 278 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 16/353 into predefined classes
• CPC Class Codes: G06F 18/2411 based on the proximity to a decision surface, e.g. support vector machines
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 3/04 Architecture, e.g. interconnection topology
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/047 Probabilistic or stochastic networks
• CPC Class Codes: G06N 3/063 using electronic means
• CPC Class Codes: G06N 3/08 Learning methods

Abstract

Systems and methods for understanding (imputing meaning to) multi modal data streams may be used in intelligent surveillance and allow a) real-time integration of streaming data from video, audio, infrared and other sensors; b) processing of the results of such integration to obtain understanding of the situation as it unfolds; c) assessing the level of threat inherent in the situation; and d) generating of warning advisories delivered to appropriate recipients as necessary for mitigating the threat. The system generates understanding of the system by creating and manipulating models of the situation as it unfolds. The creation and manipulation involve “neuronal packets” formed in mutually constraining associative networks of four basic types. The process is thermodynamically driven, striving to produce a minimal number of maximally stable models. Obtaining such models is experienced as grasping, or understanding the input stream (objects, their relations and the flow of changes).

Claims

1. A computer-implemented method in a self-adaptive multi modal data stream processing system having at least one computer processor, the computer processor including a control module that establishes a system of “

artificial neurons” and

associates data elements and various combinations of data elements with said neurons, a construction module under control of the control module that constructs components of situation models, and at least one spatiotemporal associative memory coupled to the at least one computer processor, the method comprising;

receiving multi modal data streams by the computer processor from multiple data stream sources, the multi modal data streams representing an environment of the multi modal data stream processing system;

constructing, by the construction module, at least one three-partite situation model of a situation, by making associations of artificial neurons of a plurality of artificial neurons of various types in an artificial neural network in the at least on spatiotemporal associative memory, wherein the three-partite situation model represents at least two entities and a relation between the at least two entities or at least two states of the same entity and a relation between the at least two states, wherein the step of constructing of the at least one three-partite situation model includes;

developing, by the control module, link-weighted associative artificial neural networks in the spatiotemporal associative memory, wherein the step of developing the link-weighted associative artificial neural networks includes;

corresponding, by the multi modal data stream processing system, individual nodes of the link-weighted associative artificial neural networks to respective artificial neurons of the plurality of artificial neurons that respond to different data elements of data representing the plurality of different data streams representing a situation; and

establishing link weights of the link-weighted associative artificial neural networks which represent a frequency of co-occurrence of the different data elements of the data representing the plurality of different data streams;

dynamically partitioning as the situation unfolds over time, by the control module, the link-weighted associative artificial neural networks into internally cohesive subnetworks and externally weakly coupled subnetworks and placing energy barriers at a subnetwork boundary of each of the subnetworks, with the barrier height computed as a function of the weights of the links inside the subnetwork and weights of the links connecting the subnetwork to its surrounds, wherein the subnetworks are neuronal packets, each corresponding to at least a respective one of various different combinations of the data elements;

performing dynamic mapping, by the control module, between the neuronal packets as the situation unfolds over time to adjust the at least one three-partite situation model to improve the at least one three-partite situation model for use in understanding of the situation, wherein the partitioning and dynamic mapping leave the artificial neural network intact by not changing synaptic weights in the artificial neural network in the partitioning and the dynamic mapping;

based on the at least one three-partite situation model, generating, by the multi modal data stream processing system, situational understanding of the situation;

reducing, by the multi modal data stream processing system, energy consumption and dissipation accompanying neuronal packet adjustments in the constructing, partitioning and dynamically mapping by the control module seeking progressively more general and adequate models persisting through various situations and wherein the reducing energy consumption and dissipation translates into negentropy production; and

based on a generated situational understanding of a situation, generating in real time by the multi modal data stream processing system appropriate output to facilitate one or more responses to the situation selected from the group consisting of an assessed threat level when objects or conditions in the situation constitute a threat when acting in coordination, identification of objects in an environment of a robotic vehicle or other robotic system, automatic detection and evaluation of malware in a computer network, and a disturbance in a reactor system;

if the situation is an assessed threat level, facilitating an automated intelligent surveillance of the situation;

if the situation is objects in an environment of a robotic vehicle or other robotic system, performing by the robotic vehicle or other robotic system adjusting pursuit of specified objectives and responding to obstacles; and

if the situation is the automatic detection and evaluation of malware in a computer network, dynamically deploying countermeasures against the malware over time,

if the situation is the disturbance in the reactor system, dynamically maintaining performance within user-defined safety or production limits for the reactor system,

wherein the plurality of artificial neurons of various different types includes a combination of;

sensory neurons, temporal neurons, feature neurons, spatial neurons, complex neurons, hyper complex neurons, and semantic neurons wherein the sensory neurons respond to different elements (features) in the incoming streams, the temporal neurons respond to various temporal relations in the activation of sensory neurons, the spatial neurons respond to different locations and relative positions of activation sources, the complex neurons respond to various activation patterns involving sensory, temporal and spatial neurons, the hyper complex neurons respond to various compositions of activation patterns involving complex neurons, and the semantic neurons respond to various patterns of activation involving hyper complex neurons and associate such patterns with labels in a finite set of labels defined by a user to signify meaningful relationships,

wherein the dynamic mapping includes manipulating packets by the control module, wherein the manipulating includes applying an operation of enfolding to packets comprising;

associating a neuronal pool with N-dimensional space of N dimensions (P-space), with each of the N dimensions corresponding to a particular data element type contained within a sensitivity range, which is a response vector, of one or more neurons in the neuronal pool;

in the packets, replacing a multitude of response vectors of constituent neurons by a single vector (PR-vector) computed as a function of constituent response vectors;

representing changes in packet composition and characteristics as movement of PR-vectors in P-space;

defining feature neurons by specifying points or regions in P-space residing within a sensitivity range of each feature neuron;

defining temporal neurons by specifying ordering relations in the movement of two or more PR-vectors;

defining spatial neurons by specifying configurations of points or regions in P-space subject to simultaneous traversal by two or more PR-vectors;

defining complex neurons by specifying coordinated movement of two or more PR-vectors in P-space;

defining hyper complex neurons by specifying coordinated movement of two or more PR-vectors produced by packets comprised of complex neurons;

defining semantic neurons by specifying coordinated movement of PR-vectors produced by nested packet structures comprised of hyper complex, complex and other types of neurons;

using distance between PR-vectors in P-space as a measure of packet discriminability;

representing external entities, which are sources of multi modal sensory streams received by the neuronal pool, as nested packet structures and associating behavior of such entities with the movement of PR-vectors associated with such structures;

defining a relationship between external entities A and B by specifying a form of coordination between the movement of corresponding PR-vectors;

defining two-partite situation models by specifying two external entities and a relationship between them;

defining three-partite situation models by specifying two entities A and B and specifying a third entity C such that a PR-vector associated with C moves between PR-vectors associated with A and B;

defining variable and invariant components of situation models by specifying varying and fixed components of constituent PR-vectors; and

deriving a likely future and past changes in the situation from trajectories of PR-vectors in P-space obtained by the control module via manipulating packet structures comprised in the situation model.

2. The method of claim 1 wherein the changes represented result from varying packet responses by the control module.

3. The method of claim 1 wherein defining feature neurons includes defining that a feature neuron fires when a particular point or region in P-space is traversed by a PR-vector of a packet comprised of sensory neurons.

4. The method of claim 1 wherein the defining temporal neurons includes defining that a temporal neuron fires when a particular region in P-space is traversed by a particular PR-vector followed by one or more other PR-vectors.

5. The method of claim 1 wherein the simultaneous traversal occurs within a certain time window.

6. The method of claim 1 wherein the defining spatial neurons includes defining that a spatial neuron fires when points two points or regions in P-space are simultaneously traversed by two respective PR-vectors.

7. The method of claim 1 wherein the defining complex neurons includes defining that a complex neuron X fires when PR-vector A and PR-vector B remain equidistant in P-space within some time interval ??, that complex neuron Y fires when PR-vector A revolves around PR-vector B, and that complex neuron Z fires when PR-vector A moves toward PR-vector B.

8. The method of claim 1 wherein the defining hyper complex neurons includes defining that hyper complex neuron X fires when firing of complex neuron Y is followed by firing of complex neuron Z such that Y responds to particular PR-vectors A and B moving towards each other while neuron Z responds to the particular PR-vectors moving away from each other).

9. The method of claim 1 wherein the movement of PR-vectors is movement of terminal points.

10. The method of claim 1 wherein the measure of packet discriminability is a probability of confusion.

11. The method of claim 10 wherein the probability of confusion includes is based on one or more scenarios in which PR-vector A is proximal to PR-vector B in which A and B are determined likely to be confused, or in which PR-vector C is distant from PR-vector B in which confusion is determined to be unlikely.

12. The method of claim 1 wherein the PR-vector associated with entity C moves by oscillating between the PR-vectors associated with the specified entities A and B.

13. The method of claim 1 wherein the varying and fixed components of the constituent PR-vectors are projections of each of the PR-vector onto one or more axes in the N-dimensional P-space.

14. The method of claim 1 wherein the deriving the likely future and past changes in the situation is prediction and retrodiction, respectively.

15. A computer implemented method in a self-adaptive multi modal data stream processing system having at least one computer processor and at least one spatiotemporal associative memory coupled to the at least one computer processor, the method comprising:

receiving multi modal data streams by the computer processor from multiple data stream sources, the multi modal data streams representing an environment of the multi modal data stream processing system;

constructing, by a construction module under control of a control module of the multi modal data stream processing system, a model of a situation built upon an underlying associative neural network that is partitioned into neuronal packets which are internally cohesive and externally weakly coupled subnetworks surrounded by energy barriers at a boundary of the subnetworks;

storing the underlying associative neural network in the associative memory to establish situational understanding of the situation;

associating neuronal packet groupings into stable (invariant) and changing (variable) entities and relationships between the entities;

assigning a relationship type to the components based on their content and behavior thereby creating a model of the situation, wherein each entity is able to be nested by the control module by being comprised of lower level models and wherein the lower level models are formed of neuronal packets and are groups of neuronal packets;

manipulating the lower level models by the control module of the multi modal data stream processing system, by manipulating neuronal packets while leaving the underlying associative neural network intact by not changing synaptic weights in the underlying associative neural network in the manipulation of the lower level models;

reducing, by the multi modal data stream processing system, energy consumption and energy dissipation in the constructing and the manipulating of the models by the control module seeking progressively more general and adequate models persisting through various situations and wherein the reducing energy consumption and dissipation translates into entropy reduction, or system negentropy production in the system;

based on a generated situational understanding of a situation, generating in real time by the multi modal data stream processing system appropriate output to facilitate one or more responses to the situation selected from the group consisting of an assessed threat level when objects or conditions in the situation constitute a threat when acting in coordination, identification of objects in an environment of a robotic vehicle or other robotic system, automatic detection and evaluation of malware in a computer network, and a disturbance in a reactor system;

if the situation is an assessed threat level, facilitating an automated intelligent surveillance of the situation;

if the situation is objects in an environment of a robotic vehicle or other robotic system, performing by the robotic vehicle or other robotic system adjusting pursuit of specified objectives and responding to obstacles; and

if the situation is the automatic detection and evaluation of malware in a computer network, dynamically deploying countermeasures against the malware over time,

if the situation is the disturbance in the reactor system, dynamically maintaining performance within user-defined safety or production limits for the reactor system,

wherein the plurality of artificial neurons of various different types includes a combination of;

sensory neurons, temporal neurons, feature neurons, spatial neurons, complex neurons, hyper complex neurons, and semantic neurons wherein the sensory neurons respond to different elements (features) in the incoming streams, the temporal neurons respond to various temporal relations in the activation of sensory neurons, the spatial neurons respond to different locations and relative positions of activation sources, the complex neurons respond to various activation patterns involving sensory, temporal and spatial neurons, the hyper complex neurons respond to various compositions of activation patterns involving complex neurons, and the semantic neurons respond to various patterns of activation involving hyper complex neurons and associate such patterns with labels in a finite set of labels defined by a user to signify meaningful relationships,

wherein the dynamic mapping includes manipulating packets by the control module, wherein the manipulating includes applying an operation of enfolding to packets comprising;

associating a neuronal pool with N-dimensional space of N dimensions (P-space), with each of the N dimensions corresponding to a particular data element type contained within a sensitivity range, which is a response vector, of one or more neurons in the neuronal pool;

in the packets, replacing a multitude of response vectors of constituent neurons by a single vector (PR-vector) computed as a function of constituent response vectors;

representing changes in packet composition and characteristics as movement of PR-vectors in P-space;

defining feature neurons by specifying points or regions in P-space residing within a sensitivity range of each feature neuron;

defining temporal neurons by specifying ordering relations in the movement of two or more PR-vectors;

defining spatial neurons by specifying configurations of points or regions in P-space subject to simultaneous traversal by two or more PR-vectors;

defining complex neurons by specifying coordinated movement of two or more PR-vectors in P-space;

defining hyper complex neurons by specifying coordinated movement of two or more PR-vectors produced by packets comprised of complex neurons;

defining semantic neurons by specifying coordinated movement of PR-vectors produced by nested packet structures comprised of hyper complex, complex and other types of neurons;

using distance between PR-vectors in P-space as a measure of packet discriminability;

representing external entities, which are sources of multi modal sensory streams received by the neuronal pool, as nested packet structures and associating behavior of such entities with the movement of PR-vectors associated with such structures;

defining a relationship between external entities A and B by specifying a form of coordination between the movement of corresponding PR-vectors;

defining two-partite situation models by specifying two external entities and a relationship between them;

defining three-partite situation models by specifying two entities A and B and specifying a third entity C such that a PR-vector associated with C moves between PR-vectors associated with A and B;

defining variable and invariant components of situation models by specifying varying and fixed components of constituent PR-vectors; and

deriving a likely future and past changes in the situation from trajectories of PR-vectors in P-space obtained by the control module via manipulating packet structures comprised in the situation model.