Systems and methods for a computer understanding multi modal data streams

• US 9,378,455 B2
• Filed: 02/07/2013
• Issued: 06/28/2016
• Est. Priority Date: 05/10/2012
• Status: Active Grant

• Current Assignee: Yan M. Yufik
• Original Assignee: Yan M. Yufik
• Inventors: Yufik, Yan M.
• Primary Examiner(s): Chaki, Kakali
• Assistant Examiner(s): Seck, Ababacar
• Application Number:
  US13/762,230
• Publication Number:
  US 20130304684A1
• Time in Patent Office: 1,237 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 16/353 into predefined classes
• CPC Class Codes: G06F 18/2411 based on the proximity to a decision surface, e.g. support vector machines
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 3/04 Architecture, e.g. interconnection topology
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/047 Probabilistic or stochastic networks
• CPC Class Codes: G06N 3/063 using electronic means
• CPC Class Codes: G06N 3/08 Learning methods

Abstract

Systems and methods for understanding (imputing meaning to) multi modal data streams may be used in intelligent surveillance and allow a) real-time integration of streaming data from video, audio, infrared and other sensors; b) processing of the results of such integration to obtain understanding of the situation as it unfolds; c) assessing the level of threat inherent in the situation; and d) generating of warning advisories delivered to appropriate recipients as necessary for mitigating the threat. The system generates understanding of the system by creating and manipulating models of the situation as it unfolds. The creation and manipulation involve “neuronal packets” formed in mutually constraining associative networks of four basic types. The process is thermodynamically driven, striving to produce a minimal number of maximally stable models. Obtaining such models is experienced as grasping, or understanding the input stream (objects, their relations and the flow of changes).

Claims

1. A computer implemented method in a self-adaptive multi modal data stream processing system having at least one computer processor and at least one spatiotemporal associative memory coupled to the at least one computer processor, the method comprising:

receiving multi modal data streams by the computer processor from multiple data stream sources, the multi modal data streams representing an environment of the multi modal data stream processing system;

constructing, by a construction module under control of a control module of the multi modal data stream processing system, a model of a situation built upon an underlying associative neural network that is partitioned into neuronal packets which are internally cohesive and externally weakly coupled subnetworks surrounded by energy barriers at a boundary of the subnetworks;

storing the underlying associative neural network in the associative memory to establish situational understanding of the situation;

associating neuronal packet groupings into stable (invariant) and changing (variable) entities and relationships between the entities;

assigning a relationship type to the components based on their content and behavior thereby creating a model of the situation, wherein each entity is able to be nested by the control module by being comprised of lower level models and wherein the lower level models are formed of neuronal packets and are groups of neuronal packets;

manipulating the lower level models by the control module of the multi modal data stream processing system, by manipulating neuronal packets while leaving the underlying associative neural network intact by not changing synaptic weights in the underlying associative neural network in the manipulation of the lower level models;

reducing, by the multi modal data stream processing system, energy consumption and energy dissipation in the constructing and the manipulating of the models by the control module seeking progressively more general and adequate models persisting through various situations and wherein the reducing energy consumption and dissipation translates into entropy reduction, or system negentropy production in the system;

based on a generated situational understanding of a situation, generating in real time by the multi modal data stream processing system appropriate output to facilitate one or more responses to the situation selected from the group consisting of an assessed threat level when objects or conditions in the situation constitute a threat when acting in coordination, identification of objects in an environment of a robotic vehicle or other robotic system, and automatic detection and evaluation of malware in a computer network;

if the situation is an assessed threat level, facilitating an automated intelligent surveillance of the situation;

if the situation is objects in an environment of a robotic vehicle or other robotic system, performing by the robotic vehicle or other robotic system adjusting pursuit of specified objectives and responding to obstacles; and

if the situation is the automatic detection and evaluation of malware in a computer network, dynamically deploying countermeasures against the malware over time.

2. The method of claim 1 wherein the step of manipulating the lower level models includes:

grouping the neuronal packets;

adjusting the neuronal packets including adding or removing artificial neurons of the underlying associative neural network in the associative memory; and

varying neuronal packet responses of the packets including rotating a vector of neuronal responses inside the neuronal packets assigning a relationship type based on the form of coordination in the rotation of neuronal packet vectors.

3. The method of claim 1 wherein activation patterns associated with complex, hyper complex and semantic neurons are initially defined based on input from a system user, and further comprising:

when stable neuronal packets of sensory neurons emerge within the system, generating, by the system, new complex neurons responding to various patterns of neuronal packet activation different from the patterns defined by the user; and

generating, by the system, hyper complex neurons responding to activation patterns involving neuronal packets of complex neurons.

4. A computer-implemented method in a self-adaptive multi modal data stream processing system having at least one computer processor, the computer processor including a control module that establishes a system of “

artificial neurons” and

associates data elements and various combinations of data elements with said neurons, a construction module under control of the control module that constructs components of situation models, and at least one spatiotemporal associative memory coupled to the at least one computer processor, the method comprising;

receiving multi modal data streams by the computer processor from multiple data stream sources, the multi modal data streams representing an environment of the multi modal data stream processing system;

constructing, by the construction module, at least one three-partite situation model of a situation represented by a plurality of different ones of the multi modal data streams, by making associations of artificial neurons of a plurality of artificial neurons of various different types in an artificial neural network in the spatiotemporal associative memory, wherein the step of constructing of the at least one three-partite situation model includes;

developing, by a the control module link-weighted associative artificial neural networks in the spatiotemporal associative memory, wherein the step of developing the link-weighted associative artificial neural networks includes;

corresponding, by the multi modal data stream processing system, individual nodes of the link-weighted associative artificial neural networks to respective artificial neurons of the plurality of artificial neurons that respond to different data elements of data representing the plurality of different data streams representing a situation; and

establishing link weights of the link-weighted associative artificial neural networks which represent a frequency of co-occurrence of the different data elements of the data representing the plurality of different data streams;

dynamically partitioning as the situation unfolds over time, by the control module, the link-weighted associative artificial neural networks into internally cohesive subnetworks and externally weakly coupled subnetworks and placing energy barriers at a subnetwork boundary of each of the subnetworks, with the barrier height computed as a function of the weights of the links inside the subnetwork and weights of the links connecting the subnetwork to its surrounds, wherein the subnetworks are neuronal packets, each corresponding to at least a respective one of various different combinations of the data elements;

performing dynamic mapping, by the control module between the neuronal packets as the situation unfolds over time to adjust the at least one three-partite situation model to improve the at least one three-partite situation model for use in understanding of the situation, wherein the partitioning and dynamic mapping leave the artificial neural network intact by not changing synaptic weights in the artificial neural network in the partitioning and the dynamic mapping;

based on the at least one three-partite situation model, generating, by the multi modal data stream processing system, situational understanding of the situation;

reducing, by the multi modal data stream processing system, energy consumption and dissipation accompanying neuronal packet adjustments in the constructing, partitioning and dynamically mapping by the control module seeking progressively more general and adequate models persisting through various situations and wherein the reducing energy consumption and dissipation translates into negentropy production; and

based on a generated situational understanding of a situation, generating in real time by the multi modal data stream processing system appropriate output to facilitate one or more responses to the situation selected from the group consisting of an assessed threat level when objects or conditions in the situation constitute a threat when acting in coordination, identification of objects in an environment of a robotic vehicle or other robotic system, and automatic detection and evaluation of malware in a computer network;

if the situation is an assessed threat level, facilitating an automated intelligent surveillance of the situation;

if the situation is objects in an environment of a robotic vehicle or other robotic system, performing by the robotic vehicle or other robotic system adjusting pursuit of specified objectives and responding to obstacles; and

if the situation is the automatic detection and evaluation of malware in a computer network, dynamically deploying countermeasures against the malware over time.

5. The method of claim 4 wherein the step of performing dynamic mapping between the neuronal packets as the situation unfolds over time includes:

associating, by the multi modal data stream processing system, the neuronal packets into distinct neuronal packet groupings representing situational entities of the situation;

associating, by the multi modal data stream processing system, the neuronal packet groupings into invariant situational components of the situation;

associating, by the multi modal data stream processing system, neuronal packet groupings into variable situational components of the situation;

assigning, by the multi modal data stream processing system, a relationship type to each respective situational component of the invariant situational components and to each respective situational component of the variable situational components based on contents of and behavior of the respective situational component to construct the at least one three-partite situation model;

manipulating the at least one three-partite situation model;

comparing results of the manipulation with relevant data of the integrated data stream to facilitate determining validity of the at least one three-partite situation model; and

adjusting the at least one three-partite situation model based on the comparing.

6. The method of claim 4, further comprising:

receiving, by the multi modal data stream processing system, the plurality of different data streams, each of the different data streams originating from different sensors; and

in response to the receiving the plurality of different data streams, automatically integrating, by the multi modal data stream processing system, the plurality of data streams in real time to produce an integrated data stream; and

based on the generated situational understanding, the multi modal data stream processing system generating warning advisories for delivery to relevant selected recipients to mitigate a threat posed by the situation.

7. The method of claim 6 wherein the step of constructing the at least one three-partite situation model of a situation represented by the plurality of different data streams occurs in response to the integrating the plurality of data streams in real time to produce an integrated data stream.

8. The method of claim 4 wherein the step of performing the dynamic mapping between the neuronal packets as the situation unfolds over time includes automatically assessing, by the multi modal data stream processing system, a level of threat existing in the situation as the situation unfolds over time.

9. The method of claim 8 wherein the step of assessing the level of a threat existing in the situation as the situation unfolds over time includes:

predicting, by the multi modal data stream processing system, changes in the situation using the at least one three-partite situation model; and

determining, by the multi modal data stream processing system, adequate responses.

10. The method of claim 4 wherein at least some of the different data elements represent pixels.

11. The method of claim 4 wherein the frequency of the co-occurrence of the different data elements of the data representing the plurality of different data streams is correlated to an associative strength between the artificial neurons of the plurality of artificial neurons of various different types.

12. The method of claim 4 wherein the plurality of artificial neurons of various different types includes a combination of:

sensory neurons, temporal neurons, feature neurons, spatial neurons, complex neurons, hyper complex neurons, and semantic neurons wherein the sensory neurons respond to different elements (features) in the incoming streams, the temporal neurons respond to various temporal relations in the activation of sensory neurons, the spatial neurons respond to different locations and relative positions of activation sources, the complex neurons respond to various activation patterns involving sensory, temporal and spatial neurons, the hyper complex neurons respond to various compositions of activation patterns involving complex neurons, and the semantic neurons respond to various patterns of activation involving hyper complex neurons and associate such patterns with labels in a finite set of labels defined by a user to signify meaningful relationships.

13. The method of claim 4 wherein the link weights of the link-weighted associative artificial neural networks reflect co-occurrence of state changes in the plurality of artificial neurons of various different types.

14. The method of claim 4 wherein the step of performing dynamic mapping between the neuronal packets as the situation unfolds over time includes computing, by the multi modal data stream processing system, response strength of each respective artificial neuron of at least some of the artificial neurons of the plurality of artificial neurons as a function of activation probabilities of artificial neurons connected to the respective artificial neuron.

15. The method of claim 4 wherein the step of performing the dynamic mapping between the neuronal packets as the situation unfolds over time includes inserting or removing links or nodes in the link-weighted associative artificial neural networks.

16. The method of claim 4 wherein the step of perfoiniing the dynamic mapping between the neuronal packets as the situation unfolds over time includes expanding or shrinking the neuronal packets.

17. The method of claim 4 wherein the step of performing the dynamic mapping between the neuronal packets as the situation unfolds over time causes improvement in predictive or retrodictive performance of the three-partite situation model for use in threat level assessment of the situation.

18. The computer implemented method of claim 4, wherein the established situational understanding of the situation is an assessed threat level, further comprising the step of facilitating an automated intelligent surveillance of the situation.

19. The computer implemented method of claim 18, wherein the assessed threat level includes identification of objects or conditions in the situation that constitute the threat when acting in coordination.

20. The computer implemented method of claim 19, wherein the assessed threat level facilitates a robotic vehicle or other robotic system automatically responding to an environment of the robotic vehicle or other robotic system to perform one or more of pursuing specified objectives and responding to obstacles.

21. The computer implemented method of claim 18, further comprising the step of generating warning advisories for delivery to relevant selected recipients to mitigate a threat posed by the situation based on the assessed threat level.

22. The computer implemented method of claim 18, further comprising the steps of:

facilitating timely automatic detection and evaluation of malware in a computer network, wherein various features of the computer network are represented by link-weighted associative artificial neural networks; and

dynamically improving deployment of countermeasures against the malware over time.

23. The computer implemented method of claim 4, wherein the multi modal data streams are generated by sensors connected to the computer processor, which sensors generate data elements corresponding to video, audio, spatial, and temporal features corresponding to a situation.

24. A multi modal data stream processing system, comprising:

a plurality of multi modal data stream sources producing multi modal data streams representing an environment of the multi modal data stream processing system;

at least one computer processor receiving the multi modal data streams, the computer processor including a control module that establishes a system of artificial neurons and associates data elements and various combinations of data elements with said neurons and a construction module, under control of the control module, that constructs components of situation models;

at least one non-transitory spatiotemporal associative memory coupled to the at least one computer processor; and

at least one non-transitory memory communicatively coupled to the computer processor having computer-executable instructions stored thereon that, when executed by the computer processor, cause the computer processor to;

dynamically partition, as a situation represented by a plurality of different data streams unfolds over time, link-weighted associative artificial neural networks into internally cohesive subnetworks and externally weakly coupled subnetworks, wherein the subnetworks are neuronal packets, each corresponding to at least a respective one of various different combinations of data elements of data representing the plurality of different multi modal data streams;

perform dynamic mapping between the neuronal packets as the situation unfolds over time to adjust at least one three-partite situation model to improve the at least one three-partite situation model for use in understanding of the situation by the system, wherein the partitioning and dynamic mapping leave the artificial neural network intact by not changing synaptic weights in the artificial neural network in the partitioning and the dynamic mapping;

generate situational understanding of the situation based on the at least one three-partite situation model;

reduce energy consumption and dissipation in the partitioning and the dynamically mapping by seeking progressively more general and adequate models persisting through various situations wherein the reducing energy consumption and dissipation during neuronal packet adjustments translates into negentropy production; and

based on a generated situational understanding of a situation, generating in real time by the multi modal data stream processing system appropriate output to facilitate one or more responses to the situation selected from the group consisting of an assessed threat level when objects or conditions in the situation constitute a threat when acting in coordination, identification of objects in an environment of a robotic vehicle or other robotic system, and automatic detection and evaluation of malware in a computer network;

if the situation is an assessed threat level, facilitating an automated intelligent surveillance of the situation;

if the situation is objects in an environment of a robotic vehicle or other robotic system, performing by the robotic vehicle or other robotic system adjusting pursuit of specified objectives and responding to obstacles; and

if the situation is the automatic detection and evaluation of malware in a computer network, dynamically deploying countermeasures against the malware over time.

25. The multi modal data stream processing system of claim 24 wherein the computer-executable instructions that cause the computer processor to automatically assess the level of threat existing in the situation as the situation unfolds over time based on the at least one three-partite situation model, include computer-executable instructions that, when executed, cause the computer processor to:

based on the generated situational understanding, automatically assess a level of a threat existing in the situation as the situation unfolds over time based on the at least one three-partite situation model;

based on the assessed level of threat, generate warning advisories for delivery to relevant selected recipients to mitigate a threat posed by the situation;

facilitate timely automatic detection and evaluation of malware in a computer network, wherein various features of the computer network are represented by the link-weighted associative artificial neural networks; and

dynamically improve deployment of countermeasures against the malware over time.

26. The multi modal data stream processing system of claim 25 wherein the computer-executable instructions that cause the computer processor to facilitate timely automatic detection and evaluation of malware, include computer-executable instructions that, when executed, cause the computer processor to identify correlations between elements of the malware, geospatial locations within the computer network and temporal organization of attack patterns of the malware revealed by the dynamic partition of the link-weighted associative artificial neural networks into internally cohesive subnetworks and externally weakly coupled subnetworks.

27. The multi modal data stream processing system of claim 24, wherein the established situational understanding of the situation is an assessed threat level, wherein the computer executable instructions further cause the computer processor to facilitate an automated intelligent surveillance of the situation.

28. The multi modal data stream processing system of claim 27, wherein the assessed threat level includes identification of objects or conditions in the situation that constitute the threat when acting in coordination.

29. The multi modal data stream processing system of claim 27, wherein the assessed threat level facilitates a robotic vehicle or other robotic system automatically responding to an environment of the robotic vehicle or other robotic system to perform one or more of pursuing specified objectives and responding to obstacles.

30. The multi modal data stream processing system of claim 27, wherein the computer-executable instructions further cause the computer to generate warning advisories for delivery to relevant selected recipients to mitigate a threat posed by the situation based on the assessed threat level.

31. The multi modal data stream processing system of claim 27, wherein the computer-executable instructions further cause the computer to:

facilitate timely automatic detection and evaluation of malware in a computer network, wherein various features of the computer network are represented by link-weighted associative artificial neural networks; and

dynamically improve deployment of countermeasures against the malware over time.

32. The multi modal data stream processing system of claim 24, further comprising a plurality of sensors which generate the multi modal data streams, the plurality of sensors being connected to the computer processor, wherein said sensors generate data elements corresponding to video, audio, spatial, and temporal features corresponding to a situation.

33. The multi modal data stream processing system of claim 24 wherein said computer processor is part of a spin device.

34. A non-transitory computer readable storage medium, having computer-executable instructions stored thereon that when executed by a computer processor cause the computer processor to:

construct, by a control module of the multi modal data stream processing system, at least one three-partite situation model of a situation represented by a plurality of different data streams, by making associations of artificial neurons of a plurality of artificial neurons of various different types in an artificial neural network in a spatiotemporal associative memory;

dynamically partition, as the situation unfolds over time, by the control module of the multi modal data stream processing system, link-weighted associative artificial neural networks of the artificial neural network into neuronal packets, each network corresponding to at least a respective one of various different combinations of data elements of data representing the plurality of different data streams;

perform dynamic mapping, by the control module of the multi modal data stream processing system, between the neuronal packets as the situation unfolds over time to adjust the at least one three-partite situation model to improve the at least one three-partite situation model for use in understanding of the situation, wherein the constructing, dynamically partitioning and dynamic mapping leave the artificial neural network intact by not changing synaptic weights in the artificial neural network in the constructing, partitioning and the dynamic mapping;

generate situational understanding of the situation as the situation unfolds over time based on the at least one three-partite situation model;

reduce energy consumption and dissipation in the constructing, dynamically partitioning and dynamically mapping by seeking progressively more general and adequate models persisting through various situations wherein the reducing energy consumption and dissipation translates into negentropy production; and

based on a generated situational understanding of a situation, generating in real time by the multi modal data stream processing system appropriate output to facilitate one or more responses to the situation selected from the group consisting of an assessed threat level when objects or conditions in the situation constitute a threat when acting in coordination, identification of objects in an environment of a robotic vehicle or other robotic system, and automatic detection and evaluation of malware in a computer network;

if the situation is an assessed threat level, facilitating an automated intelligent surveillance of the situation;

if the situation is objects in an environment of a robotic vehicle or other robotic system, performing by the robotic vehicle or other robotic system adjusting pursuit of specified objectives and responding to obstacles; and

if the situation is the automatic detection and evaluation of malware in a computer network, dynamically deploying countermeasures against the malware over time.