Graphical rendition of multi-modal data

• US 9,586,314 B2
• Filed: 05/20/2015
• Issued: 03/07/2017
• Est. Priority Date: 08/30/2011
• Status: Active Grant

• Current Assignee: Humatics Corporation
• Original Assignee: 5D Robotics, Inc. (Humatics Corporation)
• Inventors: Bruemmer, David J.
• Primary Examiner(s): Nguyen, Kimbinh T
• Application Number:
  US14/717,219
• Publication Number:
  US 20150269757A1
• Time in Patent Office: 657 Days
• Field of Search: 345/418, 345/419, 345/440, 345/441
• US Class Current: 1/1
• CPC Class Codes: B25J 9/08 characterised by modular constructions
• CPC Class Codes: B25J 9/1602 characterised by the control system, structure, architecture
• CPC Class Codes: B25J 9/161 Hardware, e.g. neural networks, fuzzy logic, interfaces, processor
• CPC Class Codes: B25J 9/1633 compliant, force, torque control, e.g. combined with position control
• CPC Class Codes: B25J 9/1666 Avoiding collision or forbidden zones
• CPC Class Codes: B25J 9/1684 Tracking a line or surface by means of sensors
• CPC Class Codes: B25J 9/1694 characterised by use of sensors other than normal servo-feedback from position, speed or acceleration sensors, perception control, multi-sensor controlled systems, sensor fusion
• CPC Class Codes: B25J 9/1697 Vision controlled systems
• CPC Class Codes: G01C 21/34 Route searching; Route guidance
• CPC Class Codes: G06F 18/256 of results relating to different input data, e.g. multimodal recognition
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06T 11/206 Drawing of charts or graphs
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: G08G 1/22 Platooning, i.e. convoy of communicating vehicles
• CPC Class Codes: Y10S 901/09 Closed loop, sensor feedback controls arm movement
• CPC Class Codes: Y10S 901/10 Sensor physically contacts and follows work contour

Abstract

Changes and anomalies in multi-modal data are detected, collected and abstracted into understandable and actionable formats utilizing, for example, color, intensity, icons and texture creating a rendition of current and developing situations and events. Changes and anomalies in multi-modal sensor data are detected, aggregated, abstracted and filtered using case-based reasoning providing a tractable data dimensionality. From this collection of data situations are recognized and presented in a means so as to assist a user in accessing an environment and formulate the basis a recommended course of action.

Claims

1. A system for rendition of multi-modal data, comprising:

a plurality of sources of multi-modal data configured to collect data from an environment wherein each of the plurality of sources is associated with a distinct sensor modality having distinct data modalities;

a processor communicatively coupled to each of the plurality of sources of multi-modal data; and

a storage medium communicatively coupled to the processor and tangibly embodying a plurality of executable modules of instructions wherein each module of instructions is connected to the processor via a bus, the plurality of executable modules including

a data collection module, configured to asynchronously collect event data across the plurality of data modalities from each of the plurality of sources of multi-modal data and store the collected event data in memory,

an anomaly detection module, configured to independently detect from the collected event data one or more event anomalies for each of a plurality of data modalities against background/historical data and to associate each event anomaly with a data modality temporal identifier and a data modality spatial identifier,

an event anomaly recognition module configured to identify a spatial and a temporal relationship between each of the one or more event anomalies among the plurality of data modalities,

an anomaly correlation module configured to combine identified event anomalies to form a coherent common event representation that is cross-correlated based on a predetermined modality temporal window and a predetermined spatial window, and

a rendition module configured to render cross-correlated common representations of detected event anomalies and the environment from a plurality of perspectives.

2. The system for rendition of multi-modal data according to claim 1, wherein collected event data from each data modality is processed asynchronously and in parallel with collected event data from other data modalities.

3. The system for rendition of multi-modal data according to claim 1, wherein the plurality of sources of multi-modal data includes a ultra-wide band transceiver.

4. The system for rendition of multi-modal data according to claim 1, wherein the collected event data collected from the plurality of sources includes a spatial and temporal reference.

5. The system for rendition of multi-modal data according to claim 1, wherein the anomaly correlation module combines two or more related event anomalies within a data modality.

6. The system for rendition of multi-modal data according to claim 1, wherein the anomaly detection module prioritizes event anomalies based on one or more predetermined criteria.

7. The system for rendition of multi-modal data according to claim 1, wherein the anomaly detection module filters detected anomalies based on background/historical data.

8. The system for rendition of multi-modal data according to claim 1, wherein the rendition module represents clusters of anomalies in visual formats.

9. A computer implemented method for rendering anomaly events based on multi-modal data, comprising executing on a processor the steps of:

collecting from a plurality of sources of multi-modal data within an environment static and/or dynamic multimodal data wherein each of the plurality of sources is associated with a distinct sensor modality having distinct data modalities;

detecting independently from the collected static and/for dynamic multimodal data and based on background and historical data, one or more anomalies;

associating each of the one or more detected anomalies with a data modality temporal identifier and a data modality spatial identifier;

identifying from the one or more detected anomalies, one or more anomaly events based on a correlation between the data modality temporal identifiers and the data modality spatial identifiers;

combining identified anomaly events to form a coherent common event representation that is cross-correlated based on a predetermined modality temporal window and a predetermined spatial window; and

rendering on a graphical display the coherent common event representation of identified anomaly events and the environment in which they occur from a plurality of perspectives.

10. The method for rendering anomaly events based on multi-modal data according to claim 9, further comprising abstracting collected static and/or dynamic data.

11. The method for rendering anomaly events based on multi-modal data according to claim 10, further comprising comparing abstracted collected data to pre-identified event temporal and spatial constraints.

12. The method for rendering anomaly events based on multi-modal data according to claim 9, wherein the plurality of sources of multi-modal data includes a ultra-wide band transceiver.

13. The method for rendering anomaly events based on multi-modal data according to claim 12, wherein change analysis includes motion tracking and occupancy change analysis.

14. The method for rendering anomaly events based on multi-modal data according to claim 12, wherein change analysis includes radio frequency analysis.

15. The method for rendering anomaly events based on multi-modal data according to claim 12, wherein change analysis includes acoustical change analysis.

16. The method for rendering anomaly events based on multi-modal data according to claim 12, wherein change analysis includes visual and thermal imagery analysis.

17. The method for rendering anomaly events based on multi-modal data according to claim 12, wherein change analysis includes chemical change analysis.

18. The method for rendering anomaly events based on multi-modal data according to claim 9, wherein detecting includes recognizing data fluctuations within each modality.

19. The method for rendering anomaly events based on multi-modal data according to claim 9, wherein identifying includes determining whether two or more detected anomalies occur within a pre-identified temporal period of time and within a pre-identified spatial proximity.

20. The method for rendering anomaly events based on multi-modal data according to claim 19, wherein the pre-identified spatial proximity is based on environmental features.

21. The method for rendering anomaly events based on multi-modal data according to claim 9, wherein identifying includes recognizing physical and temporal relationships between and among the detected one or more anomalies.

22. The method for rendering anomaly events based on multi-modal data according to claim 9, further comprising clustering the one or more anomalies based on common spatially salient features.

23. A non-transitory computer-readable storage medium comprising instructions stored thereon for rendering anomaly events based on multi-modal data that, when executed on a processor, perform the steps of:

collecting from a plurality of sources of multi-modal data within an environment static and/or dynamic multimodal data wherein each of the plurality of sources is associated with a distinct sensor modality having distinct data modalities;

detecting independently from the collected static and/or dynamic multimodal data and based on background and historical data, one or more anomalies;

associating each of the one or more detected anomalies with a data modality temporal identifier and a data modality spatial identifier;

identifying from the one or more detected anomalies, one or more anomaly events based on a correlation between the data modality temporal identifiers and the data modality spatial identifiers;

combining identified anomaly events to form a coherent common event representation that is cross-correlated based on a predetermined modality temporal window and a predetermined spatial window; and

rendering on a graphical display the coherent common event representation of identified anomaly events and the environment in which they occur from a plurality of perspectives.

24. The non-transitory computer-readable storage medium of claim 23, further comprising instructions for abstracting collected static and/or dynamic multimodal data.

25. The non-transitory computer-readable storage medium of claim 24, further comprising instructions for comparing abstracted collected data to pre-identified event characteristics.

26. The non-transitory computer-readable storage medium of claim 23, further comprising instructions for clustering the detected one or more anomalies based on common spatially salient features.

27. The non-transitory computer-readable storage medium of claim 23, wherein the instructions for identifying includes identifying physical and temporal relationships between and among the detected one or more anomalies.