Systems and methods for intelligent and interpretive analysis of sensor data and generating spatial intelligence using machine learning

• US 10,628,706 B2
• Filed: 05/10/2019
• Issued: 04/21/2020
• Est. Priority Date: 05/11/2018
• Status: Active Grant

• Current Assignee: Ambient AI, Inc.
• Original Assignee: Ambient AI, Inc.
• Inventors: Shrestha, Shikhar, Khanna, Vikesh
• Primary Examiner(s): Allison, Andrae S
• Application Number:
  US16/408,937
• Publication Number:
  US 20190347518A1
• Time in Patent Office: 347 Days
• Field of Search: None
• CPC Class Codes: G06F 18/213 Feature extraction, e.g. by transforming the feature space; Summarisation; Mappings, e.g. subspace methods
• CPC Class Codes: G06N 20/20 Ensemble learning
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06N 5/02 Knowledge representation; Symbolic representation
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G06V 20/44 Event detection
• CPC Class Codes: G06V 20/47 Detecting features for summarising video content
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands

Abstract

Systems and methods for augmenting real-time semantic information to a spatial rendering of a predefined space and providing a real-time situational awareness feed.

Claims

1. A method for augmenting real-time semantic information to a spatial rendering, comprising:

with a comprehension system that is coupled to a plurality of image data sources arranged within a predefined space;

identifying at least one blind spot within the predefined space based on information that identifies a field-of-view (FOV) for each of the plurality of image data sources;

providing, to a user interface system, a user interface that includes a visual representation of a spatial rendering of the predefined space, the user interface including a visual representation of each identified blind spot;

using a semantic module to generate semantic information that identifies a plurality of entities within the predefined space in-real time from image data generated by the plurality of image data sources;

for each identified entity;

tracking a location of the entity in real-time within a coordinate space of the spatial rendering by using contextual metadata stored in a knowledge graph, and

augmenting the visual representation of the spatial rendering with a visual representation of the tracked location, and

for at least one blind spot;

estimating location of at least one identified entity within the blind spot based on a trajectory of the at least one identified entity determined by using image data generated by at least one of the plurality of image data sources, and

augmenting the visual representation of the spatial rendering with a visual representation of each estimated location within the blind spot.

2. The method of claim 1, wherein the contextual metadata includes, for each image data source, a location and orientation of the image data source within the coordinate space of the spatial rendering and information identifying a field-of-view (FOV) of the image data source.

3. The method of claim 2, wherein the contextual information includes, for each image data source, a mapping of coordinates in a coordinate space of image frames of the image data source to coordinates in the coordinate space of the spatial rendering, wherein tracking a location of the entity comprises using a mapping of at least one image data source to track the location of the entity.

4. The method of claim 3, wherein the contextual information includes semantic information generated by the semantic module from at least one of the plurality of image data sources.

5. The method of claim 4, wherein generating semantic information comprises:

generating semantic information based on image data of the plurality of image data sources and the contextual metadata.

6. The method of claim 5, further comprising:

with an event detection module, detecting at least one event by using the contextual metadata and the semantic information.

7. The method of claim 6, further comprising:

with the event detection module, identifying an event entity associated with the detected event, and tracking location of the event entity across FOV'"'"'s of different image data sources by comparing semantic information generated by the semantic module for at least two of the plurality of image data sources.

8. The method of claim 7, further comprising:

with the comprehension system;

updating the user interface to include a live situational awareness feed that includes semantic information generated by the semantic module in real-time for the detected event by using the image data of the plurality of image capture devices.

9. The method of claim 8, further comprising:

with the comprehension system;

receiving sensor information from the user interface system and storing the sensor information in the knowledge graph, wherein the received sensor information includes information identifying a location, orientation and field-of-view (FOV) of at least one image data source; and

receiving from the user interface system a description for at least one location and storing the description in the knowledge graph in association with the respective location.

10. The method of claim 9, further comprising:

updating the visual representation of the spatial rendering to display a view that corresponds to a FOV of an image data source identified by user-selection received via the user interface.