Methods and devices for cognitive-based image data analytics in real time

• US 10,078,791 B2
• Filed: 01/09/2015
• Issued: 09/18/2018
• Est. Priority Date: 01/09/2014
• Status: Expired due to Fees

• Current Assignee: Irvine Sensors Corp.
• Original Assignee: Irvine Sensors Corp.
• Inventors: Justice, James, Ludwig, David, Villacorta, Virgilio, Asadi, Omar, Knutson, Fredrik, Yam, Mannchuoy, Weaver, Eric
• Primary Examiner(s): Kholdebarin, Iman K
• Application Number:
  US14/593,021
• Publication Number:
  US 20170372153A1
• Time in Patent Office: 1,348 Days
• Field of Search: None
• CPC Class Codes: G06F 18/217 Validation; Performance evaluation; Active pattern learning techniques
• CPC Class Codes: G06F 18/2415 based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate
• CPC Class Codes: G06T 1/20 Processor architectures; Processor configuration, e.g. pipelining
• CPC Class Codes: G06T 2200/28 involving image processing hardware
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/30232 Surveillance
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/277 involving stochastic approaches, e.g. using Kalman filters
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 10/955 using specific electronic processors
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69

Abstract

An invention is disclosed for a real time video analytic processor that embodies algorithms and processing architectures that process a wide variety of sensor images in a fashion that emulates how the human visual path processes and interprets image content. spatial, temporal, and color content of images are analyzed and the salient features of the images determined. These salient features are then compared to the salient features of objects of user interest in order to detect, track, classify, and characterize the activities of the objects. Objects or activities of interest are annotated in the image streams and alerts of critical events are provided to users. Instantiation of the cognitive processing can be accomplished on multi-FPGA and multi-GPU processing hardwares.

Claims

1. A real time video analytic image processor which emulates the human visual path comprising:

an imaging sensor configured for capturing and outputting a sequence of image frames to define a video stream;

an edge processor configured to receive the video stream to define an edge processor video stream;

a core processor configured to receive the video stream to define a core processor video stream;

the edge processor and the core processor comprising a hash synchronization function whereby the image frames are synchronized to the edge processor and the core processor using a unique hash identifier;

the edge processor configured to perform a video stream pre-processing function, an analytic function, an analytic metadata output function and a post-processing video compression function to provide an analytic metadata output and a compressed video output;

the core processor configured to receive the analytic metadata output and the compressed video output from the edge processor; and

;

the core processor configured to perform a salient feature extraction function by means of a convolution, an object classification and an event detection from the core processor video stream, the analytic metadata output and the compressed video output based on a correlation between an observed object and an object of interest.

2. The image processor of claim 1 comprising a plurality of imaging sensors, each imaging sensor configured for capturing and outputting a sequence of image frames to define a plurality of independent video streams.

3. The image processor of claim 1 configured for processing still imagery.

4. The image processor of claim 1 configured for processing high definition (HD) video or full motion video (FMV) imagery.

5. The image processor of claim 1 configured for processing thermal imagery.

6. The image processor of claim 1 configured for processing multispectral imagery.

7. The image processor of claim 1 configured for processing hyperspectral imagery.

8. The image processor of claim 1 configured for processing LIDAR imagery.

9. The image processor of claim 1 configured for processing radar imagery including synthetic aperture array (SAR) and ground moving target indicator (GMTI) imagery.

10. The image processor of claim 1 wherein the salient feature extraction, classification and annotation function is performed in real time at the same rate as the sensor is producing image data.