Multi-source multi-modal activity recognition in aerial video surveillance

• US 9,934,453 B2
• Filed: 06/01/2015
• Issued: 04/03/2018
• Est. Priority Date: 06/19/2014
• Status: Active Grant

• Current Assignee: BAE Systems Information and Electronic Systems Integration Incorporated (BAE Systems Plc)
• Original Assignee: BAE Systems Information and Electronic Systems Integration Incorporated (BAE Systems Plc)
• Inventors: Hammoud, Riad, Sahin, Cem S, Blasch, Erik
• Primary Examiner(s): Niu, Feng
• Assistant Examiner(s): Thomas, Mia M
• Application Number:
  US14/727,074
• Publication Number:
  US 20170024899A1
• Time in Patent Office: 1,037 Days
• CPC Class Codes: G06F 18/214 Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06F 18/2415 based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate
• CPC Class Codes: G06F 18/2431 Multiple classes
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10032 Satellite or aerial image; Remote sensing
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/30184 Infrastructure
• CPC Class Codes: G06T 2207/30232 Surveillance
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/254 involving subtraction of images
• CPC Class Codes: G06V 10/758 Involving statistics of pixels or of feature values, e.g. histogram matching
• CPC Class Codes: G06V 10/759 Region-based matching
• CPC Class Codes: G06V 10/761 Proximity, similarity or dissimilarity measures
• CPC Class Codes: G06V 20/13 Satellite images
• CPC Class Codes: G06V 2201/07 Target detection
• CPC Class Codes: H04L 51/046 Interoperability with other network applications or services
• CPC Class Codes: H04W 4/025 using location based information parameters

Abstract

Multi-source multi-modal activity recognition for conducting aerial video surveillance comprising detecting and tracking multiple dynamic targets from a moving platform, representing FMV target tracks and chat-messages as graphs of attributes, associating FMV tracks and chat-messages using a probabilistic graph based mapping approach; and detecting spatial-temporal activity boundaries.

Claims

1. A system for multi-source multi-modal activity recognition in conducting aerial video surveillance comprising:

from a moving platform, detecting and tracking, with a video imager, multiple dynamic targets, wherein the detecting and tracking comprises

differencing registered frames;

using high pixel difference point features to establish correspondences between other points in a previous frame; and

clustering point-velocity pairs into motion regions assumed to be individual targets;

recording analyst call outs or chats, and appending said analyst call outs or chats to a file;

representing full motion video (FMV) target tracks and chat-messages as graphs of attributes;

associating said FMV tracks and said chat-messages using a probabilistic graph based mapping approach;

detecting spatial-temporal activity boundaries;

categorizing activity of said detected multiple dynamic targets; and

on a display, presenting said activity.

2. The system of claim 1, wherein said step of representing FMV target tracks and chat-message as graphs of attributes comprises:

dividing tracks into segments;

representing attributes of targets as nodes;

characterizing relationships between said nodes as edges; and

chat parsing.

3. The system of claim 1, wherein said step of associating FMV tracks and chat-messages comprise probabilistic matching comprising:

extracting a chat-message and all video tracks in a given time interval from data sets;

generating graph representations of video-tracks and chat messages; and

performing partial graph matching using a probabilistic distance measure.

4. The system of claim 1, wherein said step of detecting spatial-temporal activity boundaries comprises:

extracting features from a labeled track segment;

clustering features in each activity space;

representing each track by a sequence of clusters; and

computing a histogram of human motion flow and neighboring intensity variance.

5. The system of claim 1 wherein overhead imagery review excludes eye tracking and/or touch screen input to determine screen locations of targets-of-interest (TOIs) corresponding to analyst call-outs (ACOs).

6. The system of claim 1 wherein learning comprises generating a similarity score.

7. The method of claim 6 wherein a similarity score between a new track and an index is defined by a similarity metric which considers only common clusters and a similarity score of temporal gradient for a cluster sequence.

8. The system of claim 1 further comprising an output report and querying module.

9. The system of claim 1 wherein unlabeled tracks are matched to a learned activity pattern.

10. The system of claim 1 further comprising,

a querying module, wherein the querying module further comprises, an activities-of-interest index (AOI) where activities are grouped together by movement type and geo-location;

and

an adaptive data play-back summary of associated text (ACO) and activity video segments of targets-of-interest (TOIs) in both pixel and geo-coordinates that allow for user-selected filtering by geographic location.

11. A method for multi-source multi-modal activity recognition in conducting aerial video surveillance comprising:

tracking a target using a video device on an airborne platform;

mapping tracks to graphs comprising multi-graph representation of a single full motion video (FMV) track;

parsing and graph representation of chats;

associating multi-source graphs and assigning activity classes;

learning activity patterns from multi-source associated data; and

visualizing event/activity reports on a display and querying by activity type and geo-location,

wherein the reports comprise a video summary of activities-of-interest/targets-of-interest AOIs/TOIs allowing non-linear browsing of video content, annotated text-over-video media where only TOIs are highlighted with bounding boxes and synchronized with chat-messages;

grouping activities of a same type into an activities index, and

allowing adaptive data play-back for user-selected filtering by geographic location.

12. The method of claim 11 wherein probabilities are assigned according to user-defined weights of attributes for actor, shape, time, color, direction, spatial location, tracking confidence, and target mobility.

13. The method of claim 11 comprising using Cluster Objects Using Recognized Sequence of Estimates (COURSE).

14. The method of claim 11 wherein outlier rejection comprises RANdom SAmple Consensus (RANSAC) to remove bad guesses.

15. The method of claim 11 comprising a Multi-INT Activity Pattern Learning and Exploitation (MAPLE) tool.

16. The method of claim 11 comprising a Hyper-Elliptical Learning and Matching (HELM) unsupervised clustering algorithm to learn activity patterns.

17. The method of claim 11 comprising a Multi-media INdexing and explorER (MINER) showing an automatically generated description.

18. The method of claim 11 comprising video-indexed by voice annotations (VIVA) stabilization when two frames being registered have greater than about 35% overlap.

19. A system for a multi-source multi-modal probabilistic graph-based association framework for aerial video surveillance comprising:

reviewing by a reviewer at least some of the aerial video surveillance providing reviewed FMV data with a resulting set of non-reviewed FMV data;

identifying targets-of-interest corresponding to chat-messages, wherein said chat-messages are the only source to describe a true activity of a target of interest (TOI);

extracting objects from a full motion video (FMV) of the aerial video surveillance;

detecting activity boundaries comprising segmenting full motion video (FMV) tracks from said aerial video surveillance into semantic sub-tracks/segments;

learning activity patterns in low-level feature spaces using the reviewed FMV data;

indexing non-reviewed FMV data; and

providing to FMV analysts a user interface display for querying and non-linear browsing of multi-source data.