MULTI-SOURCE MULTI-MODAL ACTIVITY RECOGNITION IN AERIAL VIDEO SURVEILLANCE

• US 20170024899A1
• Filed: 06/01/2015
• Published: 01/26/2017
• Est. Priority Date: 06/19/2014
• Status: Active Grant

• Current Assignee: BAE Systems Information and Electronic Systems Integration Incorporated (BAE Systems Plc)
• Original Assignee: BAE Systems Information and Electronic Systems Integration Incorporated (BAE Systems Plc)
• Inventors: Hammoud, Riad, Sahin, Cem S
• Granted Patent: US 9,934,453 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 18/214 Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06F 18/2415 based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate
• CPC Class Codes: G06F 18/2431 Multiple classes
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10032 Satellite or aerial image; Remote sensing
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/30184 Infrastructure
• CPC Class Codes: G06T 2207/30232 Surveillance
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/254 involving subtraction of images
• CPC Class Codes: G06V 10/758 Involving statistics of pixels or of feature values, e.g. histogram matching
• CPC Class Codes: G06V 10/759 Region-based matching
• CPC Class Codes: G06V 10/761 Proximity, similarity or dissimilarity measures
• CPC Class Codes: G06V 20/13 Satellite images
• CPC Class Codes: G06V 2201/07 Target detection
• CPC Class Codes: H04L 51/046 Interoperability with other network applications or services
• CPC Class Codes: H04W 4/025 using location based information parameters

Abstract

Multi-source multi-modal activity recognition for conducting aerial video surveillance comprising detecting and tracking multiple dynamic targets from a moving platform, representing FMV target tracks and chat-messages as graphs of attributes, associating FMV tracks and chat-messages using a probabilistic graph based mapping approach; and detecting spatial-temporal activity boundaries.

Claims

1. A system for multi-source multi-modal activity recognition in conducting aerial video surveillance comprising:

from a moving platform, detecting and tracking, with a video imager, multiple dynamic targets;

recording analyst call outs or chats, and appending said analyst call outs or chats to a file;

representing full motion video (FMV) target tracks and chat-message as graphs of attributes;

associating said FMV tracks and said chat-messages using a probabilistic graph based mapping approach;

detecting spatial-temporal activity boundaries;

categorizing activity of said detected multiple dynamic targets; and

on a display, presenting said activity.

2. The system of claim 1, wherein said detecting and tracking multiple dynamic targets from a moving platform comprises:

differencing registered frames;

using high pixel intensity difference point features to establish correspondences between other points in a previous frame; and

clustering point-velocity pairs into motion regions assumed to be individual targets.

3. The system of claim 1, wherein said step of representing FMV target tracks and chat-message as graphs of attributes comprises:

dividing tracks into segments;

representing attributes of targets as nodes;

characterizing relationships between said nodes as edges; and

chat parsing.

4. The system of claim 1, wherein said step of associating FMV tracks and chat-messages comprise probabilistic matching comprising:

extracting a chat-message and all video tracks in a given time interval from data sets;

generating graph representations of video-tracks and chat messages; and

performing partial graph matching using a probabilistic distance measure.

5. The system of claim 1, wherein said step of detecting spatial-temporal activity boundaries comprises:

extracting features from each labeled track segment;

clustering features in each activity space;

representing each track by a sequence of clusters; and

computing a histogram of human motion flow and neighboring intensity variance.

6. The system of claim 1 wherein overhead imagery review excludes eye tracking and/or touch screen input to determine screen locations of targets-of-interest (TOIs) corresponding to analyst call-outs (ACOS).

7. The system of claim 1 wherein learning comprises generating a similarity score.

8. The system of claim 1 further comprising an output report and querying module.

9. The system of claim 1 wherein unlabeled tracks are matched to a learned activity pattern.

12. The method of claim 7 wherein a similarity score between a new track and an index is defined by a similarity metric which considers only common clusters and a similarity score of temporal gradient for a cluster sequence.

10. A method for multi-source multi-modal activity recognition in conducting aerial video surveillance comprising the steps of:

tracking a target using a video device on an airborne platform;

mapping tracks to graphs comprising multi-graph representation of a single full motion video (FMV) track;

parsing and graph representation of chats;

associating multi-source graphs and assigning activity classes;

learning activity patterns from multi-source associated data; and

visualizing event/activity reports on a display and querying by activity type and geo-location.

11. The method of claim 10 wherein probabilities are assigned according to user-defined weights of attributes for actor, shape, time, color, direction, spatial location, tracking confidence, and target mobility.

13. The method of claim 10 comprising using Cluster Objects Using Recognized Sequence of Estimates (COURSE).

14. The method of claim 10 wherein outlier rejection comprises RANdom SAmple Consensus (RANSAC) to remove bad guesses.

15. The method of claim 10 comprising a Multi-INT Activity Pattern Learning and Exploitation (MAPLE) tool.

16. The method of claim 10 comprising a Hyper-Elliptical Learning and Matching (HELM) unsupervised clustering algorithm to learn activity patterns.

17. The method of claim 10 comprising a Multi-media INdexing and explorER (MINER) showing an automatically generated description.

18. The method of claim 10, wherein output comprises a video summary of activities-of-interest/targets-of-interest AOIs/TOIs allowing non-linear browsing of video content, annotated text-over-video media where only TOIs are highlighted with bounding boxes and synchronized with chat-messages;

an activities index where activities of a same type are grouped together, and

adaptive data play-back allowing for user-selected filtering by geographic location.

19. The method of claim 10 comprising video-indexed by voice annotations (VIVA) stabilization when two frames being registered have greater than about 35% overlap.

20. A system for a multi-source multi-modal probabilistic graph-based association framework for aerial video surveillance comprising:

identifying targets-of-interest corresponding to chat-messages, wherein said chat-messages are the only source to describe a true activity of a target of interest (TOI);

extracting buildings from said FMV;

detecting activity boundaries comprising segmenting full motion video (FMV) tracks from said aerial video surveillance into semantic sub-tracks/segments;

learning activity patterns in low-level feature spaces using reviewed FMV data;

indexing non-reviewed FMV data; and

providing to FMV analysts a user interface display for querying and non-linear browsing of multi-source data.