Diverse radio frequency signature, video, and image sensing for detection and localization

• US 9,880,256 B2
• Filed: 10/30/2015
• Issued: 01/30/2018
• Est. Priority Date: 10/30/2014
• Status: Active Grant

• Current Assignee: Bastille Networks, Inc.
• Original Assignee: Bastille Networks, Inc.
• Inventors: Baxley, Robert John, Rouland, Christopher Jay, Engle, Michael Thomas
• Primary Examiner(s): Liu, Harry
• Application Number:
  US14/929,212
• Publication Number:
  US 20160124071A1
• Time in Patent Office: 823 Days
• Field of Search: 342452
• CPC Class Codes: G01S 2205/01 specially adapted for specific applications
• CPC Class Codes: G06F 16/285 Clustering or classification
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/60 Analysis of geometric attributes
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G06V 20/46 Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 2201/10 Recognition assisted with metadata
• CPC Class Codes: G08B 13/00 Burglar, theft or intruder alarms
• CPC Class Codes: H04B 1/18 Input circuits, e.g. for coupling to an antenna or a transmission line coupling networks between antennas or lines and receivers independent of the nature of the receiver H03H
• CPC Class Codes: H04B 17/10 of transmitters
• CPC Class Codes: H04B 17/391 Modelling the propagation channel
• CPC Class Codes: H04K 2203/16 for telephony
• CPC Class Codes: H04K 2203/18 for wireless local area networks or WLAN
• CPC Class Codes: H04K 3/22 including jamming detection and monitoring
• CPC Class Codes: H04L 63/1416 Event detection, e.g. attack signature detection
• CPC Class Codes: H04L 63/1425 Traffic logging, e.g. anomaly detection
• CPC Class Codes: H04L 63/302 gathering intelligence information for situation awareness or reconnaissance
• CPC Class Codes: H04N 5/265 Mixing
• CPC Class Codes: H04W 12/08 : Access security
• CPC Class Codes: H04W 12/10 : Integrity
• CPC Class Codes: H04W 12/12 : Detection or prevention of fraud
• CPC Class Codes: H04W 12/122 : Counter-measures against attacks; Protection against rogue devices
• CPC Class Codes: H04W 12/63 : Location-dependent; Proximity-dependent
• CPC Class Codes: H04W 12/64 : using geofenced areas
• CPC Class Codes: H04W 12/79 : Radio fingerprint
• CPC Class Codes: H04W 24/08 : Testing, supervising or monitoring using real traffic
• CPC Class Codes: H04W 4/021 : Services related to particular areas, e.g. point of interest [POI] services, venue services or geofences
• CPC Class Codes: H04W 4/70 : Services for machine-to-machine communication [M2M] or machine type communication [MTC]
• CPC Class Codes: H04W 84/18 : Self-organising networks, e.g. ad-hoc networks or sensor networks

Abstract

Systems and methods can support coprocessing radio signals and video to identify and locate a radio transmitter. Positions and orientations for cameras and RF sensors may be maintained. An RF signature associated with the radio transmitter may be received from the RF sensors to determine an RF persona. A first physical location for the radio transmitter may be estimated according to a physical radio propagation model operating on RF signals. A video stream from one or more of the cameras may be received. An individual may be identified in the video stream using computer vision techniques. A second physical location for the radio transmitter may be estimated from the video stream. Relationships may be established between the first physical location and the second physical location and between the RF persona and the identified individual. The relationships may be presented to an operator interface.

Claims

1. A method for coprocessing radio signals and video to identify and locate a radio transmitter, comprising:

receiving positions and orientations for cameras and RF sensors;

receiving an RF signature associated with the radio transmitter from the RF sensors;

determining an RF persona associated with the RF signature;

estimating a first physical location for the radio transmitter according to a physical radio propagation model operating on signals associated with the RF sensors;

associating the RF persona with the first physical location;

receiving a video stream from one or more of the cameras covering an area associated with the RF sensors;

identifying an individual in the video stream using computer vision techniques;

estimating a second physical location for the radio transmitter according to the identified individual within the video stream;

establishing a first relationship between the first physical location and the second physical location;

establishing a second relationship between the RF persona and the identified individual; and

presenting the first established relationship and the second established relationship to one or more operator interfaces.

2. The method of claim 1, further comprising overlaying the video stream with information associated with the RF persona.

3. The method of claim 1, further comprising establishing one or more geometrical transformations between a geometry associated with the cameras and a geometry associated with the RF sensors.

4. The method of claim 1, further comprising stitching together video clips from the video stream over time and space in response to those clips being associated with the RF persona.

5. The method of claim 1, further comprising adapting orientations for the cameras according to a position or a motion of the RF persona.

6. The method of claim 1, further comprising refining the physical radio propagation model according to an error distance between the first physical location and the second physical location.

7. The method of claim 1, further comprising identifying a mismatch between the video stream and the RF persona and classifying the mismatch as a potential threat.

8. The method of claim 1, further comprising overlaying the video stream with a radio frequency heatmap according to signals associated with the RF sensors.

9. The method of claim 1, further comprising supporting indexed search of the video stream to ease location of clips of the video stream associated with the RF persona.

10. The method of claim 1, further comprising overlaying the video stream with motion tracks based upon changes in the first physical location.

11. A system for determining a physical position of a radio transmitter, comprising:

one or more sensor antennas;

one or more cameras;

one or more RF sensors configured to couple radio frequency signals from the one or more sensor antennas and generate data samples representing at least a portion of the radio frequency signals; and

a signal analysis engine comprising one or more processing units, and one or more processing modules configuring the one or more processing units to;

receive positions and orientations associated with the cameras;

receive positions and orientations associated with the RF sensors;

receive an RF signature associated with the radio transmitter from the RF sensors;

determine an RF persona associated with the RF signature;

estimate a first physical location for the radio transmitter according to a physical radio propagation model operating on signals associated with the RF sensors;

associate the RF persona with the first physical location;

receive a video stream from the cameras covering an area associated with the RF sensors;

identify an individual in the video stream using computer vision techniques;

estimate a second physical location for the radio transmitter according to the identified individual within the video stream;

establish an error distance between the first physical location and the second physical location; and

refine the physical radio propagation model according to the error distance.

12. The system of claim 11, wherein the one or more processing units are further configured to overlay the video stream with information associated with the RF persona.

13. The system of claim 11, wherein the one or more processing units are further configured to overlay the video stream with motion tracks based upon changes in the first physical location.

14. The system of claim 11, wherein the one or more processing units are further configured to overlay the video stream with a radio frequency heatmap according to signals associated with the RF sensors.

15. The system of claim 11, wherein the one or more processing units are further configured to establish one or more geometrical transformations between a geometry associated with the cameras and a geometry associated with the RF sensors.

16. The system of claim 11, wherein the one or more processing units are further configured to stitch together video clips from the video stream over time and space in response to those clips being associated with the RF persona.

17. The system of claim 11, wherein the one or more processing units are further configured to adapt orientations for the cameras according to a position or a motion of the RF persona.

18. The system of claim 11, wherein the one or more processing units are further configured to identify a mismatch between the identified individual within the video stream and the RF persona and classify the mismatch as a potential threat.

19. The system of claim 11, wherein the one or more processing units are further configured to maintain a search index of the video stream to ease location of clips within the video stream associated with the RF persona.

20. A system for coprocessing radio signals and video to identify and locate a radio transmitter, comprising:

one or more RF sensors within the electromagnetic environment;

one or more cameras covering an area associated with the electromagnetic environment;

one or more operator interfaces for communicating information associated with wireless devices; and

a signal analysis engine comprising one or more processing units, and one or more processing modules configuring the one or more processing units to;

receive positions and orientations for the cameras and the RF sensors;

establish one or more geometrical transformations between a geometry associated with the cameras and a geometry associated with the RF sensors;

receive an RF signature associated with the radio transmitter from the RF sensors;

determine an RF persona associated with the RF signature;

estimate a first physical location for the radio transmitter according to a physical radio propagation model operating on signals associated with the RF sensors;

associate the RF persona with the first physical location;

receive a video stream from one or more of the cameras;

identify an individual in the video stream using computer vision techniques;

estimate a second physical location for the radio transmitter according to the identified individual within the video stream;

overlay the video stream with information associated with the RF persona;

establish a first relationship between the first physical location and the second physical location;

establish a second relationship between the RF persona and the identified individual; and

present the first established relationship and the second established relationship to the one or more operator interfaces.