Systems, methods, apparatuses, and devices for identifying, tracking, and managing unmanned aerial vehicles

• US 10,229,329 B2
• Filed: 11/08/2016
• Issued: 03/12/2019
• Est. Priority Date: 11/08/2016
• Status: Active Grant

• Current Assignee: Dedrone Holdings, Inc.
• Original Assignee: Dedrone Holdings, Inc.
• Inventors: Seeber, Rene, Seebach, Ingo, Meyer, Henning, Schoeler, Markus, Baumgart, Kai, Scheibe, Christian, Prantl, David
• Primary Examiner(s): Bali, Vikkram
• Application Number:
  US15/346,269
• Publication Number:
  US 20180129881A1
• Time in Patent Office: 854 Days
• Field of Search: None
• CPC Class Codes: B64D 1/02 Dropping, ejecting, or releasing articles jettisonable fuel reservoirs B64D37/12
• CPC Class Codes: B64U 20/87 Mounting of imaging devices, e.g. mounting of gimbals
• CPC Class Codes: B64U 2101/16 for controlling, capturing or immobilising other vehicles
• CPC Class Codes: B64U 2101/17 for detecting, disrupting or countering communications
• CPC Class Codes: B64U 2101/31 for surveillance
• CPC Class Codes: G01S 13/04 Systems determining presence of a target based on relative movement of target G01S13/56
• CPC Class Codes: G01S 13/536 using transmission of continuous unmodulated waves, amplitude-, frequency-, or phase-modulated waves
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/20104 Interactive definition of region of interest [ROI]
• CPC Class Codes: G06T 2207/30212 Military
• CPC Class Codes: G06T 2207/30232 Surveillance
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G06V 10/25 Determination of region of interest [ROI] or a volume of interest [VOI]
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 2201/07 Target detection
• CPC Class Codes: G08G 5/0026 located on the ground
• CPC Class Codes: G08G 5/006 in accordance with predefined flight zones, e.g. to avoid prohibited zones
• CPC Class Codes: G08G 5/0069 specially adapted for an unmanned aircraft
• CPC Class Codes: G08G 5/0082 : for monitoring traffic from a ground station radar or analogous systems specially adapted for traffic control G01S13/91

Abstract

Systems, methods, and apparatus for identifying and tracking UAVs including a plurality of sensors operatively connected over a network to a configuration of software and/or hardware. Generally, the plurality of sensors monitors a particular environment and transmits the sensor data to the configuration of software and/or hardware. The data from each individual sensor can be directed towards a process configured to best determine if a UAV is present or approaching the monitored environment. The system generally allows for a detected UAV to be tracked, which may allow for the system or a user of the system to predict how the UAV will continue to behave over time. The sensor information as well as the results generated from the systems and methods may be stored in one or more databases in order to improve the continued identifying and tracking of UAVs.

Claims

1. A method for identifying unmanned aerial vehicles (UAVs) in a particular air space via the use of one or more video sensors, comprising the steps of:

receiving a video frame from a video feed of the particular air space, the video frame comprising a plurality of pixels, wherein the video feed was captured by a particular video sensor proximate to the particular air space;

identifying at least one region of interest (ROI) in the video frame, the at least one ROI comprising an image of an object that may be a UAV flying within the particular air space, wherein the at least one ROI comprises a subset of the plurality of pixels, and wherein the at least one ROI includes a first ROI inter-frame position and a first ROI velocity;

performing a scene learning process with respect to the at least one ROI to determine whether the at least one ROI is part of a learned scene represented by the video frame, the scene learning process comprising the steps of;

comparing the first ROI inter-frame position and the first ROI velocity of the at least one ROI to a plurality of stored ROI inter-frame positions and a plurality of stored ROI velocities of a plurality of stored ROIs to determine if the at least one ROI substantially matches any of the plurality of stored ROIs, wherein the plurality of stored ROIs are associated with substantially reoccurring objects in the particular air space; and

upon determination that the at least one ROI does not substantially match any of the plurality of stored ROIs, performing an object classification process with respect to the at least one ROI to determine whether the object in the image is a UAV, the object classification process comprising the steps of;

extracting image data from the image of the at least one ROI;

comparing the extracted image data to prior image data of objects known to be UAVs to determine a probability that the object in the image is a UAV; and

upon determination that the probability that the object in the image is a UAV exceeds a predetermined threshold, denoting the object in the image as a UAV.

2. The method of claim 1, further comprising the step of prior to identifying the at least one ROI in the video frame, performing a background subtraction process with respect to the video frame.

3. The method of claim 1, further comprising the step of after denoting that the object in the image is a UAV, generating an attention region to encompass the at least one ROI for use in processing of subsequent frames to indicate a region likely to include a UAV.

4. The method of claim 3, wherein the attention region is associated with a heightened attention threshold lower than the predetermined threshold to prevent the at least one ROI from being disqualified as a UAV.

5. The method of claim 4, further comprising the steps of:

receiving a subsequent video frame from the video feed of the particular air space;

identifying at least one subsequent ROI in the subsequent video frame, the at least one subsequent ROI comprising a subsequent image of a subsequent object that may be a UAV flying within the particular airspace;

determining that the subsequent ROI lies within the attention region; and

upon determination that the subsequent ROI lies within the attention region, performing the object classification process with respect to the subsequent ROI to determine whether the subsequent object in the subsequent image is a UAV, wherein the subsequent object in the subsequent image is denoted as a UAV if a subsequent probability that the subsequent object in the subsequent image is a UAV exceeds the heighted attention threshold.

6. The method of claim 1, wherein the extracted image data comprises RGB color values at one or more locations within the at least one ROI.

7. The method of claim 1, wherein the extracted image data comprises background subtraction data at one or more locations within the at least one ROI.

8. The method of claim 1, wherein the object classification process further comprises the step of comparing the extracted image data to prior image data of objects known to be non-UAVs to determine a probability that the object in the image is a UAV.

9. The method of claim 1, wherein the step of denoting the object in the image as a UAV comprises assigning a confidence level to the object in the image as UAV, wherein the confidence level comprises a statistical confidence measure that the object in the image is a UAV.

10. The method of claim 1, further comprising the step of storing the video frame in a database.

11. The method of claim 10, further comprising the step of associating the stored frame in the database with an indication that the object in the image comprises a UAV.

12. The method of claim 1, wherein the ROI comprises a rectangular shape.

13. The method of claim 1, further comprising the step of initiating an alert to a system user that a UAV has been detected in the particular air space.

14. The method of claim 1, wherein the at least one ROI further includes a first ROI size and first ROI shape.

15. The method of claim 14, wherein the scene learning process further comprises the step of comparing the first ROI size and first ROI shape of the at least one ROI to a plurality of stored ROI sizes and a plurality of stored ROI shapes of the plurality of stored ROIs to determine if the at least one ROI substantially matches any of the plurality of stored ROIs.

16. The method of claim 1, wherein the particular video sensor is fixed in position.

17. The method of claim 16, wherein the fixed position includes an orientation with heading parameters comprising yaw, pitch, and roll values associated with ROIs from video frames of the video feed.

18. The method of claim 17, wherein the scene learning process further comprises the step of confirming that heading parameters associated with the at least one ROI correspond to heading parameters associated with the plurality of stored ROIs.

19. A system for identifying unmanned aerial vehicles (UAVs) in a particular air space, comprising:

a video sensor, wherein the video sensor is proximate to the particular air space and is configured to collect and transmit a video frame from a video feed of the particular air space, the video frame comprising a plurality of pixels; and

a processor operatively coupled to the video sensor;

wherein the processor is operative to;

identify at least one region of interest (ROI) in the video frame, the at least one ROI comprising an image of an object that may be a UAV flying within the particular air space, wherein the at least one ROI comprises a subset of the plurality of pixels, and wherein the at least one ROI includes a first ROI inter-frame position and a first ROI velocity;

perform a scene learning process with respect to the at least one ROI to determine whether the at least one ROI is part of a learned scene represented by the video frame, the scene learning process comprising the steps of;

comparing the first ROI inter-frame position and the first ROI velocity of the at least one ROI to a plurality of stored ROI inter-frame positions and a plurality of stored ROI velocities of a plurality of stored ROIs to determine if the at least one ROI substantially matches any of the plurality of stored ROIs, wherein the plurality of stored ROIs are associated with substantially reoccurring objects in the particular air space; and

upon determination that the at least one ROI does not substantially match any of the plurality of stored ROIs, perform an object classification process with respect to the at least one ROI to determine whether the object in the image is a UAV, the object classification process comprising the steps of;

extracting image data from the image of the at least one ROI;

comparing the extracted image data to prior image data of objects known to be UAVs to determine a probability that the object in the image is a UAV; and

upon determination that the probability that the object in the image is a UAV exceeds a predetermined threshold, denoting the object in the image as a UAV.

20. The system of claim 19, wherein the processor is further operative to, after denoting that the object in the image is a UAV, generate an attention region to encompass the at least one ROI for use in processing of subsequent frames to indicate a region likely to include a UAV.

21. The system of claim 20, wherein the attention region is associated with a heightened attention threshold lower than the predetermined threshold to prevent the at least one ROI from being disqualified as a UAV.

22. The system of claim 21, wherein the processor is further operative to:

receive a subsequent video frame from the video feed of the particular air space;

identify at least one subsequent ROI in the subsequent video frame, the at least one subsequent ROI comprising a subsequent image of a subsequent object that may be a UAV flying within the particular airspace;

determine that the subsequent ROI lies within the attention region; and

upon determination that the subsequent ROI lies within the attention region, perform the object classification process with respect to the subsequent ROI to determine whether the subsequent object in the subsequent image is a UAV, wherein the subsequent object in the subsequent image is denoted as a UAV if a subsequent probability that the subsequent object in the subsequent image is a UAV exceeds the heighted attention threshold.

23. The system of claim 19, wherein the extracted image data comprises RGB color values at one or more locations within the at least one ROI.

24. The system, of claim 19, wherein the extracted image data comprises background subtraction data at one or more locations within the at least one ROI.

25. The system of claim 19, wherein the object classification process further comprises the step of comparing the extracted image data to prior image data of objects known to be non-UAVs to determine a probability that the object in the image is a UAV.

26. The system of claim 19, wherein the step of denoting the object in the image as a UAV comprises assigning a confidence level to the object in the image as UAV, wherein the confidence level comprises a statistical confidence measure that the object in the image is a UAV.

27. The system of claim 19, wherein the system further comprises a database, and the processor is further operative to store the video frame in the database.

28. The system of claim 27, wherein the processor is further operative to associate the stored frame in the database with an indication that the object in the image comprises a UAV.

29. The system of claim 19, wherein the ROI comprises a rectangular shape.

30. The system of claim 19, wherein the processor is further operative to initiate an alert to a system user that a UAV has been detected in the particular air space.

31. The system of claim 19, wherein the at least one ROI further includes a first ROI size and first ROI shape.

32. The system of claim 31, wherein the scene learning process further comprises the step of comparing the first ROI size and first ROI shape of the at least one ROI to a plurality of stored ROI sizes and a plurality of stored ROI shapes of the plurality of stored ROIs to determine if the at least one ROI substantially matches any of the plurality of stored ROIs.

33. The system of claim 19, wherein the particular video sensor is fixed in position.

34. The system of claim 33, wherein the fixed position includes an orientation with heading parameters comprising yaw, pitch, and roll values associated with ROIs from video frames of the video feed.

35. The system of claim 34, wherein the scene learning process further comprises the step of confirming that heading parameters associated with the at least one ROI correspond to heading parameters associated with the plurality of stored ROIs.