Scanning environments and tracking unmanned aerial vehicles

• US 10,416,668 B2
• Filed: 03/02/2016
• Issued: 09/17/2019
• Est. Priority Date: 03/03/2015
• Status: Active Grant

• Current Assignee: Prenav, Inc.
• Original Assignee: Prenav, Inc.
• Inventors: Hammond, Asa, Schuett, Nathan, Busek, Naimisaranya Das
• Primary Examiner(s): Maung, Nay A
• Assistant Examiner(s): Torres, Jose
• Application Number:
  US15/059,101
• Publication Number:
  US 20160292869A1
• Time in Patent Office: 1,294 Days
• CPC Class Codes: B64C 39/024 of the remote controlled vehicle type, i.e. RPV
• CPC Class Codes: B64U 10/13 Flying platforms
• CPC Class Codes: B64U 2201/20 Remote controls
• CPC Class Codes: F03D 17/00 Monitoring or testing of wind motors, e.g. diagnostics testing during commissioning of wind motors F03D13/30
• CPC Class Codes: G01C 21/3415 Dynamic re-routing, e.g. recalculating the route when the user deviates from calculated route or after detecting real-time traffic data or accidents
• CPC Class Codes: G01S 17/66 Tracking systems using electromagnetic waves other than radio waves
• CPC Class Codes: G01S 17/86 Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
• CPC Class Codes: G01S 17/87 Combinations of systems using electromagnetic waves other than radio waves
• CPC Class Codes: G01S 17/89 for mapping or imaging
• CPC Class Codes: G01S 17/933 of aircraft or spacecraft
• CPC Class Codes: G01S 7/4813 Housing arrangements
• CPC Class Codes: G05D 1/0033 by having the operator tracking the vehicle either by direct line of sight or via one or more cameras located remotely from the vehicle
• CPC Class Codes: G05D 1/0044 by providing the operator with a computer generated representation of the environment of the vehicle, e.g. virtual reality, maps maps used for automatic navigation G05D1/0274; flight directors G01C23/005
• CPC Class Codes: G05D 1/0094 involving pointing a payload, e.g. camera, weapon, sensor, towards a fixed or moving target
• CPC Class Codes: G05D 1/102 specially adapted for vertical take-off of aircraft
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06F 3/0484 for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
• CPC Class Codes: G06T 17/05 Geographic models
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/10032 Satellite or aerial image; Remote sensing
• CPC Class Codes: G06T 2207/20221 : Image fusion; Image merging
• CPC Class Codes: G06T 2207/30181 : Earth observation
• CPC Class Codes: G06T 2207/30241 : Trajectory
• CPC Class Codes: G06T 2207/30252 : Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/0004 : Industrial image inspection
• CPC Class Codes: G06T 7/20 : Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/60 : Analysis of geometric attributes
• CPC Class Codes: G06V 20/653 : by matching three-dimensional models, e.g. conformal mapping of Riemann surfaces
• CPC Class Codes: G08G 5/0069 : specially adapted for an unmanned aircraft
• CPC Class Codes: G08G 5/045 : Navigation or guidance aids, e.g. determination of anti-collision manoeuvers radar or analogous systems specially adapted for anti-collision between aircraft G01S13/933
• CPC Class Codes: H04N 13/128 : Adjusting depth or disparity
• CPC Class Codes: H04N 2013/0081 : Depth or disparity estimation from stereoscopic image signals

Abstract

Systems and methods for scanning environments and tracking unmanned aerial vehicles within the scanned environments are disclosed. A method in accordance with a particular embodiment includes using a rangefinder off-board an unmanned air vehicle (UAV) to identify points in a region. The method can further include forming a computer-based map of the region with the points and using the rangefinder and a camera to locate the UAV as it moves in the region. The location of the UAV can be compared with locations on the computer-based map and, based upon the comparison, the method can include transmitting guidance information to the UAV. In a further particular embodiment, two-dimensional imaging data is used in addition to the rangefinder data to provide color information to points in the region.

Claims

1. A method, performed by a computing system having a memory and a processor, for tracking a flight of an unmanned aerial vehicle in an environment using a three-dimensional model of the environment, the method comprising:

with a scanner system, comprising at least one camera that is remote from the unmanned aerial vehicle,

tracking the flight of the unmanned aerial vehicle in the environment based at least in part on;

identifying the unmanned aerial vehicle within at least one image captured by the at least one camera,

determining a position and/or orientation of the unmanned aerial vehicle based at least in part on at least one physical coding pattern associated with the unmanned aerial vehicle,

confirming an identity of the unmanned aerial vehicle based at least in part on the at least one physical coding pattern associated with the unmanned aerial vehicle, and

causing at least a rotation of the at least one camera so that the unmanned aerial vehicle remains within a field of view of the at least one camera, and

comparing the flight of the unmanned aerial vehicle in the environment to a flight path constructed in the three-dimensional model;

determining that the flight of the unmanned aerial vehicle has deviated from the flight path constructed in the three-dimensional model; and

in response to determining that the flight of the unmanned aerial vehicle has deviated from the flight path constructed in the three-dimensional model,

redirecting the flight of the unmanned aerial vehicle in accordance with the flight path constructed in the three-dimensional model at least in part by transmitting guidance information to the unmanned aerial vehicle.

2. The method of claim 1, wherein tracking the flight of the unmanned aerial vehicle in the environment comprises:

repeatedly capturing an image with at least one camera; and

analyzing the captured images to estimate the position and orientation of the unmanned aerial vehicle.

3. The method of claim 1, wherein tracking the flight of the unmanned aerial vehicle in the environment comprises:

repeatedly capturing an image with at least one camera;

analyzing the captured images to locate the unmanned aerial vehicle; and

determining or estimating a distance to the unmanned aerial vehicle with at least one rangefinder wherein the at least one rangefinder comprises at least one of;

a light ranging rangefinder, or

a radio frequency ranging rangefinder.

4. The method of claim 1, wherein tracking the flight of the unmanned aerial vehicle in the environment comprises:

receiving Automatic Dependent Surveillance—

Broadcast information from the unmanned aerial vehicle.

5. The method of claim 1, further comprising:

while tracking the unmanned aerial vehicle,

updating the three-dimensional model of the environment.

6. The method of claim 5, wherein updating the three-dimensional model of the environment comprises:

capturing, with the at least one camera, images of the environment;

identifying features within one or more of the captured images of the environment;

generating, with a rangefinder, depth data at least in part by determining the distance from the rangefinder to each of a plurality of the identified features;

transforming the generated depth data from spherical coordinates to Cartesian coordinates based at least in part on information collected from a tilt encoder and a pan encoder; and

generating, with the processor, an updated three-dimensional model of the environment based at least in part on the transformed depth data.

7. The method of claim 6, further comprising:

comparing the updated three-dimensional model of the environment to the three-dimensional model of the environment to identify new objects within the environment.

8. The method of claim 1, further comprising:

providing, for presentation to a user, information characterizing a visual representation of the three-dimensional model of the environment.

9. The method of claim 1, wherein comparing the actual flight of the unmanned aerial vehicle in the environment to the flight path constructed in the three-dimensional model comprises:

receiving speed and location information from the unmanned aerial vehicle; and

comparing the received speed and location information from the unmanned aerial vehicle to the flight path constructed in the three-dimensional model.

10. The method of claim 1, further comprising:

receiving, from a user, input related to a proposed flight path for the unmanned aerial vehicle through the environment; and

constructing the flight path in the three-dimensional model based at least in part on the received input and flight dynamics and capabilities of the unmanned aerial vehicle.

11. The method of claim 1, further comprising:

retrieving the three-dimensional model from a library of previously-generated three-dimensional models.

12. The method of claim 11, wherein at least one of the three-dimensional models of the library of previously-generated three-dimensional models is used to track the path of an unmanned vehicle through multiple environments.

13. The method of claim 1, wherein the at least one captured image of the unmanned aerial vehicle is captured by a camera attached to another unmanned aerial vehicle.

14. The method of claim 1, wherein the three-dimensional model of the environment is generated at least in part by:

capturing, with each of a plurality of scanners located at different locations, images of the environment and depth information associated with each of a plurality of features identified in the images, wherein each scanner comprises at least one camera and at least one rangefinder; and

based on the depth information, stitching together at least two of the images of the environment to form a composite three-dimensional model of the environment.

15. The method of claim 1, wherein tracking the flight of the unmanned aerial vehicle in the environment comprises:

comparing images taken by a camera attached to the unmanned aerial vehicle to the three-dimensional model of the environment.

16. The method of claim 1, further comprising:

receiving information corresponding to restrictions on a flight path for the unmanned aerial vehicle; and

providing feedback if the flight path constructed in the three-dimensional model violates the restrictions.

17. The method of claim 1, wherein the physical coding pattern includes at least one of an LED pattern or flash sequence.

18. The method of claim 1, wherein the physical coding pattern is passive.

19. The method of claim 1, wherein the physical coding pattern is actively transmitted.

20. A non-transitory computer-readable storage medium storing instructions that, if executed by a computing system having a processor, cause the computing system to perform a method for tracking a flight of an unmanned aerial vehicle in an environment using a three-dimensional model of the environment, the method comprising:

tracking the flight of the unmanned aerial vehicle in the environment based at least in part on;

identifying the unmanned aerial vehicle within at least one image captured by at least one camera remote from the unmanned aerial vehicle,

determining a position and/or orientation of the unmanned aerial vehicle based at least in part on at least one physical coding pattern associated with the unmanned aerial vehicle,

confirming an identity of the unmanned aerial vehicle based at least in part on the at least one physical coding pattern associated with the unmanned aerial vehicle, and

causing at least a rotation of the at least one camera so that the unmanned aerial vehicle remains within a field of view of the at least one camera, and

comparing the actual flight of the unmanned aerial vehicle in the environment to a flight path constructed in a three-dimensional model;

determining that the actual flight of the unmanned aerial vehicle has deviated from the flight path constructed in the three-dimensional model; and

in response to determining that the actual flight of the unmanned aerial vehicle has deviated from the flight path constructed in the three-dimensional model constructed in the three-dimensional model,

transmitting guidance information to the unmanned aerial vehicle.

21. The computer-readable storage medium of claim 20, wherein the tracking of the flight of the unmanned aerial vehicle in the environment is further based at least in part on position information received from the unmanned aerial vehicle.

22. A system comprising:

a scanner comprising;

a camera,

a rangefinder,

a tilt encoder, and

a pan encoder; and

one or more processors that, when executing instructions stored on at least a memory, are configured to;

generate a three-dimensional model of an environment based at least in part on;

information collected from the camera,

information collected from the rangefinder,

information collected from the tilt encoder, and

information collected from the pan encoder;

calculate a projected flight path for an unmanned aerial vehicle through three-dimensional model of the environment, wherein the scanner is remote from the unmanned aerial vehicle;

cause the unmanned aerial vehicle to begin a flight through the environment in accordance with the projected flight path;

track the flight of the unmanned aerial vehicle through the environment based at least in part on;

information collected from the camera,

information collected from the tilt encoder,

information collected from the pan encoder,

identifying the unmanned aerial vehicle within at least one image captured by the camera,

determining a position and/or orientation of the unmanned aerial vehicle based at least in part on at least one physical coding pattern associated with the unmanned aerial vehicle,

confirming an identity of the unmanned aerial vehicle based at least in part on the at least one physical coding pattern associated with the unmanned aerial vehicle, and

causing at least a rotation of the camera so that the unmanned aerial vehicle remains within a field of view of the camera, and

compare the tracked flight of the unmanned aerial vehicle through environment to the projected flight path of the unmanned aerial vehicle through the three-dimensional model of the environment.

23. The system of claim 22, wherein the one or more processors are further configured to:

receive an indication of a plurality of target features of the environment, wherein the camera captures only images of the environment that include one or more of the target features of the environment.