Scanning environments and tracking unmanned aerial vehicles

• US 10,162,353 B2
• Filed: 03/02/2016
• Issued: 12/25/2018
• Est. Priority Date: 03/03/2015
• Status: Active Grant

• Current Assignee: Prenav, Inc.
• Original Assignee: Prenav, Inc.
• Inventors: Hammond, Asa, Schuett, Nathan, Busek, Naimisaranya Das
• Primary Examiner(s): Anwari, Maceeh
• Application Number:
  US15/059,069
• Publication Number:
  US 20160291593A1
• Time in Patent Office: 1,028 Days
• Field of Search: 701 3
• CPC Class Codes: B64U 10/13 Flying platforms
• CPC Class Codes: B64U 2101/30 for imaging, photography or videography
• CPC Class Codes: B64U 2201/20 Remote controls
• CPC Class Codes: F03D 17/00 Monitoring or testing of wind motors, e.g. diagnostics testing during commissioning of wind motors F03D13/30
• CPC Class Codes: G01C 21/3415 Dynamic re-routing, e.g. recalculating the route when the user deviates from calculated route or after detecting real-time traffic data or accidents
• CPC Class Codes: G01S 17/66 Tracking systems using electromagnetic waves other than radio waves
• CPC Class Codes: G01S 17/86 Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
• CPC Class Codes: G01S 17/87 Combinations of systems using electromagnetic waves other than radio waves
• CPC Class Codes: G01S 17/89 for mapping or imaging
• CPC Class Codes: G01S 17/933 of aircraft or spacecraft
• CPC Class Codes: G01S 7/4813 Housing arrangements
• CPC Class Codes: G05D 1/0033 by having the operator tracking the vehicle either by direct line of sight or via one or more cameras located remotely from the vehicle
• CPC Class Codes: G05D 1/0044 by providing the operator with a computer generated representation of the environment of the vehicle, e.g. virtual reality, maps maps used for automatic navigation G05D1/0274; flight directors G01C23/005
• CPC Class Codes: G05D 1/0094 involving pointing a payload, e.g. camera, weapon, sensor, towards a fixed or moving target
• CPC Class Codes: G05D 1/102 specially adapted for vertical take-off of aircraft
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06F 3/0484 for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
• CPC Class Codes: G06T 17/05 Geographic models
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/10032 Satellite or aerial image; Remote sensing
• CPC Class Codes: G06T 2207/20221 : Image fusion; Image merging
• CPC Class Codes: G06T 2207/30181 : Earth observation
• CPC Class Codes: G06T 2207/30241 : Trajectory
• CPC Class Codes: G06T 2207/30252 : Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/0004 : Industrial image inspection
• CPC Class Codes: G06T 7/20 : Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/60 : Analysis of geometric attributes
• CPC Class Codes: G06V 20/653 : by matching three-dimensional models, e.g. conformal mapping of Riemann surfaces
• CPC Class Codes: G08G 5/0069 : specially adapted for an unmanned aircraft
• CPC Class Codes: G08G 5/045 : Navigation or guidance aids, e.g. determination of anti-collision manoeuvers radar or analogous systems specially adapted for anti-collision between aircraft G01S13/933
• CPC Class Codes: H04N 13/128 : Adjusting depth or disparity
• CPC Class Codes: H04N 2013/0081 : Depth or disparity estimation from stereoscopic image signals

Abstract

Systems and methods for scanning environments and tracking unmanned aerial vehicles within the scanned environments are disclosed. A method in accordance with a particular embodiment includes using a rangefinder off-board an unmanned air vehicle (UAV) to identify points in a region. The method can further include forming a computer-based map of the region with the points and using the rangefinder and a camera to locate the UAV as it moves in the region. The location of the UAV can be compared with locations on the computer-based map and, based upon the comparison, the method can include transmitting guidance information to the UAV. In a further particular embodiment, two-dimensional imaging data is used in addition to the rangefinder data to provide color information to points in the region.

Claims

1. A method, performed by a computing system having a memory and a processor, for planning a flight of an unmanned aerial vehicle in an environment, using a three-dimensional model of the environment, the method comprising:

providing, for presentation to a user, information characterizing a visual representation of the three-dimensional model of the environment, the environment including a plurality of objects;

based on an input from the user, shifting a viewpoint from which the user views the visual representation of the three-dimensional model of the environment;

receiving, from the user, input related to a proposed flight path for the unmanned aerial vehicle through the environment, wherein the input indicates one or more regions of interest;

calculating a predicted trajectory of the unmanned aerial vehicle through the environment based at least in part on the received input, one or more constraints or capabilities for pointing a sensor of the unmanned aerial vehicle toward the one or more regions of interest, and flight dynamics and capabilities of the unmanned aerial vehicle;

determining whether the calculated predicted trajectory of the unmanned aerial vehicle through the environment is free of collisions;

if the calculated predicted trajectory is determined to be not free of collisions;

providing feedback to the user that the calculated predicted trajectory of the unmanned aerial vehicle is not free of collisions; and

preventing an area causing a collision from being included in subsequent calculation of predicted trajectory of the unmanned aerial vehicle; and

causing the unmanned aerial vehicle to perform at least a portion of a flight based at least in part on the predicted trajectory.

2. The method of claim 1, further comprising:

generating the three-dimensional model of the environment at least in part by scanning the environment with a camera and a rangefinder.

3. The method of claim 1, further comprising:

importing the three-dimensional model of the environment.

4. The method of claim 1, further comprising:

generating the three-dimensional model of the environment at least in part by extracting spatial information from images captured by the unmanned aerial vehicle or another unmanned aerial vehicle.

5. The method of claim 1, wherein the received input related to the preferred flight path for the unmanned aerial vehicle comprises:

for an individual region of interest, a distance away from the region of interest that the unmanned aerial vehicle is to fly.

6. The method of claim 1, wherein checking to determine whether the calculated predicted trajectory of the unmanned aerial vehicle through the environment is free of collisions comprises:

comparing the three-dimensional model of the environment to the predicted trajectory of the unmanned aerial vehicle through the environment to determine whether the unmanned aerial vehicle would make contact with an object in the environment.

7. The method of claim 1, further comprising:

if the calculated predicted trajectory is determined to be not free of collisions;

identifying an interfering portion of the environment, and

wherein providing feedback to the user that calculated predicted trajectory of the unmanned aerial vehicle is not free of collisions comprises displaying the interfering portion of the environment in a different color from non-interfering portions of the environment.

8. The method of claim 1, further comprising:

if the calculated predicted trajectory is determined to be not free of collisions;

modifying the information characterizing the visual representation of the three-dimensional model of the environment to alter the display of the area causing a collision.

9. The method of claim 1, further comprising:

if the calculated predicted trajectory is determined to be not free of collisions,

modifying the predicted trajectory of the unmanned aerial vehicle to ensure that the unmanned aerial vehicle will follow a path that is free of collisions.

10. The method of claim 1, wherein the received input related to the proposed flight path for the unmanned aerial vehicle through the environment includes proximity constraints and a speed for the unmanned aerial vehicle.

11. The method of claim 1, further comprising:

generating the three-dimensional model of the environment based at least in part on information collected from the camera, rangefinder, tilt encoder, and pan encoder at least in part by;

capturing images of the environment with the camera;

generating depth data for objects within the captured images at least in part by determining or estimating the distance from the rangefinder to each of a plurality of the objects; and

transforming the generated depth data to Cartesian coordinates based at least in part on information collected from the tilt encoder and the pan encoder.

12. The method of claim 1, wherein causing the unmanned aerial vehicle to perform at least a portion of a flight includes

causing the unmanned aerial vehicle to begin a flight in accordance with the predicted trajectory of the unmanned aerial vehicle through the environment;

and wherein the method further comprises;

tracking the flight of the unmanned aerial vehicle through the environment with a first scanner.

13. The method of claim 12, wherein tracking the flight of the unmanned aerial vehicle through the environment comprises combining corner cube interferometry with output from a laser-based position sensing device to generate position estimates for the unmanned aerial vehicle.

14. The method of claim 12, further comprising:

handing off responsibility for tracking the unmanned aerial vehicle from the first scanner to a second scanner while the unmanned aerial vehicle is in flight.

15. The method of claim 12, further comprising:

while tracking the flight of the unmanned aerial vehicle through the environment, comparing the flight of the unmanned aerial vehicle through the environment to the predicted trajectory of the unmanned aerial vehicle through the environment.

16. The method of claim 15, further comprising:

adjusting the flight of the unmanned aerial vehicle in response to determining, based on the comparing, that the unmanned aerial vehicle has deviated from the predicted trajectory of the unmanned aerial vehicle through the environment.

17. The method of claim 1, wherein preventing an area causing a collision from being included in subsequent calculation of predicted trajectory of the unmanned aerial vehicle comprises calculating the predicted trajectory based only on areas that do not cause collisions.

18. The method of claim 1, further comprising:

if the calculated predicted trajectory is determined to be not free of collisions;

distinguishing, in presentation to the user, at least one portion of the predicted trajectory interfered with a collision from at least one portion of the predicted trajectory free of collisions.

19. A non-transitory computer-readable storage medium storing instructions that, if executed by a computing system having a memory and a processor, cause the computing system to perform a method, the method comprising:

providing, for presentation to a user, information characterizing a visual representation of a three-dimensional model of the environment, the environment including a plurality of objects;

receiving, from the user, input related to a proposed flight path for an unmanned aerial vehicle through the environment, wherein the input indicates one or more regions of interest;

calculating a predicted trajectory of the unmanned aerial vehicle through the environment based at least in part on the received input, one or more constraints or capabilities for pointing a sensor of the unmanned aerial vehicle toward the one or more regions of interest, and flight dynamics and capabilities of the unmanned aerial vehicle;

determining whether the calculated predicted trajectory of the unmanned aerial vehicle through the environment is free of collisions;

if the calculated predicted trajectory is determined to be not free of collisions;

providing feedback to the user that the calculated predicted trajectory of the unmanned aerial vehicle is not free of collisions; and

preventing an area causing a collision from being included in subsequent calculation of predicted trajectory of the unmanned aerial vehicle; and

causing the unmanned aerial vehicle to perform at least a portion of a flight based at least in part on the predicted trajectory.

20. The computer-readable medium of claim 19, the method further comprising:

based on an input from the user, shifting a viewpoint from which the user views the visual representation of the three-dimensional model of the environment.

21. A system comprising:

a camera;

a rangefinder;

a tilt encoder;

a pan encoder; and

one or more processors that, when executing instructions stored on at least a memory, are configured to;

generate a three-dimensional model of an environment based at least in part on information collected from the camera, information collected from the rangefinder, information collected from the tilt encoder, and information collected from the pan encoder,

provide, for presentation to a user, information characterizing a visual representation of the three-dimensional model of the environment, the environment including a plurality of objects,

receive, from the user, input related to a proposed flight path for an unmanned aerial vehicle through the environment, wherein the input indicates one or more regions of interest,

calculate a predicted trajectory of the unmanned aerial vehicle through the environment based at least in part on the received input, one or more constraints or capabilities for pointing a sensor of the unmanned aerial vehicle toward the one or more regions of interest, and flight dynamics and capabilities of the unmanned aerial vehicle,

determine whether the calculated predicted trajectory of the unmanned aerial vehicle through the environment is free of collisions,

if the calculated predicted trajectory is determined to be not free of collisions;

provide feedback to the user that the calculated predicted trajectory of the unmanned aerial vehicle is not free of collisions,

prevent an area causing a collision from being included in subsequent calculation of predicted trajectory of the unmanned aerial vehicle, and

prevent the unmanned aerial vehicle from performing a flight corresponding to the predicted trajectory until the area causing a collision is removed, and

cause the unmanned aerial vehicle to perform at least a portion of a flight based at least in part on the predicted trajectory.

22. The system of claim 21, wherein information collected from the camera and information collected from the rangefinder comprise information collected from each of a plurality of locations around the environment.