Unmanned aerial vehicle rooftop inspection system

• US 9,613,538 B1
• Filed: 03/11/2016
• Issued: 04/04/2017
• Est. Priority Date: 12/31/2015
• Status: Active Grant

• Current Assignee: Skydio, Inc.
• Original Assignee: Unmanned Innovation Inc (Delair SAS)
• Inventors: Kuehn, Justin Eugene, Poole, Alan Jay, Bauer, Mark Patrick, Gurel, Volkan, Michini, Bernard J.
• Primary Examiner(s): Holwerda, Stephen
• Application Number:
  US15/068,255
• Time in Patent Office: 389 Days
• Field of Search: 701 2, 701 3, 701 4, 701 5, 701 6, 701 7, 701 11, 700257, 700258, 700259
• CPC Class Codes: B64U 2101/26 for manufacturing, inspections or repairs
• CPC Class Codes: B64U 2101/30 for imaging, photography or videography
• CPC Class Codes: B64U 2201/10 autonomous, i.e. by navigating independently from ground or air stations, e.g. by using inertial navigation systems [INS]
• CPC Class Codes: B64U 2201/20 Remote controls
• CPC Class Codes: B64U 2201/202 using tethers for connecting to ground station
• CPC Class Codes: G01C 21/20 Instruments for performing navigational calculations G01C21/24, G01C21/26 take precedence
• CPC Class Codes: G05D 1/0016 characterised by the operator's input device input arrangements for computing systems in general G06F3/00
• CPC Class Codes: G05D 1/0038 by providing the operator with simple or augmented images from one or more cameras located onboard the vehicle, e.g. tele-operation images analyzed by a computer and used for automatic navigation G05D1/0246
• CPC Class Codes: G05D 1/0044 by providing the operator with a computer generated representation of the environment of the vehicle, e.g. virtual reality, maps maps used for automatic navigation G05D1/0274; flight directors G01C23/005
• CPC Class Codes: G05D 1/0094 involving pointing a payload, e.g. camera, weapon, sensor, towards a fixed or moving target
• CPC Class Codes: G05D 1/042 specially adapted for aircraft
• CPC Class Codes: G05D 1/0653 during a phase of take-off or landing
• CPC Class Codes: G06F 18/214 Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06F 3/048 Interaction techniques based on graphical user interfaces [GUI]
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06Q 10/063114 Status monitoring or status determination for a person or group
• CPC Class Codes: G06Q 10/1097 Task assignment
• CPC Class Codes: G06Q 50/16 Real estate
• CPC Class Codes: G06T 17/05 Geographic models
• CPC Class Codes: G06T 2215/16 : Using real world measurements to influence rendering
• CPC Class Codes: G06V 20/17 : taken from planes or by drones
• CPC Class Codes: G06V 20/176 : Urban or other man-made structures
• CPC Class Codes: G06V 20/58 : Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G06V 2201/06 : Recognition of objects for industrial automation
• CPC Class Codes: G08G 5/0013 : with a ground station
• CPC Class Codes: G08G 5/0034 : Assembly of a flight plan
• CPC Class Codes: G08G 5/0065 : for taking-off
• CPC Class Codes: G08G 5/0069 : specially adapted for an unmanned aircraft
• CPC Class Codes: G08G 5/025 : Navigation or guidance aids radar or analogous systems specially adapted for landing purposes G01S13/913
• CPC Class Codes: H04N 23/631 : Graphical user interfaces [GUI] specially adapted for controlling image capture or setting capture parameters
• CPC Class Codes: H04N 5/44504 : Circuit details of the additional information generator, e.g. details of the character or graphics signal generator, overlay mixing circuits
• CPC Class Codes: H04N 7/183 : for receiving images from a single remote source
• CPC Class Codes: H04N 7/185 : from a mobile camera, e.g. for remote control

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for an unmanned aerial system inspection system. One of the methods is performed by a UAV and includes receiving, by the UAV, flight information describing a job to perform an inspection of a rooftop. A particular altitude is ascended to, and an inspection of the rooftop is performed including obtaining sensor information describing the rooftop. Location information identifying a damaged area of the rooftop is received. The damaged area of the rooftop is traveled to. An inspection of the damaged area of the rooftop is performed including obtaining detailed sensor information describing the damaged area. A safe landing location is traveled to.

Claims

1. A flight planning system comprising one or more processors, and a computer storage media storing instructions, that when executed by the one or more processors, cause the one or more processors to perform operations, wherein the operations are performed pre-flight of an unmanned aerial vehicle (UAV), and wherein the operations comprise:

receiving, via a user interface, a location for an aerial survey of a property to be conducted by an unmanned aerial vehicle (UAV);

displaying, via the user interface, one or more images depicting a view of the location;

accessing information describing a property boundary associated with the property;

determining a geofence boundary based on the accessed information describing the property boundary, wherein the geofence boundary represents a geospatial boundary in which to limit flight of the UAV;

determining a flight pattern for an area within the property boundary for inspection by the UAV, the flight pattern having flight waypoints, each flight waypoint indicating a location where the UAV will obtain sensor information describing the location;

determining a flight plan having a take-off location, a landing location, and the flight pattern, wherein the take-off location, the landing location and the flight pattern are set within the geofence boundary, wherein the user interface is configured to receive modifications to one or more of the take-off location, landing location, or flight pattern; and

generating a flight data package, including the flight plan, for use by the UAV for navigation during the aerial survey.

2. The flight planning system of claim 1, wherein the take-off location is determined to be located in an area identified, by the flight planning system, as clear of obstructions.

3. The flight planning system of claim 1, wherein the operations further comprise:

identifying, using image analysis of the one or more images, one or more features included in the images that are indicative of a suitable location for a take off location, and designating a particular suitable location as the take off location.

4. The flight planning system of claim 3, wherein a suitable location includes a sidewalk, driveway, or roadway.

5. The flight planning system of claim 1, wherein the operations further comprise:

identifying, using image analysis of the one or more images, one or more features included in the images that are indicative of a non-suitable location for a take off location, and preventing the launch location to be designated as a non-suitable location.

6. The flight planning system of claim 5, wherein a non-suitable location includes a tree, pool, or a location within a threshold distance of a power line.

7. The flight planning system of claim 5, wherein preventing the take-off location to be designated as a non suitable location comprises:

receiving, via the user interface, input specifying a selection of a location in the one or more images to be designated as a take off location;

determining, that the selected location is a non suitable location; and

providing, for presentation in the user interface, information specifying that the selected location cannot be selected.

8. The flight planning system of claim 1, wherein the operations further comprise determining, based at least in part on colors or hue of a first object depicted in the one or more images, that a location of the first object in the images is a take-off location.

9. The flight planning system of claim 8, wherein the first object is determined to be a drive way based at least in part on the first object being depicted as substantially gray.

10. The flight planning system of claim 1, wherein the operations further comprise:

detecting one or more building footprints of the property using image analysis of the one or more images, and determining the flight waypoints using the detected building footprints.

11. A non-transitory computer storage medium comprising instructions that when executed by a system of one or more computers, cause the system to perform operations, wherein the operations are performed pre-flight of an unmanned aerial vehicle (UAV), and wherein the operations comprise:

receiving, via a user interface, a location for an aerial survey of a property to be conducted by an unmanned aerial vehicle (UAV);

displaying, via the user interface, one or more images depicting a view of the location;

accessing information describing a property boundary associated with the property;

determining a geofence boundary based on the accessed information describing the property boundary, wherein the geofence boundary represents a geospatial boundary in which to limit flight of the UAV;

determining a flight pattern for an area within the property boundary for inspection by the UAV, the flight pattern having flight waypoints, each flight waypoint indicating a location where the UAV will obtain sensor information describing the location;

determining a flight plan having a take-off location, a landing location, and the flight pattern, wherein the take-off location, the landing location and the flight pattern are set within the geofence boundary, wherein the user interface is configured to receive modifications to one or more of the take-off location, landing location, or flight pattern; and

generating a flight data package, including the flight plan, for use by the UAV for navigation during the aerial survey.

12. The computer storage medium of claim 11, wherein the take-off location is determined to be located in an area identified, by the flight planning system, as clear of obstructions.

13. The computer storage medium of claim 11, wherein the operations further comprise:

identifying, using image analysis of the one or more images, one or more features included in the images that are indicative of a suitable location for a take off location, and designating a particular suitable location as the take off location.

14. The computer storage medium of claim 13, wherein a suitable location includes a sidewalk, driveway, or roadway.

15. The computer storage medium of claim 11, wherein the operations further comprise:

identifying, using image analysis of the one or more images, one or more features included in the images that are indicative of a non-suitable location for a take off location, and preventing the launch location to be designated as a non-suitable location.

16. The computer storage medium of claim 15, wherein a non-suitable location includes a tree, pool, or a location within a threshold distance of a power line.

17. The computer storage medium of claim 15, wherein preventing the take-off location to be designated as a non suitable location comprises:

receiving, via the user interface, input specifying a selection of a location in the one or more images to be designated as a take off location;

determining, that the selected location is a non suitable location; and

providing, for presentation in the user interface, information specifying that the selected location cannot be selected.

18. The computer storage medium of claim 11, wherein the operations further comprise determining, based at least in part on colors or hue of a first object depicted in the one or more images, that a location of the first object in the images is a take-off location.

19. The computer storage medium of claim 18, wherein the first object is determined to be a drive way based at least in part on the first object being depicted as substantially gray.

20. The computer storage medium of claim 11, wherein the operations further comprise:

detecting one or more building footprints of the property using image analysis of the one or more images, and determining the flight waypoints using the detected building footprints.

21. A computerized method performed by a system of one or more computers, the method being performed pre-flight of an unmanned aerial vehicle (UAV), wherein the method comprises:

receiving, via a user interface, a location for an aerial survey of a property to be conducted by an unmanned aerial vehicle (UAV);

displaying, via the user interface, one or more images depicting a view of the location;

accessing information describing a property boundary associated with the property;

determining a geofence boundary based on the accessed information describing the property boundary, wherein the geofence boundary represents a geospatial boundary in which to limit flight of the UAV;

determining a flight pattern for an area within the property boundary for inspection by the UAV, the flight pattern having flight waypoints, each flight waypoint indicating a location where the UAV will obtain sensor information describing the location;

determining a flight plan having a take-off location, a landing location, and the flight pattern, wherein the take-off location, the landing location and the flight pattern are set within the geofence boundary, wherein the user interface is configured to receive modifications to one or more of the take-off location, landing location, or flight pattern; and

generating a flight data package, including the flight plan, for use by the UAV for navigation during the aerial survey.

22. The computerized method of claim 21 further comprising:

identifying, using image analysis of the one or more images, one or more features included in the images that are indicative of a suitable location for a take off location, and designating a particular suitable location as the take off location.

23. The computerized method of claim 22, wherein a suitable location includes a sidewalk, driveway, or roadway.

24. The computerized method of claim 21 further comprising:

identifying, using image analysis of the one or more images, one or more features included in the images that are indicative of a non-suitable location for a take off location, and preventing the launch location to be designated as a non-suitable location.

25. The computerized method of claim 24, wherein a non-suitable location includes a tree, pool, or a location within a threshold distance of a power line.

26. The computerized method of claim 24, wherein preventing the take-off location to be designated as a non suitable location comprises:

receiving, via the user interface, input specifying a selection of a location in the one or more images to be designated as a take off location;

determining, that the selected location is a non suitable location; and

providing, for presentation in the user interface, information specifying that the selected location cannot be selected.

27. The computerized method of claim 21 further comprising determining, based at least in part on colors or hue of a first object depicted in the one or more images, that a location of the first object in the images is a take-off location.

28. The computerized method of claim 27, wherein the first object is determined to be a drive way based at least in part on the first object being depicted as substantially gray.

29. The computerized method of claim 21 further comprising:

detecting one or more building footprints of the property using image analysis of the one or more images, and determining the flight waypoints using the detected building footprints.

30. The computerized method of claim 21, wherein subsequent to determination of the geofence boundary, the geofence boundary is fixed during the aerial survey such that the geofence boundary is not modifiable in the user interface.