Unmanned aerial vehicle rooftop inspection system

• US 9,881,213 B2
• Filed: 03/17/2017
• Issued: 01/30/2018
• Est. Priority Date: 12/31/2015
• Status: Active Grant

• Current Assignee: Skydio, Inc.
• Original Assignee: Unmanned Innovation Inc (Delair SAS)
• Inventors: Michini, Bernard J., Li, Hui, Bethke, Brett Michael
• Primary Examiner(s): Melton, Todd M
• Application Number:
  US15/462,577
• Publication Number:
  US 20170193297A1
• Time in Patent Office: 319 Days
• CPC Class Codes: B64C 39/024 of the remote controlled vehicle type, i.e. RPV
• CPC Class Codes: B64D 47/08 Arrangements of cameras
• CPC Class Codes: B64U 10/13 Flying platforms
• CPC Class Codes: B64U 2101/00 UAVs specially adapted for particular uses or applications
• CPC Class Codes: B64U 2101/26 for manufacturing, inspections or repairs
• CPC Class Codes: B64U 2101/30 for imaging, photography or videography
• CPC Class Codes: B64U 2201/10 autonomous, i.e. by navigating independently from ground or air stations, e.g. by using inertial navigation systems [INS]
• CPC Class Codes: B64U 2201/20 Remote controls
• CPC Class Codes: B64U 2201/202 using tethers for connecting to ground station
• CPC Class Codes: G01C 21/20 Instruments for performing navigational calculations G01C21/24, G01C21/26 take precedence
• CPC Class Codes: G05D 1/0016 characterised by the operator's input device input arrangements for computing systems in general G06F3/00
• CPC Class Codes: G05D 1/0038 by providing the operator with simple or augmented images from one or more cameras located onboard the vehicle, e.g. tele-operation images analyzed by a computer and used for automatic navigation G05D1/0246
• CPC Class Codes: G05D 1/0044 by providing the operator with a computer generated representation of the environment of the vehicle, e.g. virtual reality, maps maps used for automatic navigation G05D1/0274; flight directors G01C23/005
• CPC Class Codes: G05D 1/0094 involving pointing a payload, e.g. camera, weapon, sensor, towards a fixed or moving target
• CPC Class Codes: G05D 1/042 specially adapted for aircraft
• CPC Class Codes: G05D 1/0653 during a phase of take-off or landing
• CPC Class Codes: G06F 18/214 Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06F 3/048 Interaction techniques based on graphical user interfaces [GUI]
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06Q 10/063114 : Status monitoring or status determination for a person or group
• CPC Class Codes: G06Q 10/1097 : Task assignment
• CPC Class Codes: G06Q 50/16 : Real estate
• CPC Class Codes: G06T 17/05 : Geographic models
• CPC Class Codes: G06T 2215/16 : Using real world measurements to influence rendering
• CPC Class Codes: G06V 20/17 : taken from planes or by drones
• CPC Class Codes: G06V 20/176 : Urban or other man-made structures
• CPC Class Codes: G06V 20/58 : Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G06V 2201/06 : Recognition of objects for industrial automation
• CPC Class Codes: G08G 5/0013 : with a ground station
• CPC Class Codes: G08G 5/0034 : Assembly of a flight plan
• CPC Class Codes: G08G 5/0065 : for taking-off
• CPC Class Codes: G08G 5/0069 : specially adapted for an unmanned aircraft
• CPC Class Codes: G08G 5/025 : Navigation or guidance aids radar or analogous systems specially adapted for landing purposes G01S13/913
• CPC Class Codes: H04N 23/631 : Graphical user interfaces [GUI] specially adapted for controlling image capture or setting capture parameters
• CPC Class Codes: H04N 5/44504 : Circuit details of the additional information generator, e.g. details of the character or graphics signal generator, overlay mixing circuits
• CPC Class Codes: H04N 7/183 : for receiving images from a single remote source
• CPC Class Codes: H04N 7/185 : from a mobile camera, e.g. for remote control

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for an unmanned aerial system inspection system. One of the methods is performed by a UAV and includes receiving, by the UAV, flight information describing a job to perform an inspection of a rooftop. A particular altitude is ascended to, and an inspection of the rooftop is performed including obtaining sensor information describing the rooftop. Location information identifying a damaged area of the rooftop is received. The damaged area of the rooftop is traveled to. An inspection of the damaged area of the rooftop is performed including obtaining detailed sensor information describing the damaged area. A safe landing location is traveled to.

Claims

1. A computerized method performed by a system comprising one or more processors, the method comprising:

displaying, via an interface, a first graphical representation of a structure to be inspected, the first graphical representation comprising one or more images describing an aerial view of the structure;

determining a flight pattern according to which an unmanned aerial vehicle (UAV) is to navigate above the structure and obtain sensor data describing an aerial view of the structure, the flight pattern including waypoints;

instructing the UAV to perform the flight pattern and navigate the UAV according to the flight pattern, and obtain sensor data;

receiving sensor data obtained by the UAV during performance of the flight pattern, wherein the sensor data includes multiple digital images describing an aerial view depicting a rooftop of the structure, and wherein a subset of the digital images include greater than a threshold level of detail of particular areas of the rooftop;

generating a second graphical representation of the structure based on the multiple digital images, wherein the second graphical representation comprises composite imagery of the rooftop based on the multiple digital images;

determining the occurrence of damaged areas of the rooftop in the multiple digital images; and

displaying, via a user interface, the second graphical representation of the rooftop with damaged areas of the rooftop identified in the second graphical representation, wherein the damaged areas of the rooftop are represented by a graphical indication of the respective damaged areas, wherein the user interface is configured to receive selection of a damaged area identified in the second graphical representation and display one or more digital images that include greater than the threshold level of detail of the damaged area.

2. The method of claim 1, wherein determining the occurrence of damaged areas of the rooftop comprises:

executing a visual classifier trained to identify rooftop damage, that a particular area of the rooftop identified from the obtained sensor information is indicated as being damaged.

3. The method of claim 1, wherein determining whether surface damage has occurred about a particular area of the rooftop comprises:

determining whether a feature indicative of rooftop damage exists in a digital image; and

weighting the feature to determine that the feature is indicative of a damaged area or an undamaged area.

4. The method of claim 1, further comprising training a visual classifier based on a first dataset of images comprising examples of rooftop damage.

5. The method of claim 1, further comprising:

determining based on the aerial images, a material composition of the rooftop, wherein the material composition for the rooftop is one of wood, tile or clay.

6. The method of claim 1, wherein the type of rooftop damage identified is one of hail damage, cracked tiles, broken tiles, surface dents or organism growth.

7. The method of claim 1, further comprising:

receiving, via a user interface, positions about the second graphical representation of the rooftop indicating locations of damaged areas.

8. The method of claim 1, wherein the positions above the structure are set at an altitude such that a required image pixel resolution is obtained.

9. The method of claim 1, further comprising:

displaying, via the interface, a graphical representation of the flight pattern above the first graphical representation of the property.

10. The method of claim 1, further comprising:

determining the type of damage; and

assigning a description to the type of damage.

11. A system comprising one or more processors, and a computer storage media storing instructions, that when executed by the one or more processors, cause the one or more processors to perform operations comprising:

displaying, via an interface, a first graphical representation of a structure to be inspected, the first graphical representation comprising one or more images describing an aerial view of the structure;

determining a flight pattern according to which an unmanned aerial vehicle (UAV) is to navigate above the structure and obtain sensor data describing an aerial view of the structure, the flight pattern including waypoints;

instructing the UAV to perform the flight pattern and navigate the UAV according to the flight pattern, and obtain sensor data;

receiving sensor data obtained by the UAV during performance of the flight pattern, wherein the sensor data includes multiple digital images describing an aerial view depicting a rooftop of the structure, and wherein a subset of the digital images are detailed images;

generating a second graphical representation of the structure based on the multiple digital images, wherein the second graphical representation comprises composite imagery of the rooftop based on the multiple digital images;

determining the occurrence of damaged areas of the rooftop in the multiple digital images; and

displaying, via a user interface, the second graphical representation of the rooftop with damaged areas of the rooftop identified in the second graphical representation, wherein the damaged areas of the rooftop are represented by a graphical indication of the respective damaged areas, wherein the user interface is configured to receive selection of a damaged area and display one or more detailed images of the damaged area.

12. The system of claim 11, wherein determining the occurrence of damaged areas of the rooftop comprises:

executing a visual classifier trained to identify rooftop damage, that a particular area of the rooftop is indicated as being damaged.

13. The system of claim 12, further comprising the operations of:

training the visual classifier based on a first dataset of images comprising examples of rooftop damage.

14. The system of claim 11, wherein determining the occurrence of damaged areas of the rooftop comprises:

determining whether a feature indicative of rooftop damage exists in a digital image; and

weighting the feature to determine that the feature is indicative of a damaged area or an undamaged area.

15. The system of claim 11, further comprising the operations of:

determining based on the aerial images, a material composition of the rooftop, wherein the material composition for the rooftop is one of wood, tile or clay.

16. The system of claim 11, wherein the type of rooftop damage identified is one of hail damage, cracked tiles, broken tiles, surface dents or organism growth.

17. The system of claim 11, further comprising the operations of:

receiving, via a user interface, positions about the second graphical representation of the rooftop indicating locations of damaged areas of the rooftop.

18. The system of claim 11, wherein waypoints above the structure are set at an altitude such that a required image pixel resolution for the multiple digital images of the rooftop is obtained.

19. The system of claim 11, further comprising the operations of:

displaying, via the interface, a graphical representation of the flight pattern above the first graphical representation of the property.

20. The system of claim 11, further comprising the operations of:

determining the type of damage; and

assigning a description to the type of damage.

21. A computer storage medium comprising instructions that when executed by one or more processors, cause the processors to perform operations comprising:

displaying, via an interface, a first graphical representation of a structure to be inspected, the first graphical representation comprising one or more images describing an aerial view of the structure;

determining a flight pattern according to which an unmanned aerial vehicle (UAV) is to navigate above the structure and obtain sensor data describing an aerial view of the structure, the flight pattern including waypoints;

instructing the UAV to perform the flight pattern and navigate the UAV according to the flight pattern, and obtain sensor data;

receiving sensor data obtained by the UAV during performance of the flight pattern, wherein the sensor data includes multiple digital images describing an aerial view depicting a rooftop of the structure, and wherein a subset of the digital images include greater than a threshold level of detail of particular areas of the rooftop;

generating a second graphical representation of the structure based on the multiple digital images, wherein the second graphical representation comprises composite imagery of the rooftop based on the multiple digital images;

determining the occurrence of damaged areas of the rooftop in the multiple digital images; and

displaying, via a user interface, the second graphical representation of the rooftop with damaged areas of the rooftop identified in the second graphical representation, wherein the damaged areas of the rooftop are represented by a graphical indication of the respective damaged areas, wherein the user interface is configured to receive selection of a damaged area and display one or more digital images that include greater than the threshold level of detail of the damaged area.

22. The computer storage medium of claim 21, wherein determining the occurrence of damaged areas of the rooftop comprises:

executing a visual classifier trained to identify rooftop damage, that a particular area of the rooftop is indicated as being damaged.

23. The computer storage medium of claim 22, further comprising the operations of:

training a visual classifier based on a first dataset of images comprising examples of rooftop damage.

24. The computer storage medium of claim 21, wherein determining the occurrence of damaged areas of the rooftop comprises:

determining whether a feature indicative of rooftop damage exists in a digital image; and

weighting the feature to determine that the feature is indicative of a damaged area or an undamaged area.

25. The computer storage medium of claim 21, further comprising the operations of:

determining based on the aerial images, a material composition of the rooftop, wherein the material composition for the rooftop is one of wood, tile or clay.

26. The computer storage medium of claim 21, wherein the type of rooftop damage identified is one of hail damage, cracked tiles, broken tiles, surface dents or organism growth.

27. The computer storage medium of claim 21, further comprising the operations of:

receiving, via a user interface, positions about the second graphical representation of the rooftop indicating locations of damaged areas of the rooftop.

28. The computer storage medium of claim 21, wherein waypoints above the structure are set at an altitude such that a required image pixel resolution for the multiple digital images of the rooftop is obtained.

29. The computer storage medium of claim 21, further comprising the operations of:

displaying, via the interface, a graphical representation of the flight pattern above the first graphical representation of the property.

30. The computer storage medium of claim 21, further comprising the operations of:

determining the type of damage; and

assigning a description to the type of damage.