Unmanned aerial vehicle rooftop inspection system

• US 9,618,940 B1
• Filed: 03/11/2016
• Issued: 04/11/2017
• Est. Priority Date: 12/31/2015
• Status: Active Grant

• Current Assignee: Skydio, Inc.
• Original Assignee: Unmanned Innovation Inc (Delair SAS)
• Inventors: Michini, Bernard J., Li, Hui, Bethke, Brett Michael
• Primary Examiner(s): Badii, Behrang
• Assistant Examiner(s): Testardi, David
• Application Number:
  US15/068,292
• Time in Patent Office: 396 Days
• CPC Class Codes: B64C 39/024 of the remote controlled vehicle type, i.e. RPV
• CPC Class Codes: B64D 47/08 Arrangements of cameras
• CPC Class Codes: B64U 10/13 Flying platforms
• CPC Class Codes: B64U 2101/00 UAVs specially adapted for particular uses or applications
• CPC Class Codes: B64U 2101/26 for manufacturing, inspections or repairs
• CPC Class Codes: B64U 2101/30 for imaging, photography or videography
• CPC Class Codes: B64U 2201/10 autonomous, i.e. by navigating independently from ground or air stations, e.g. by using inertial navigation systems [INS]
• CPC Class Codes: B64U 2201/20 Remote controls
• CPC Class Codes: B64U 2201/202 using tethers for connecting to ground station
• CPC Class Codes: G01C 21/20 Instruments for performing navigational calculations G01C21/24, G01C21/26 take precedence
• CPC Class Codes: G05D 1/0016 characterised by the operator's input device input arrangements for computing systems in general G06F3/00
• CPC Class Codes: G05D 1/0038 by providing the operator with simple or augmented images from one or more cameras located onboard the vehicle, e.g. tele-operation images analyzed by a computer and used for automatic navigation G05D1/0246
• CPC Class Codes: G05D 1/0044 by providing the operator with a computer generated representation of the environment of the vehicle, e.g. virtual reality, maps maps used for automatic navigation G05D1/0274; flight directors G01C23/005
• CPC Class Codes: G05D 1/0094 involving pointing a payload, e.g. camera, weapon, sensor, towards a fixed or moving target
• CPC Class Codes: G05D 1/042 specially adapted for aircraft
• CPC Class Codes: G05D 1/0653 during a phase of take-off or landing
• CPC Class Codes: G06F 18/214 Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06F 3/048 Interaction techniques based on graphical user interfaces [GUI]
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06Q 10/063114 : Status monitoring or status determination for a person or group
• CPC Class Codes: G06Q 10/1097 : Task assignment
• CPC Class Codes: G06Q 50/16 : Real estate
• CPC Class Codes: G06T 17/05 : Geographic models
• CPC Class Codes: G06T 2215/16 : Using real world measurements to influence rendering
• CPC Class Codes: G06V 20/17 : taken from planes or by drones
• CPC Class Codes: G06V 20/176 : Urban or other man-made structures
• CPC Class Codes: G06V 20/58 : Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G06V 2201/06 : Recognition of objects for industrial automation
• CPC Class Codes: G08G 5/0013 : with a ground station
• CPC Class Codes: G08G 5/0034 : Assembly of a flight plan
• CPC Class Codes: G08G 5/0065 : for taking-off
• CPC Class Codes: G08G 5/0069 : specially adapted for an unmanned aircraft
• CPC Class Codes: G08G 5/025 : Navigation or guidance aids radar or analogous systems specially adapted for landing purposes G01S13/913
• CPC Class Codes: H04N 23/631 : Graphical user interfaces [GUI] specially adapted for controlling image capture or setting capture parameters
• CPC Class Codes: H04N 5/44504 : Circuit details of the additional information generator, e.g. details of the character or graphics signal generator, overlay mixing circuits
• CPC Class Codes: H04N 7/183 : for receiving images from a single remote source
• CPC Class Codes: H04N 7/185 : from a mobile camera, e.g. for remote control

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for an unmanned aerial system inspection system. One of the methods is performed by a UAV and includes receiving, by the UAV, flight information describing a job to perform an inspection of a rooftop. A particular altitude is ascended to, and an inspection of the rooftop is performed including obtaining sensor information describing the rooftop. Location information identifying a damaged area of the rooftop is received. The damaged area of the rooftop is traveled to. An inspection of the damaged area of the rooftop is performed including obtaining detailed sensor information describing the damaged area. A safe landing location is traveled to.

Claims

1. A computerized method performed by an unmanned aerial vehicle (UAV) system comprising one or more processors, the method comprising:

in initial steps;

receiving, by an unmanned aerial vehicle (UAV), flight information describing a job to perform an inspection of a rooftop, the flight information specifying a flight pattern according to which the UAV is to navigate over the rooftop during an inspection and is to make turns to change direction of flight of the UAV;

navigating, by the UAV, to at least a particular altitude;

in subsequent steps;

performing the inspection of the rooftop according to the flight pattern specified in the received flight information, the inspection including obtaining sensor information describing the rooftop;

determining, by the one or more processors executing a visual classifier trained to identify rooftop damage, that a particular area of the rooftop identified from the obtained sensor information is indicated as being damaged;

determining, by the one or more processors, a corresponding location associated with the particular area of the rooftop indicated as being damaged, and navigating the UAV to a position above the corresponding location associated with the particular area of the rooftop;

performing, by the UAV, a detailed inspection of the particular area of the rooftop indicated as being damaged, including obtaining detailed sensor information describing the particular area of the rooftop; and

navigating the UAV to a safe landing location.

2. The method of claim 1, wherein the particular altitude indicates an altitude that is free of obstructions within a property boundary indicated in the information describing the job.

3. The method of claim 1, wherein the UAV enforces a geofence envelope limiting the UAV to real-world locations within a property boundary indicated in the flight information describing the job.

4. The method of claim 1, wherein performing an inspection of the particular area comprises:

descending from the position above the corresponding location, and while descending, obtaining sensor information of the particular area;

determining that the UAV is within a threshold distance of the rooftop; and

ascending along a vertical direction from the corresponding location, and while ascending, obtaining sensor data of the particular area.

5. The method of claim 4, wherein the sensor information obtained during the inspection of the rooftop includes images of the rooftop, and wherein the method further comprises:

obtaining one or more images, obtained during the inspection of the rooftop, that include the particular area,

wherein descending comprises;

performing visual odometry using the obtained images and sensor information obtained as the UAV descends, and

correcting a flight path of the UAV, along a horizontal direction, while descending based on the performed visual odometry.

6. The method of claim 1, further comprising:

subsequent to traveling to the safe landing location, generating, by the UAV, a report for presentation on a user device of an operator, wherein the report includes a graphical representation of the rooftop and a location within the graphical representation of the damaged area, and wherein the report comprises an interactive document or a static document.

7. The method of claim 6, further comprising:

determining, using the visual classifier, one or more features indicative of rooftop damage that are included in detailed sensor information describing the damaged area; and

including, in the report, information describing the one or more features.

8. The method of claim 1, wherein the visual classifier determines rooftop damage based on one or more features included in sensor information that are indicative of rooftop damage, wherein a feature comprises:

cracks in ceramic/clay tiles, broken corners of tiles, gaps between shingles, curled edges of a rooftop, dents in a wooden rooftop, dents or divots in tiles, moss or algae on shingles, shingle splitting, surface nail damage, spatter marks, or nails raised above a surface of the rooftop.

9. The method of claim 1, wherein determining that a particular area of a rooftop is indicated as being damaged comprises:

obtaining sensor information describing the rooftop at a prior point in time;

comparing the sensor information associated with the prior point in time with the sensor information describing the rooftop obtained during the inspection of the rooftop; and

determining, using the visual classifier, that the particular area of the rooftop is indicated as being damaged based on the comparisons.

10. A non-transitory computer storage medium comprising instructions that when executed by one or more processors included in an Unmanned Aerial Vehicle (UAV), cause the UAV to perform operations comprising:

in initial steps;

receiving, by the UAV, flight information describing a job to perform an inspection of a rooftop, the flight information specifying a flight pattern according to which the UAV is to navigate over the rooftop during an inspection and is to make turns to change direction of flight of the UAV;

navigating, by the UAV, to at least a particular altitude;

in subsequent steps;

performing the inspection of the rooftop according to the flight pattern specified in the received flight information, the inspection including obtaining sensor information describing the rooftop;

determining, by the UAV executing a visual classifier trained to identify rooftop damage, that a particular area of the rooftop identified from the obtained sensor information is indicated as being damaged;

determining, by the UAV, a corresponding location associated with the particular area of the rooftop indicated as being damaged, and navigating the UAV to a position above the corresponding location associated with the particular area of the rooftop;

performing, by the UAV, a detailed inspection of the particular area of the rooftop indicated as being damaged, including obtaining detailed sensor information describing the particular area of the rooftop; and

navigating the UAV to a safe landing location.

11. The computer storage medium of claim 10, wherein the particular altitude indicates an altitude that is free of obstructions within a property boundary indicated in the information describing the job.

12. The computer storage medium of claim 10, wherein the UAV enforces a geofence envelope limiting the UAV to real-world locations within a property boundary indicated in the flight information describing the job.

13. The computer storage medium of claim 10, wherein performing an inspection of the particular area comprises:

descending from the position above the corresponding location, and while descending, obtaining sensor information of the particular area;

determining that the UAV is within a threshold distance of the rooftop; and

ascending along a vertical direction from the corresponding location, and while ascending, obtaining sensor data of the particular area.

14. The computer storage medium of claim 13, wherein the sensor information obtained during the inspection of the rooftop includes images of the rooftop, and wherein the operations further comprise:

obtaining one or more images, obtained during the inspection of the rooftop, that include the particular area,

wherein descending comprises;

performing visual odometry using the obtained images and sensor information obtained as the UAV descends, and

correcting a flight path of the UAV, along a horizontal direction, while descending based on the performed visual odometry.

15. The computer storage medium of claim 10, wherein the operations further comprise:

subsequent to traveling to the safe landing location, generating, by the UAV, a report for presentation on a user device of an operator, wherein the report includes a graphical representation of the rooftop and a location within the graphical representation of the damaged area, and wherein the report comprises an interactive document or a static document.

16. The computer storage medium of claim 15, wherein the operations further comprise:

determining, using the visual classifier, one or more features indicative of rooftop damage that are included in detailed sensor information describing the damaged area; and

including, in the report, information describing the one or more features.

17. The computer storage medium of claim 10, wherein the visual classifier determines rooftop damage based on one or more features included in sensor information that are indicative of rooftop damage, wherein a feature comprises:

cracks in ceramic/clay tiles, broken corners of tiles, gaps between shingles, curled edges of a rooftop, dents in a wooden rooftop, dents or divots in tiles, moss or algae on shingles, shingle splitting, surface nail damage, spatter marks, or nails raised above a surface of the rooftop.

18. The computer storage medium of claim 10, wherein determining that a particular area of a rooftop is indicated as being damaged comprises:

obtaining sensor information describing the rooftop at a prior point in time;

comparing the sensor information associated with the prior point in time with the sensor information describing the rooftop obtained during the inspection of the rooftop; and

determining, using the visual classifier, that the particular area of the rooftop is indicated as being damaged based on the comparisons.

19. A system included in an unmanned aerial vehicle (UAV) comprising one or more processors, and a non-transitory computer storage media storing instructions, that when executed by the one or more processors, cause the one or more processors to perform operations comprising:

in initial steps;

receiving flight information describing a job to perform an inspection of a rooftop, the flight information specifying a flight pattern according to which the UAV is to navigate over the rooftop during an inspection and is to make turns to change direction of flight of the UAV;

navigating, by the UAV, to at least a particular altitude;

in subsequent steps;

performing, by the UAV, the inspection of the rooftop according to the flight pattern specified in the received flight information, the inspection including obtaining sensor information describing the rooftop;

determining, using an executed visual classifier trained to identify rooftop damage, that a particular area of the rooftop identified from the obtained sensor information is indicated as being damaged;

determining a corresponding location associated with the particular area of the rooftop indicated as being damaged, and navigating the UAV to a position above the corresponding location associated with the particular area of the rooftop;

performing, by the UAV, a detailed inspection of the particular area of the rooftop indicated as being damaged, including obtaining detailed sensor information describing the particular area of the rooftop; and

navigating, by the UAV, to a safe landing location.

20. The system of claim 19, wherein the particular altitude indicates an altitude that is free of obstructions within a property boundary indicated in the information describing the job.

21. The system of claim 19, wherein the UAV enforces a geofence envelope limiting the UAV to real-world locations within a property boundary indicated in the flight information describing the job.

22. The system of claim 19, wherein performing an inspection of the particular area comprises:

descending, by the UAV, from the position above the corresponding location, and while descending, obtaining sensor information of the particular area;

determining that the UAV is within a threshold distance of the rooftop; and

ascending, by the UAV, along a vertical direction from the corresponding location, and while ascending, obtaining sensor data of the particular area.

23. The system of claim 22, wherein the sensor information obtained during the inspection of the rooftop includes images of the rooftop, and wherein the operations further comprise:

obtaining one or more images, obtained during the inspection of the rooftop, that include the particular area,

wherein descending comprises;

performing visual odometry using the obtained images and sensor information obtained as the UAV descends, and

correcting a flight path of the UAV, along a horizontal direction, while descending based on the performed visual odometry.

24. The system of claim 19, wherein the operations further comprise:

subsequent to traveling to the safe landing location, generating, by the UAV, a report for presentation on a user device of an operator, wherein the report includes a graphical representation of the rooftop and a location within the graphical representation of the damaged area, and wherein the report comprises an interactive document or a static document.

25. The system of claim 24, wherein the operations further comprise:

determining, using the visual classifier, one or more features indicative of rooftop damage that are included in detailed sensor information describing the damaged area; and

including, in the report, information describing the one or more features.

26. The system of claim 19, wherein the visual classifier determines rooftop damage based on one or more features included in sensor information that are indicative of rooftop damage, wherein a feature comprises:

cracks in ceramic/clay tiles, broken corners of tiles, gaps between shingles, curled edges of a rooftop, dents in a wooden rooftop, dents or divots in tiles, moss or algae on shingles, shingle splitting, surface nail damage, spatter marks, or nails raised above a surface of the rooftop.

27. The system of claim 19, wherein determining that a particular area of a rooftop is indicated as being damaged comprises:

obtaining sensor information describing the rooftop at a prior point in time;

comparing the sensor information associated with the prior point in time with the sensor information describing the rooftop obtained during the inspection of the rooftop; and

determining, using the visual classifier, that the particular area of the rooftop is indicated as being damaged based on the comparisons.

28. A method performed by an unmanned aerial vehicle (UAV) comprising one or more processors and one or more cameras, the method comprising:

obtaining, by the UAV, first sensor information describing a particular area of a rooftop, the sensor information captured at a particular altitude above the rooftop;

determining, by the one or more processors of the UAV executing a visual classifier trained to identify rooftop damage and using the obtained first sensor information, that the particular area of the rooftop is indicated as being damaged;

determining a corresponding location associated with the particular area of the rooftop indicated as being damaged, and navigating the UAV to a position above the corresponding location; and

descending, from the position, towards the particular area and obtaining second sensor information periodically while descending, wherein descending comprises;

performing visual odometry using the obtained first sensor information and second sensor information, and

correcting a flight path of the UAV, along a horizontal direction, while descending based on the performed visual odometry, such that the UAV maintains descent along a vertical direction from the position towards the particular area.

29. The method of claim 28, wherein correcting a flight path of the UAV while descending comprises:

determining, from the first sensor information, a center of the damage included in the particular area; and

correcting, using the second sensor information, the flight path along the horizontal direction to maintain the descent towards the center of the damage.

30. The method of claim 28, wherein the visual classifier determines rooftop damage based on one or more features included in sensor information that are indicative of rooftop damage, wherein a feature comprises:

cracks in ceramic/clay tiles, broken corners of tiles, gaps between shingles, curled edges of a rooftop, dents in a wooden rooftop, dents or divots in tiles, moss or algae on shingles, shingle splitting, surface nail damage, spatter marks, or nails raised above a surface of the rooftop.