Unmanned aerial vehicle rooftop inspection system

• US 10,083,616 B2
• Filed: 03/11/2016
• Issued: 09/25/2018
• Est. Priority Date: 12/31/2015
• Status: Active Grant

• Current Assignee: Skydio, Inc.
• Original Assignee: Unmanned Innovation Inc (Delair SAS)
• Inventors: Bauer, Mark Patrick, Richman, Brian, Poole, Alan Jay, Michini, Bernard J., Lovegren, Jonathan Anders, Bethke, Brett Michael, Li, Hui
• Primary Examiner(s): Cass, Jean Paul
• Application Number:
  US15/068,272
• Publication Number:
  US 20170193829A1
• Time in Patent Office: 928 Days
• Field of Search: 701 5
• CPC Class Codes: B64C 39/024 of the remote controlled vehicle type, i.e. RPV
• CPC Class Codes: B64D 47/08 Arrangements of cameras
• CPC Class Codes: B64U 10/13 Flying platforms
• CPC Class Codes: B64U 2101/00 UAVs specially adapted for particular uses or applications
• CPC Class Codes: B64U 2101/26 for manufacturing, inspections or repairs
• CPC Class Codes: B64U 2101/30 for imaging, photography or videography
• CPC Class Codes: B64U 2201/10 autonomous, i.e. by navigating independently from ground or air stations, e.g. by using inertial navigation systems [INS]
• CPC Class Codes: B64U 2201/20 Remote controls
• CPC Class Codes: B64U 2201/202 using tethers for connecting to ground station
• CPC Class Codes: G01C 21/20 Instruments for performing navigational calculations G01C21/24, G01C21/26 take precedence
• CPC Class Codes: G05D 1/0016 characterised by the operator's input device input arrangements for computing systems in general G06F3/00
• CPC Class Codes: G05D 1/0038 by providing the operator with simple or augmented images from one or more cameras located onboard the vehicle, e.g. tele-operation images analyzed by a computer and used for automatic navigation G05D1/0246
• CPC Class Codes: G05D 1/0044 by providing the operator with a computer generated representation of the environment of the vehicle, e.g. virtual reality, maps maps used for automatic navigation G05D1/0274; flight directors G01C23/005
• CPC Class Codes: G05D 1/0094 involving pointing a payload, e.g. camera, weapon, sensor, towards a fixed or moving target
• CPC Class Codes: G05D 1/042 specially adapted for aircraft
• CPC Class Codes: G05D 1/0653 during a phase of take-off or landing
• CPC Class Codes: G06F 18/214 Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06F 3/048 Interaction techniques based on graphical user interfaces [GUI]
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06Q 10/063114 : Status monitoring or status determination for a person or group
• CPC Class Codes: G06Q 10/1097 : Task assignment
• CPC Class Codes: G06Q 50/16 : Real estate
• CPC Class Codes: G06T 17/05 : Geographic models
• CPC Class Codes: G06T 2215/16 : Using real world measurements to influence rendering
• CPC Class Codes: G06V 20/17 : taken from planes or by drones
• CPC Class Codes: G06V 20/176 : Urban or other man-made structures
• CPC Class Codes: G06V 20/58 : Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G06V 2201/06 : Recognition of objects for industrial automation
• CPC Class Codes: G08G 5/0013 : with a ground station
• CPC Class Codes: G08G 5/0034 : Assembly of a flight plan
• CPC Class Codes: G08G 5/0065 : for taking-off
• CPC Class Codes: G08G 5/0069 : specially adapted for an unmanned aircraft
• CPC Class Codes: G08G 5/025 : Navigation or guidance aids radar or analogous systems specially adapted for landing purposes G01S13/913
• CPC Class Codes: H04N 23/631 : Graphical user interfaces [GUI] specially adapted for controlling image capture or setting capture parameters
• CPC Class Codes: H04N 5/44504 : Circuit details of the additional information generator, e.g. details of the character or graphics signal generator, overlay mixing circuits
• CPC Class Codes: H04N 7/183 : for receiving images from a single remote source
• CPC Class Codes: H04N 7/185 : from a mobile camera, e.g. for remote control

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for an unmanned aerial system inspection system. One of the methods is performed by a UAV and includes receiving, by the UAV, flight information describing a job to perform an inspection of a rooftop. A particular altitude is ascended to, and an inspection of the rooftop is performed including obtaining sensor information describing the rooftop. Location information identifying a damaged area of the rooftop is received. The damaged area of the rooftop is traveled to. An inspection of the damaged area of the rooftop is performed including obtaining detailed sensor information describing the damaged area. A safe landing location is traveled to.

Claims

1. A computerized method performed by an unmanned aerial vehicle (UAV) comprising one or more processors, the method comprising:

receiving, by the UAV, a flight plan comprising a plurality of inspection locations for a structure, wherein the plurality of inspection locations each comprise a waypoint having a geospatial reference;

navigating the UAV to ascend to a clearance altitude above a height of the structure;

conducting an inspection for an inspection location of the plurality of inspection locations according to the received flight plan, the inspection comprising;

navigating the UAV to a position above a surface of the structure associated with the inspection location, and descending, from the position, to within a particular distance from the surface of the structure based on monitoring an onboard distance measuring device,

the onboard distance measuring device measuring distances to locations on the surface, and

the particular distance above the surface being determined based on a ground sampling distance, such that at least a particular level of detail is included in obtained sensor information; and

obtaining, while the UAV is within the particular distance from the surface of the structure, information from one or more sensors describing the structure, such that the obtained information includes at least the particular level of detail;

navigating the UAV to another inspection location of the plurality of inspection locations; and

navigating the UAV to a landing location,

wherein a first inspection location is associated with a first ground sampling distance, and wherein a second inspection location is associated with a second different ground sampling distance, such that sensor information associated with the first inspection location and second inspection location include different levels of detail.

2. The method of claim 1, wherein descending comprises:

accessing one or more digital images obtained by the UAV during a prior inspection of the structure, the one or more digital images including less than the particular level of detail and obtained while the UAV was at greater than the particular distance from the surface of the structure;

performing, based on portions of the accessed digital images associated with the inspection location, visual odometry using sensor information obtained while descending; and

correcting a position of the UAV, along a horizontal direction, while descending based on the performed visual odometry, such that the UAV descends along a vertical direction to the surface of the structure associated with the inspection location.

3. The method of claim 1, wherein conducting an inspection further comprises triggering the one or more sensors to obtain sensor information, and wherein triggering comprises one or more of:

periodically triggering the sensors while the UAV is within the particular distance from the surface of the structure, triggering the sensors in response to an event.

4. The method of claim 3, wherein triggering the sensors in response to an event comprises:

determining a value from accelerometers or rotational rate gyros of the UAV while the UAV is obtaining sensor information; and

determining whether the value exceeds a threshold value, and in response to a positive determination, triggering the sensors to again obtain the sensor information.

5. The method of claim 3, wherein triggering the sensors in response to an event comprises:

evaluating obtained sensor information to determine whether the sensor information passes a quality criteria; and

determining whether the sensor information passes the quality criteria, and in response to a negative determination, triggering the sensor again to obtain the sensor information.

6. The method of claim 1, wherein the onboard distance measuring device comprises an optical ranging device or a sonar device.

7. The method of claim 1 further comprising:

obtaining, from a user device, a 3D model of the structure;

while navigating, obtaining geospatial location information of the UAV;

determining based on the 3D model, a position of the UAV in relation to the 3D model; and

adjusting a location of the UAV based on the determined position.

8. The method of claim 1, wherein navigating the UAV to a position comprises:

navigating the UAV at a speed to the inspection location based on a waypoint transition speed value, wherein the waypoint transition speed value is set at different values for travel to different inspection locations.

9. The method of claim 1 further comprising:

transmitting the obtained sensor information to a user device, the sensor information comprising one or more digital images, the one or more digital images having associated geospatial references.

10. The method of claim 1, wherein conducting an inspection further comprises:

allowing limited control of the UAV by a user device, the limited control comprising one or more of;

controlling a vertical rate of ascent or descent of the UAV, controlling rotation of the UAV, triggering of one or more sensors, controlling a direction of one or more sensors, nudging of the UAV or resuming auto-piloted navigation of the UAV.

11. A computer storage medium comprising instructions that when executed by one or more processors included in a Unmanned Aerial Vehicle (UAV), cause the UAV to perform operations comprising:

receiving, by the UAV, a flight plan comprising a plurality of inspection locations for a structure, wherein the plurality of inspection locations each comprise a waypoint having a geospatial reference;

navigating the UAV to ascend to a clearance altitude above a height of the structure;

conducting an inspection for an inspection location of the plurality of inspection locations, the inspection comprising;

navigating the UAV to a position above a surface of the structure associated with the inspection location, and descending, from the position, to within a particular distance from the surface of the structure based on monitoring an onboard distance measuring device,

the onboard distance measuring device measuring distances to locations on the surface, and

the particular distance above the surface being determined to obtain at least a particular level of detail in sensor information; and

obtaining, while the UAV is within the particular distance from the surface of the structure, information from one or more sensors describing the structure, such that the obtained sensor information includes at least the particular level of detail;

navigating the UAV to another inspection location of the plurality of inspection locations; and

navigating the UAV to a landing location,

wherein a first inspection location is associated with a first ground sampling distance, and wherein a second inspection location is associated with a second different ground sampling distance, such that sensor information associated with the first inspection location and second inspection location include different levels of detail.

12. The computer storage medium of claim 11, wherein descending comprises:

accessing one or more digital images obtained by the UAV during a prior inspection of the structure, the one or more digital images including less than the particular level of detail and obtained while the UAV was at greater than the particular distance from the surface of the structure;

performing, based on portions of the accessed digital images associated with the inspection location, visual odometry using sensor information obtained while descending; and

correcting a position of the UAV, along a horizontal direction, while descending based on the performed visual odometry, such that the UAV descends along a vertical direction to the surface of the structure associated with the inspection location.

13. The computer storage medium of claim 11, wherein conducting an inspection further comprises triggering the one or more sensors to obtain sensor information, and wherein triggering comprises one or more of:

periodically triggering the sensors while the UAV is within the particular distance from the surface of the structure, triggering the sensors in response to an event.

14. The computer storage medium of claim 13, wherein triggering the sensors in response to an event comprises:

determining a value from accelerometers or rotational rate gyros of the UAV while the UAV is obtaining sensor information; and

determining whether the value exceeds a threshold value, and in response to a positive determination, triggering the sensors to again obtain the sensor information.

15. The computer storage medium of claim 13, wherein triggering the sensors in response to an event comprises:

evaluating obtained sensor information to determine whether the sensor information passes a quality criteria; and

determining whether the sensor information passes the quality criteria, and in response to a negative determination, triggering the sensor again to obtain the sensor information.

16. The computer storage medium of claim 11, wherein the particular level of detail indicates an acceptable image pixel resolution of digital images included in the obtained sensor information.

17. The computer storage medium of claim 11, wherein the operations further comprise:

obtaining, from a user device, a 3D model of the structure;

while navigating, obtaining geospatial location information of the UAV;

determining based on the 3D model, a position of the UAV in relation to the 3D model; and

adjusting a location of the UAV based on the determined position.

18. The computer storage medium of claim 11, wherein navigating the UAV to a position comprises:

navigating the UAV at a speed to the inspection location based on a waypoint transition speed value, wherein the waypoint transition speed value is set at different values for travel to different inspection locations.

19. The computer storage medium of claim 11, wherein the operations further comprise:

transmitting the obtained sensor information to a user device, the sensor information comprising one or more digital images, the one or more digital images having associated geospatial references.

20. The computer storage medium of claim 11, wherein conducting an inspection further comprises:

allowing limited control of the UAV by a user device, the limited control comprising one or more of;

controlling a vertical rate of ascent or descent of the UAV, controlling rotation of the UAV, triggering of one or more sensors, controlling a direction of one or more sensors, nudging of the UAV or resuming auto-piloted navigation of the UAV.

21. An unmanned aerial vehicle (UAV) system comprising one or more processors, and a computer storage media storing instructions, that when executed by the one or more processors, cause the one or more processors to perform operations comprising:

receiving, by the UAV, a flight plan comprising a plurality of inspection locations for a structure, wherein the plurality of inspection locations each comprise a waypoint having a geospatial reference;

navigating the UAV to ascend to a clearance altitude above a height of the structure;

conducting an inspection for an inspection location of the plurality of inspection locations, the inspection comprising;

navigating the UAV to a position above a surface of the structure associated with the inspection location, and descending, from the position, to within a particular distance from the surface of the structure based on monitoring an onboard distance measuring device,

the onboard distance measuring device measuring distances to locations on the surface, and

the particular distance above the surface being determined to obtain at least a particular level of detail in sensor information; and

while the UAV is within the particular distance from the surface of the structure, obtaining information from one or more sensors describing the structure, such that the obtained information includes at least the particular level of detail;

navigating the UAV to another inspection location of the plurality of inspection locations; and

navigating the UAV to a landing location,

wherein a first inspection location is associated with a first ground sampling distance, and wherein a second inspection location is associated with a second different ground sampling distance, such that sensor information associated with the first inspection location and second inspection location include different levels of detail.

22. The system of claim 21, wherein descending comprises:

accessing one or more digital images obtained by the UAV during a prior inspection of the structure, the one or more digital images including less than the particular level of detail and obtained while the UAV was at greater than the particular distance from the surface of the structure;

performing, based on portions of the accessed digital images associated with the inspection location, visual odometry using sensor information obtained while descending; and

correcting a position of the UAV, along a horizontal direction, while descending based on the performed visual odometry, such that the UAV descends along a vertical direction to the surface of the structure associated with the inspection location.

23. The system of claim 21, wherein conducting an inspection further comprises triggering the one or more sensors to obtain sensor information, and wherein triggering comprises one or more of:

periodically triggering the sensors while the UAV is within the particular distance from the surface of the structure, triggering the sensors in response to an event.

24. The system of claim 23, wherein triggering the sensors in response to an event comprises:

determining a value from accelerometers or rotational rate gyros of the UAV while the UAV is obtaining sensor information; and

determining whether the value exceeds a threshold value, and in response to a positive determination, triggering the sensors to again obtain the sensor information.

25. The system of claim 23, wherein triggering the sensors in response to an event comprises:

evaluating obtained sensor information to determine whether the sensor information passes a quality criteria; and

determining whether the sensor information passes the quality criteria, and in response to a negative determination, triggering the sensor again to obtain the sensor information.

26. The system of claim 21, wherein the particular level of detail indicates an acceptable image pixel resolution of digital images included in the obtained sensor information.

27. The system of claim 21, wherein the operations further comprise:

obtaining, from a user device, a 3D model of the structure;

while navigating, obtaining geospatial location information of the UAV;

determining based on the 3D model, a position of the UAV in relation to the 3D model; and

adjusting a location of the UAV based on the determined position.

28. The system of claim 21, wherein the operations further comprise:

transmitting the obtained sensor information to a user device, the sensor information comprising one or more digital images, the one or more digital images having associated geospatial references.

29. The system of claim 21, wherein conducting an inspection further comprises:

allowing limited control of the UAV by a user device, the limited control comprising one or more of;

controlling a vertical rate of ascent or descent of the UAV, controlling rotation of the UAV, triggering of one or more sensors, controlling a direction of one or more sensors, nudging of the UAV or resuming auto-piloted navigation of the UAV.