Identifying camera position of a UAV in flight utilizing real time kinematic satellite navigation

• US 10,189,567 B2
• Filed: 06/09/2016
• Issued: 01/29/2019
• Est. Priority Date: 06/09/2016
• Status: Active Grant

• Current Assignee: Skycatch, Inc.
• Original Assignee: Skycatch, Inc.
• Inventors: Chen, David, Twitty, Colleen Kelly, Schanfeldt, Blaine, Johnson, James J, Federico, Jacob
• Primary Examiner(s): Smith, Jelani A
• Assistant Examiner(s): Wallace, Donald J
• Application Number:
  US15/178,052
• Publication Number:
  US 20170355458A1
• Time in Patent Office: 964 Days
• CPC Class Codes: B64C 29/0033 the propellers being tiltable relative to the fuselage
• CPC Class Codes: B64C 39/024 of the remote controlled vehicle type, i.e. RPV
• CPC Class Codes: B64U 10/10 Rotorcrafts
• CPC Class Codes: B64U 10/13 Flying platforms
• CPC Class Codes: B64U 2101/30 for imaging, photography or videography
• CPC Class Codes: B64U 2201/104 using satellite radio beacon positioning systems, e.g. GPS
• CPC Class Codes: G05D 1/0094 involving pointing a payload, e.g. camera, weapon, sensor, towards a fixed or moving target
• CPC Class Codes: G05D 1/0858 specially adapted for vertical take-off of aircraft
• CPC Class Codes: G06T 17/05 Geographic models
• CPC Class Codes: G06V 20/13 Satellite images
• CPC Class Codes: G06V 20/17 taken from planes or by drones
• CPC Class Codes: G08G 5/0021 located in the aircraft
• CPC Class Codes: G08G 5/0069 specially adapted for an unmanned aircraft
• CPC Class Codes: G08G 5/0086 for monitoring terrain radar or analogous systems specially adapted for terrain avoidance G01S13/935

Abstract

Systems and methods are disclosed for determining a position of a camera affixed to a UAV in flight when the camera captures a digital aerial image. In particular, in one or more embodiments, the disclosed systems and methods utilize real time kinematic satellite navigation techniques to identify a position of a UAV. Moreover, the disclosed systems and methods precisely determine the time that a camera captures a digital aerial image together with an attitude of the UAV at the time of capture to calculate a location of a camera at the time of capture. In one or more embodiments, the disclosed systems and methods can utilize the determined position of the camera together with the captured digital aerial image to generate a three-dimensional representation of a site.

Claims

1. A method comprising:

determining, by at least one processor during a flight mission of a UAV, a position of the UAV at a first time and a position of the UAV at a second time based on a number of wave lengths of a signal transmitted from a satellite to the UAV at the first time, a number of wave lengths of the signal at the second time, RTK correction data at the first time, and RTK correction data at the second time;

identifying, by the at least one processor during the flight mission, a time that a camera affixed to the UAV captures a digital aerial image of a site by detecting when a shutter of the camera affixed to the UAV is open;

detecting an attitude of the UAV at the time that the camera affixed to the UAV captures the digital aerial image, wherein the attitude comprises a measurement of pitch, roll, and yaw of the UAV corresponding to the time that the camera affixed to the UAV captures the digital aerial image; and

determining, by the at least one processor, a position of the camera based on a time of capturing the digital aerial image, the attitude of the UAV at the time that the camera affixed to the UAV captures the digital aerial image, the position of the UAV at the first time, and the position of the UAV at the second time;

determining, by the at least one processor during the flight mission, a quality of a calculation of the position of the UAV at a third time, wherein determining the quality of the calculation comprises determining whether the calculation is fixed or floating; and

modifying flight of the UAV based on the determination of the quality of the calculation.

2. The method of claim 1, further comprising modifying navigation of the UAV based on the position of the camera at the time of capturing the digital aerial image.

3. The method of claim 1, further comprising generating a three-dimensional representation of the site based in part on the digital aerial image of the site and the position of the camera at the time of capturing the digital aerial image.

4. The method of claim 1 further comprising determining the position of the UAV at the time of capturing the digital aerial image by interpolating between the position of the UAV at the first time and the position of the UAV at the second time.

5. The method of claim 1, wherein the position of the UAV comprises a position of a GPS receiver affixed to the UAV and determining the position of the camera comprises:

generating a distance vector from the position of the GPS receiver and the camera based on the detected attitude of the UAV at the time that the camera affixed to the UAV captures the digital aerial image; and

applying the distance vector to the position of the GPS receiver at the time of capturing the digital aerial image.

6. The method of claim 1, further comprising:

synchronizing a flight controller clock of the UAV with a GPS satellite using a time transmission received from one or more satellites during the flight mission of the UAV;

detecting the first time, the second time, and the time that the camera affixed to the UAV captures the digital aerial image utilizing the flight controller clock; and

synchronizing the RTK correction data at the first time and the RTK correction data at the second time utilizing the flight controller clock.

7. The method of claim 1, wherein detecting the time that the camera affixed to the UAV captures the digital aerial image further comprises detecting, by a light sensor, a flash generated by the camera when the camera shutter is open.

8. The method of claim 1, wherein determining the position of the UAV at the first time comprises receiving, during the flight mission, a feed of RTK correction data generated by at least one of an RTK network comprising a plurality of reference stations or a reference station at the site corresponding to the flight mission.

9. The method of claim 8, further comprising:

sending an indication in relation to beginning the flight mission; and

in response to sending the indication, receiving the feed of RTK correction data by the UAV during the flight mission, the feed of RTK correction data comprising the RTK correction data at the first time and the RTK correction data at the second time.

10. The method of claim 1, wherein detecting when the shutter of the camera affixed to the UAV is open comprises measuring a signal from output pins within the camera indicating that the shutter is open.

11. A method comprising:

calculating, by at least one processor, a position of a UAV at a first time and a position of the UAV at a second time based on a signal transmitted from a satellite to the UAV at the first time, a signal at the second time, RTK correction data at the first time, and RTK correction data at the second time;

identifying, by the at least one processor, a time that the UAV captured a digital aerial image of a site based on a measurement by the UAV of when a shutter of a camera affixed to the UAV was open;

obtaining an attitude of the UAV corresponding to the time that the UAV captured the digital aerial image, wherein the attitude comprises a measurement of pitch, roll, and yaw of the UAV corresponding to the time that the UAV captured the digital aerial image;

determining a position of the camera at a time of capturing the digital aerial image based on the attitude of the UAV at the time of capturing the digital aerial image, the position of the UAV at the first time, and the position of the UAV at the second time by;

generating a distance vector from a position of a GPS receiver to the camera based on the attitude of the UAV at the time that the camera affixed to the UAV captures the digital aerial image; and

applying the distance vector to the position of the GPS receiver at the time of capturing the digital aerial image; and

generating a three-dimensional representation of the site based on the digital aerial image and the determined position of the camera at the time of capturing the digital aerial image.

12. The method of claim 11, wherein the digital aerial image of the site lacks ground control points, the method further comprising generating the three-dimensional representation of the site without using ground control points.

13. The method of claim 11, further comprising obtaining RTK correction data from at least one of a third-party server or a reference station at the site.

14. The method of claim 11, further comprising:

determining a quality of a calculation of the position of the UAV at a third time, wherein determining the quality of the calculation comprises determining whether the calculation is fixed or floating; and

modifying flight of the UAV based on the determination of the quality of the calculation.

15. A system comprising:

a UAV with a camera affixed to the UAV;

at least one processor; and

at least one non-transitory computer readable storage medium storing instructions thereon that, when executed by the at least one processor, cause the system to;

determine, during a flight mission of the UAV, a position of the UAV at a first time and a position of the UAV at a second time based on a number of wave lengths of a signal transmitted from a satellite to the UAV at the first time, a number of wave lengths of the signal at the second time, RTK correction data at the first time, and RTK correction data at the second time;

identify, during the flight mission, a time that the camera affixed to the UAV captures a digital aerial image of a site by detecting when a shutter of the camera affixed to the UAV is open;

detect an attitude of the UAV at the time that the camera affixed to the UAV captures the digital aerial image, wherein the attitude comprises a measurement of pitch, roll, and yaw of the UAV corresponding to the time that the UAV captures the digital aerial image; and

determine a position of the camera based on a time of capturing the digital aerial image, the attitude of the UAV at the time that the camera affixed to the UAV captures the digital aerial image, the position of the UAV at the first time, and the position of the UAV at the second time by;

generating a distance vector from a position of a GPS receiver and the camera based on the detected attitude of the UAV at the time that the camera affixed to the UAV captures the digital aerial image; and

applying the distance vector to the position of the GPS receiver at the time of capturing the digital aerial image.

16. The system of claim 15, further comprising instructions that, when executed by the at least one processor, cause the system to:

generate a three-dimensional representation of the site based in part on the digital aerial image of the site and the position of the camera at the time of capturing the digital aerial image.

17. The system of claim 15, wherein identifying the time that the camera affixed to the UAV captures the digital aerial image further comprises detecting a signal generated by the camera, wherein the signal triggers the shutter to open.

18. The system of claim 15, wherein the UAV comprises a single frequency GPS receiver and determining the position of the UAV comprises:

fixing a number of full wavelengths between the GPS receiver and the satellite.

19. The system of claim 15, wherein the position of the UAV comprises a position of the GPS receiver affixed to the UAV and further comprising instructions that, when executed by the at least one processor, cause the system to:

determine the position of the GPS receiver at the time of capturing the digital aerial image by interpolating between the position of the GPS receiver at the first time and the position of the GPS receiver at the second time.

20. The system of claim 15, further comprising instructions that, when executed by the at least one processor, cause the system to:

determine, during the flight mission, a quality of a calculation of the position of the UAV at a third time, wherein determining the quality of the calculation comprises determining whether the calculation is fixed or floating; and

modify flight of the UAV based on the determination of the quality of the calculation.