Detecting of navigation data spoofing based on image data

• US 9,689,686 B1
• Filed: 09/25/2015
• Issued: 06/27/2017
• Est. Priority Date: 09/25/2015
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Carmack, Scott Gerard, Lakkakula, Narasimha Rao, Mehr, Nima Sharifi
• Primary Examiner(s): Tarcza, Thomas
• Assistant Examiner(s): Berns, Michael
• Application Number:
  US14/866,719
• Time in Patent Office: 641 Days
• CPC Class Codes: B64C 39/024 of the remote controlled vehicle type, i.e. RPV
• CPC Class Codes: B64U 10/13 Flying platforms
• CPC Class Codes: B64U 2101/30 for imaging, photography or videography
• CPC Class Codes: B64U 2101/60 for transporting passengers; for transporting goods other than weapons
• CPC Class Codes: B64U 2201/104 using satellite radio beacon positioning systems, e.g. GPS
• CPC Class Codes: G01C 21/20 Instruments for performing navigational calculations G01C21/24, G01C21/26 take precedence
• CPC Class Codes: G01S 13/86 Combinations of radar systems with non-radar systems, e.g. sonar, direction finder
• CPC Class Codes: G01S 19/00 Satellite radio beacon positioning systems; Determining position, velocity or attitude using signals transmitted by such systems
• CPC Class Codes: G01S 19/215 issues related to spoofing
• CPC Class Codes: G01S 19/485 whereby the further system is an optical system or imaging system
• CPC Class Codes: G05D 1/0022 characterised by the communication link data switching networks in general H04L12/00
• CPC Class Codes: G05D 1/0038 by providing the operator with simple or augmented images from one or more cameras located onboard the vehicle, e.g. tele-operation images analyzed by a computer and used for automatic navigation G05D1/0246
• CPC Class Codes: G06F 21/64 Protecting data integrity, e.g. using checksums, certificates or signatures
• CPC Class Codes: G06F 2221/2111 Location-sensitive, e.g. geographical location, GPS
• CPC Class Codes: H04K 2203/22 for communication related to vehicles
• CPC Class Codes: H04K 3/22 including jamming detection and monitoring
• CPC Class Codes: H04K 3/65 using deceptive jamming or spoofing, e.g. transmission of false signals for premature triggering of RCIED, for forced connection or disconnection to/from a network or for generation of dummy target signal
• CPC Class Codes: H04K 3/90 related to allowing or preventing navigation or positioning, e.g. GPS
• CPC Class Codes: H04L 63/0869 for achieving mutual authentication cryptographic mechanisms or cryptographic arrangements for mutual authentication H04L9/3273
• CPC Class Codes: H04L 63/123 received data contents, e.g. message integrity
• CPC Class Codes: H04W 12/06 : Authentication
• CPC Class Codes: H04W 12/10 : Integrity
• CPC Class Codes: H04W 12/63 : Location-dependent; Proximity-dependent
• CPC Class Codes: H04W 4/02 : Services making use of location information

Abstract

Techniques for determining whether data associated with an autonomous navigation of an unmanned vehicle may be trusted. For example, navigation-related data may be provided from a source external to the unmanned vehicle. Image data associated with the autonomous navigation may be generated. The navigation-related data and the image data may be compared to determine whether the navigation data may be trusted or not. If untrusted, the autonomous navigation may be directed independently of the navigation data.

Claims

1. A computer-implemented method, comprising:

receiving, by a computer system of an unmanned aerial vehicle (UAV), global positioning system (GPS) data, the computer system configured to direct a flight of the UAV to deliver an item from a source to a destination along a predefined route based at least in part on the GPS data;

requesting, by the computer system from an imaging device of the UAV, an image associated with a location of the UAV, the UAV being at the location based at least in part on the GPS data;

accessing, by the computing system, a known image associated with a known location along the predefined route;

determining, by the computer system, a discrepancy between the known image and the image;

determining, by the computer system, that the GPS data is spoofed based at least in part on the discrepancy between the known image and the image; and

directing, by the computer system, a portion of the flight independently of the GPS data that is spoofed.

2. The computer-implemented method of claim 1, wherein determining the discrepancy comprises:

transmitting, by the computer system, the image to a central station, the central station configured to;

select the known image from available images based at least in part on checkpoints along the predefined route of the UAV,

compare the image to the known image, and

determine that the discrepancy between the image and the known image based at least in part on the comparison, and

receiving the discrepancy from the central station.

3. The computer-implemented method of claim 1, wherein determining that the GPS data is spoofed comprises:

detecting, by the computer system, that the discrepancy exceeds a threshold, wherein the threshold is based at least in part on one or more of;

the camera, the item, historical deliveries of the UAV, or historical deliveries of another UAV.

4. The computer-implemented method of claim 1, wherein directing the portion of the flight independently of the GPS data comprises:

receiving sensor data of a navigation sensor of the UAV; and

directing the portion of the flight based at least in part on the sensor data.

5. One or more non-transitory computer readable media comprising instructions that, when executed with a computer system, cause the computer system to at least:

access navigation data of a source external to an unmanned autonomous vehicle (UAV), the navigation data associated with directing an autonomous navigation of the UAV along a predefined route;

access image data of an optical sensor installed on the UAV;

access known image data associated with a known location along the predefined route;

determine that the navigation data is untrusted based at least in part on a discrepancy between the known image data and the image data; and

direct the autonomous navigation of the UAV independently of the navigation data based at least in part on the navigation data being untrusted.

6. The one or more non-transitory computer readable media of claim 5, wherein the UAV is configured to deliver an item available from an electronic marketplace, wherein the source external to the UAV comprises a global positioning system (GPS), and wherein the navigation data comprises GPS data.

7. The one or more non-transitory computer readable media of claim 5, wherein the computer system is installed at the UAV.

8. The one or more non-transitory computer readable media of claim 5, wherein the computer system is installed at a central station and is configured to communicate with the UAV over a network, wherein accessing the image data comprises receiving an image of a location of the UAV, wherein determining that the navigation data is untrusted comprises:

detecting an object in the image;

matching the object to a known object from a map, the map indicating a particular location of the known object;

determining the discrepancy based on a comparison of the known location along the predefined route of the UAV and the particular location of the known object; and

determining that the discrepancy exceeds a threshold.

9. The one or more non-transitory computer readable media of claim 5, wherein the image data forms an image at a particular resolution of a location of the UAV, wherein the UAV is at the location based at least in part on the navigation data.

10. The one or more non-transitory computer readable media of claim 9, wherein the instructions, when executed with the computer system, further cause the computer system to at least:

request, based at least in part on determining that the navigation data is untrusted, a second image at a higher resolution from an imaging device comprising the optical sensor; and

confirm that the navigation data is untrusted based at least in part on the second image.

11. The one or more non-transitory computer readable media of claim 5, wherein the instructions, when executed with the computer system, further cause the computer system to at least, upon determining that the navigation data is untrusted:

change a location of the UAV;

request an image of the location from an imaging device comprising the optical sensor; and

confirm that the navigation data is untrusted based at least in part on the image.

12. The one or more non-transitory computer readable media of claim 5, wherein the image data forms an image of a location of the UAV, wherein the instructions, when executed with the computer system, further cause the computer system to at least:

request an imaging device comprising the optical sensor to generate the image upon a detection that the UAV is at the location based at least in part on the navigation data.

13. The one or more non-transitory computer readable media of claim 5, wherein the image data forms an image of a location of the UAV, wherein the instructions, when executed with the computer system, further cause the computer system to at least:

request an imaging device comprising the optical sensor to generate images of locations of the UAV at a dynamic frequency, wherein the dynamic frequency increases upon determining that the navigation data is untrusted.

14. A system associated with an unmanned vehicle, the system comprising:

one or more processors; and

one or more non-transitory computer readable media comprising instructions that, when executed with the one or more processors, cause the system to at least;

access navigation data of a source external to the unmanned vehicle, the navigation data associated with directing an autonomous navigation of the unmanned vehicle along a predefined route;

access image data of an optical sensor of the unmanned vehicle, the image data associated with a location of the unmanned vehicle based at least in part on the autonomous navigation;

access known image data associated with a known location along the predefined route;

determine that the navigation data is untrusted based at least in part on a discrepancy between the known image data and the image data; and

direct the autonomous navigation of the unmanned vehicle independently of the navigation data that is untrusted and based at least in part on a corrective action.

15. The system of claim 14, wherein the image data forms an image at a resolution, wherein the instructions, when executed with the one or more processors, further cause the system to at least:

detect that the unmanned vehicle is at the location based at least in part on the navigation data, and

request an imaging device comprising the optical sensor to generate the image at the resolution upon the detecting, and

wherein the corrective action comprises;

requesting the imaging device to generate an additional image at a higher resolution upon determining that the navigation data is untrusted; and

confirming that the navigation data is untrusted based at least in part on the additional image.

16. The system of claim 14, wherein the corrective action comprises:

reporting that the navigation data is untrusted to a central station; and

facilitating a remote control over operations of the unmanned vehicle from the central station.

17. The system of claim 14, wherein determining that the navigation data is untrusted comprises:

accessing a known image from a data store, the known image corresponding to a checkpoint along a predefined route of the unmanned vehicle;

comparing the image data of the optical sensor to second image data of the known image; and

determining the discrepancy based at least in part on comparing the image data and the second image data.

18. The system of claim 14, wherein the optical sensor comprises an infrared sensor, wherein the image data forms an infrared image of the location, and wherein determination that the navigation data is untrusted comprises:

accessing a known infrared image from a data store based at least in part on the navigation data; and

comparing the infrared image to the known infrared image to detect the discrepancy.

19. The system of claim 14, wherein the navigation data is available to the unmanned vehicle from a first source, wherein the corrective action comprises transmitting the image data and an indication that the navigation data is untrusted to a central station, wherein the central station is configured to:

request second image data of the location from a second unmanned vehicle and a second indication from the second unmanned vehicle whether second navigation data associated with the location is trusted;

compare the image data to the second image data based at least in part on the second navigation data being trusted, wherein the second navigation data is available to the second unmanned vehicle from a second source different from the first source;

identifying the first source of the navigation data based at least in part on comparing the image data and the second image data.

20. The system of claim 14, wherein determining that the navigation data is untrusted comprises:

detecting an object based at least in part on the image data;

determining a known location associated with the object; and

determining the discrepancy between a predefined route of the unmanned vehicle and the known location.