Fiducial-based navigation of unmanned vehicles

• US 9,964,951 B1
• Filed: 09/25/2015
• Issued: 05/08/2018
• Est. Priority Date: 09/25/2015
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Dunn, Steven Gregory, Kelso, Carl Ryan, Woodward, III, Neil Whitney
• Primary Examiner(s): Khatib, Rami
• Assistant Examiner(s): Foster, Gerrad A
• Application Number:
  US14/866,760
• Time in Patent Office: 956 Days
• Field of Search: None
• CPC Class Codes: B64C 39/024 of the remote controlled vehicle type, i.e. RPV
• CPC Class Codes: B64D 45/04 Landing aids; Safety measures to prevent collision with earth's surface
• CPC Class Codes: B64U 2101/60 for transporting passengers; for transporting goods other than weapons
• CPC Class Codes: B64U 2101/64 for parcel delivery or retrieval
• CPC Class Codes: B64U 2201/10 autonomous, i.e. by navigating independently from ground or air stations, e.g. by using inertial navigation systems [INS]
• CPC Class Codes: B64U 60/50 with landing legs
• CPC Class Codes: G05D 1/0088 characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours using knowledge based models G06N5/00
• CPC Class Codes: G05D 1/101 specially adapted for aircraft
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10032 Satellite or aerial image; Remote sensing
• CPC Class Codes: G06T 2207/30204 Marker
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06V 10/245 by locating a pattern; Special marks for positioning
• CPC Class Codes: G06V 20/13 Satellite images
• CPC Class Codes: G06V 20/17 taken from planes or by drones
• CPC Class Codes: G08G 5/0021 located in the aircraft
• CPC Class Codes: G08G 5/0052 for cruising combined instruments indicating more than one navigational value G01C23/00
• CPC Class Codes: G08G 5/0069 specially adapted for an unmanned aircraft
• CPC Class Codes: G08G 5/0086 for monitoring terrain radar or analogous systems specially adapted for terrain avoidance G01S13/935

Abstract

Techniques for facilitating an autonomous operation, such as an autonomous navigation, of an unmanned vehicle based on one or more fiducials. For example, image data of a fiducial may be generated with an optical sensor of the unmanned vehicle. The image data may be analyzed to determine a location of the fiducial. A location of the unmanned vehicle may be estimated from the location of the fiducial and the image. The autonomous navigation of the unmanned vehicle may be directed based on the estimated location.

Claims

1. A system, comprising:

a first fiducial located at a first location and encoding information about the first location and about a dimension of the first fiducial; and

an unmanned aerial vehicle (UAV) comprising;

an imaging device configured to capture a first image of the first fiducial;

a navigation sensor configured to generate navigation data; and

a computer system configured to;

generate a navigation state of the UAV based at least in part on the navigation data;

direct an autonomous flight of the UAV to deliver an item from a source to a destination based at least in part on the navigation state;

analyze the first image of the first fiducial to determine the information about the first location and the dimension of the first fiducial;

estimate a location of the UAV based at least in part on the information about the first location and the dimension of the first fiducial determined from the first image, wherein estimating the location of the UAV comprises estimating one or more of;

a latitude, a longitude, or an altitude based at least in part on a geometric reconstruction of the latitude, the longitude, or the altitude from the first location and the dimension of the first fiducial; and

update the navigation state based at least in part on the location of the UAV.

2. The system of claim 1, wherein the first location of the first fiducial represents a waypoint of a delivery route that is generated based at least in part on the source and the destination, and wherein the first fiducial comprises a multi-dimensional barcode that encodes the information about the first location and about the size.

3. The system of claim 1, wherein the system comprises a second fiducial located at a second location, wherein the first fiducial encodes information about the second location, and wherein the computer system of the UAV is further configured to:

determine the second location from the first image of the first fiducial;

direct the autonomous flight of the UAV based at least in part on the second location;

analyze a second image of the second fiducial to determine an offset from the navigation data; and

update the navigation state based at least in part on the offset.

4. A computer-implemented method, comprising:

accessing, by a computer system, image data of a fiducial, the image data detected based at least in part on an optical sensor of an unmanned aerial vehicle (UAV);

determining, by the computer system, a first location and a dimension of the fiducial based at least in part on the image data, the fiducial associated with information about the first location and the dimension of the fiducial;

estimating, by the computer system, a location of the UAV based at least in part on the first location of the fiducial, wherein estimating the location of the UAV comprises estimating one or more of;

a latitude, a longitude, or an altitude based at least in part on a geometric reconstruction of the latitude, the longitude, or the altitude from the first location and the dimension of the fiducial; and

directing, by the computer system, an autonomous navigation of the UAV based at least in part on the location of the UAV.

5. The computer-implemented method of claim 4, wherein the UAV is configured to deliver an item offered from an electronic marketplace, wherein directing the autonomous navigation comprising updating a navigation state of the UAV based at least in part on the location of the UAV, and wherein the fiducial encodes information about the first location and about a dimension of the fiducial.

6. The computer-implemented method of claim 4, further comprising:

receiving navigation data of a navigation device of the UAV;

generating a navigation state based at least in part on the navigation data;

determining an offset associated with the navigation data based at least in part on the location; and

directing the autonomous navigation of the UAV based at least in part on the navigation state and the offset.

7. The computer-implemented method of claim 4, wherein the autonomous navigation is based at least in part on a predetermined navigation route, wherein the fiducial represents a waypoint of the navigation route, and further comprising updating the predetermined navigation route based at least in part on the location of the UAV.

8. The computer-implemented method of claim 4, wherein the autonomous navigation is based at least in part on a navigation state, and further comprising:

generating a plurality of images of the fiducial based at least in part on the optical sensor of the UAV;

analyzing the plurality of images to determine the location of the UAV over time and to update the navigation state based at least in part on the location over time;

determining that a second fiducial has entered a field of view of the optical sensor of the UAV while the fiducial has remained in the field of view;

generating a plurality of second images of the second fiducial based at least in part on the fiducial exiting the field of view; and

analyzing the plurality of second images to further update the navigation state.

9. The computer-implemented method of claim 4, wherein the autonomous navigation is based at least in part on a route that identifies the fiducial and a second fiducial, and further comprising:

determining that the fiducial is out of a field of view of the optical sensor of the UAV;

maintaining the navigation state based at least in part on the fiducial being out of the field of view;

directing the autonomous navigation of the UAV based at least in part on the route and based at least in part on the navigation state being maintained;

determining that the second fiducial is within the field of view of the optical sensor of the unmanned vehicle;

analyzing second image data of the second fiducial to determine an updated location of the unmanned vehicle; and

updating the navigation state based at least in part on the updated location of the unmanned vehicle.

10. The computer-implemented method of claim 4, wherein determining the first location of the fiducial comprises:

determining an identifier of the fiducial based at least in part on an analysis of the image data; and

accessing the first location of the fiducial from a data store based at least in part on the identifier of the fiducial, wherein the data store associates fiducials with corresponding locations.

11. The computer-implemented method of claim 10, wherein the associations of the fiducials with the corresponding locations are maintained by a central station, wherein the central station is configured to receive a plurality of images of the fiducial from a plurality of UAVs and to associate a credibility to the fiducial based at least in part on the plurality of images, and wherein estimating the location of the UAV is based at least in part on the credibility associated with the fiducial.

12. The computer-implemented method of claim 4, wherein determining the first location of the fiducial comprises:

determining an identifier of the fiducial based at least in part on an analysis of the image data;

providing the identifier of the fiducial to a second UAV; and

receiving the first location from the second UAV based at least in part on the identifier.

13. A system associated with an unmanned vehicle that comprises a navigation device and an optical sensor, the system comprising:

one or more processors; and

one or more computer-readable storage media comprising instructions that, when executed with the one or more processors, cause the system to at least;

generate a navigation state of the unmanned vehicle based at least in part on a navigation device;

direct an autonomous path of the unmanned vehicle based at least in part on the navigation state;

detect image data of a fiducial via the optical sensor to determine a first location and a dimension of the fiducial;

estimate a location of the unmanned vehicle based at least in part on the first location and the dimension of the fiducial, wherein estimating the location of the unmanned vehicle comprises estimating one or more of;

a latitude, a longitude, or an altitude based at least in part on a geometric reconstruction of the latitude, the longitude, or the altitude from the first location and the dimension of the fiducial; and

update the navigation state based at least in part on the location of the unmanned vehicle.

14. The system of claim 13, wherein the autonomous navigation is based at least in part on a route that identifies a plurality of fiducials comprising the fiducial, and wherein the navigation state is updated to correct the autonomous navigation according to the route.

15. The system of claim 13, wherein the autonomous navigation is based at least in part on a route that identifies a plurality of fiducials comprising the fiducial, and wherein the instructions when executed with the one or more processors further cause the system to at least:

request second image data at a second location of the unmanned vehicle from the optical sensor, the second image data requested based at least in part on navigation data from the navigation device;

analyze the second image data to identify a second fiducial;

determine whether the second fiducial is included in the plurality of fiducials;

initiate a corrective action associated with the autonomous navigation based at least in part on the second fiducial being excluded from the plurality of fiducials; and

update the navigation state based at least in part on the second fiducial being included in the plurality of fiducials.

16. The system of claim 13, wherein the autonomous navigation is based at least in part on a route that identifies a plurality of fiducials comprising the fiducial, wherein the instructions when executed with the one or more processors further cause the system to at least:

request second image data at a second location of the unmanned vehicle from the optical sensor, the second image data requested based at least in part on an expected fiducial of the plurality of fiducials according to navigation data from the navigation device;

determine that the expected fiducial is excluded from the second image data; and

initiate a corrective action associated with the autonomous navigation based at least in part on the expected fiducial being excluded from the second image data.

17. The system of claim 13, wherein the navigation device receives navigation data from a source external to the unmanned vehicle, wherein the autonomous navigation is based at least in part on a route that identifies the fiducial and a second fiducial, and wherein the instructions when executed with the one or more processors further cause the system to at least:

determine an offset from the navigation data based at least in part on the location of the unmanned vehicle;

determine that the offset exceeds a threshold;

stop updating the navigation state based at least in part on the navigation data;

analyze second image data of the second fiducial; and

direct the autonomous navigation based at least in part on the second image data.

18. The system of claim 13, wherein the navigation device receives navigation data from a source external to the unmanned vehicle, wherein the navigation state is based at least in part on the navigation data, wherein the autonomous navigation is based at least in part on a predetermined route, wherein the first location of the fiducial is associated with an area of the predetermined route having insufficient coverage of the navigation data from the source external to the unmanned vehicle.

19. The system of claim 13, wherein the autonomous navigation is based at least in part on a route that identifies a plurality of fiducials such that at least a field of view of the optical sensor includes at least one fiducial throughout the route.

20. The system of claim 13, wherein the unmanned vehicle comprises an unmanned aerial vehicle (UAV) configured to deliver an item to a specified location within a residential neighborhood.