Determining position or orientation relative to a marker

• US 10,254,767 B1
• Filed: 01/25/2017
• Issued: 04/09/2019
• Est. Priority Date: 01/25/2017
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Kamon, Ishay
• Primary Examiner(s): Do, Truc M
• Application Number:
  US15/415,180
• Time in Patent Office: 804 Days
• Field of Search: None
• CPC Class Codes: B64D 43/00 Arrangements or adaptations of instruments arrangements of cameras B64D47/08; aeronautical measuring instruments per se G01C
• CPC Class Codes: B64U 2101/69 the UAVs provided with means for airdropping goods, e.g. deploying a parachute during descent
• CPC Class Codes: B64U 2201/10 autonomous, i.e. by navigating independently from ground or air stations, e.g. by using inertial navigation systems [INS]
• CPC Class Codes: B64U 40/10 for adjusting control surfaces or rotors
• CPC Class Codes: B64U 70/97 Means for guiding the UAV to a specific location on the platform, e.g. platform structures preventing landing off-centre
• CPC Class Codes: G01S 1/00 Beacons or beacon systems transmitting signals having a characteristic or characteristics capable of being detected by non-directional receivers and defining directions, positions, or position lines fixed relatively to the beacon transmitters; Receivers co-operating therewith position fixing by co-ordinating a plurality of determinations of direction or position lines G01S5/00
• CPC Class Codes: G05D 1/102 specially adapted for vertical take-off of aircraft
• CPC Class Codes: G06T 2207/10032 Satellite or aerial image; Remote sensing
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06V 10/225 based on a marking or identifier characterising the area
• CPC Class Codes: G06V 20/13 Satellite images
• CPC Class Codes: G06V 20/17 taken from planes or by drones

Abstract

A system may include first and second sensors configured to be coupled to a vehicle and generate surface sensor signals representative of a surface on which a location marker is disposed, and generate marker sensor signals representative of the location marker. The system may also include a sensor processor configured to estimate at least one of a position or an orientation of the first sensor relative to the surface on which the location marker is disposed based at least in part on the surface sensor signals, and estimate at least one of a position or an orientation of the second sensor relative to the location marker based at least in part on the marker sensor signals. The sensor processor may be configured to calculate at least one of the position or the orientation of the vehicle relative to the location marker based at least in part on the estimations.

Claims

1. A system to estimate at least one of a position or an orientation of a vehicle relative to a location marker, the system comprising:

a first sensor configured to be coupled to the vehicle and to generate surface sensor signals representative of a surface on which the location marker is disposed;

a second sensor configured to be coupled to the vehicle and to generate marker sensor signals representative of the location marker; and

a sensor processor in communication with the first sensor and the second sensor and configured to;

estimate at least one of a position of the first sensor relative to the surface on which the location marker is disposed or an orientation of the first sensor relative to the surface on which the location marker is disposed based at least in part on the surface sensor signals received from the first sensor;

estimate at least one of a position of the second sensor relative to the location marker or an orientation of the second sensor relative to the location marker based at least in part on the marker sensor signals received from the second sensor;

identify at least one of an edge of the location marker or a corner of the location marker based at least in part on the marker sensor signals received from the second sensor;

associate the at least one of the edge of the location marker or the corner of the location marker with the at least one of the position of the first sensor relative to the surface on which the location marker is disposed or the orientation of the first sensor relative to the surface on which the location marker is disposed; and

calculate at least one of the position of the vehicle relative to the location marker or the orientation of the vehicle relative to the location marker based at least in part on (i) the at least one of the position of the first sensor relative to the surface or the orientation of the first sensor relative to the surface, and (ii) the at least one of the position of the second sensor relative to the location marker or the orientation of the second sensor relative to the location marker.

2. The system as recited in claim 1, wherein the system is implemented on an unmanned aerial vehicle (UAV), and the system further comprises:

one or more processors; and

memory to store a vehicle controller, the vehicle controller being executable by the one or more processors to control operation of the UAV based at least in part on at least one of the position of the vehicle relative to the location marker or the orientation of the vehicle relative to the location marker.

3. The system as recited in claim 1, wherein the first sensor comprises a first imager and the second sensor comprises a second imager, and the sensor processor comprises an image processor configured to calculate the at least one of the position of the vehicle relative to the location marker or the orientation of the vehicle relative to the location marker based at least in part on the surface sensor signals representative of the surface on which the location marker is disposed.

4. The system as recited in claim 3, wherein the first imager and the second imager form at least a portion of a stereo camera system configured to generate signals representative of three-dimensional space.

5. The system as recited in claim 1, wherein the sensor processor is configured to:

estimate a virtual marker plane in which a virtual marker lies;

compare a virtual surface plane with the virtual marker plane; and

when the virtual surface plane and the virtual marker plane are coexistent, discontinue using the at least one of the position of the sensor relative to the surface or the orientation of the sensor relative to the surface to calculate the at least one of the position of the sensor relative to the location marker or the orientation of the sensor relative to the location marker.

6. A method comprising:

receiving surface sensor signals representative of the surface on which a marker is disposed from a sensor;

estimating, based at least in part on the surface sensor signals, at least one of a position of the sensor relative to the surface on which the marker is disposed or an orientation of the sensor relative to the surface on which the marker is disposed;

receiving marker sensor signals representative of at least one of a position of the marker relative to the sensor or an orientation of the marker relative to the sensor;

identifying, based at least in part on the marker sensor signals, at least one of an edge of the marker or a corner of the marker;

associating the at least one of the edge of the marker or the corner of the marker with the at least one of the position of the sensor relative to the surface on which the marker is disposed or the orientation of the sensor relative to the surface on which the marker is disposed;

calculating at least one of the position of the sensor relative to the marker or the orientation of the sensor relative to the marker based at least in part on the at least one of the edge of the marker or the corner of the marker and the at least one of the position of the sensor relative to the surface or the orientation of the sensor relative to the surface; and

calculating at least one of the position of the sensor relative to the marker or the orientation of the sensor relative to the marker based at least in part on the at least one of the position of the sensor relative to the surface or the orientation of the sensor relative to the surface and the marker sensor signals representative of at least one of the position of the marker relative to the sensor or the orientation of the marker relative to the sensor.

7. The method as recited in claim 6, wherein the sensor comprises one of a first sensor or a second sensor, and wherein receiving the surface sensor signals representative of the surface comprises receiving the surface sensor signals from the first sensor, receiving the marker sensor signals representative of the marker comprises receiving the marker sensor signals from the second sensor, and calculating at least one of the position of the sensor relative to the marker or the orientation of the sensor relative to the marker comprises calculating at least one of the position or the orientation of at least one of the first sensor relative to the surface or the second sensor relative to the marker.

8. The method as recited in claim 7, wherein the surface sensor signals and the marker sensor signals comprise sensor signals representative of respective images of the surface and images of the marker, and calculating the at least one of the position of the sensor relative to the marker or the orientation of the sensor relative to the marker is based at least in part on the sensor signals representative of the images of the surface and the sensor signals representative of the images of the marker.

9. The method as recited in claim 7, wherein the first sensor comprises a first imager, and receiving the surface sensor signals comprises receiving first images from the first imager.

10. The method as recited in claim 9, wherein the second sensor comprises a second imager, and receiving the marker sensor signals comprises receiving second images from the second imager.

11. The method as recited in claim 7, further comprising:

generating a virtual surface plane based at least in part on the surface sensor signals;

generating a virtual marker based at least in part on the marker sensor signals;

projecting the virtual marker onto the virtual surface plane such that at least one of the edge of the marker or the corner of the marker lies in the virtual surface plane; and

calculating at least one of the position or the orientation of at least one of the first sensor relative to the marker or the second sensor relative to the marker based at least in part on the projection of the virtual marker onto the virtual surface plane.

12. The method as recited in claim 11, further comprising:

estimating a virtual marker plane in which the virtual marker lies;

comparing the virtual surface plane with the virtual marker plane; and

when the virtual surface plane and the virtual marker plane are coexistent, discontinuing using the at least one of the position of the sensor relative to the surface or the orientation of the sensor relative to the surface to calculate the at least one of the position of the sensor relative to the marker or the orientation of the sensor relative to the marker.

13. The method as recited in claim 6, wherein the sensor is coupled to a vehicle and the method further comprises:

calculating at least one of a position of the vehicle relative to the marker or an orientation of the vehicle relative to the marker based at least in part on the at least one of the position of the sensor relative to the marker or the orientation of the sensor relative to the marker; and

controlling the vehicle based at least in part on the at least one of the position of the vehicle relative to the marker or the orientation of the vehicle relative to the marker.

14. The method as recited in claim 13, wherein controlling the vehicle comprises landing the vehicle on the marker.

15. A vehicle comprising:

a frame;

a propulsion system;

at least one sensor coupled to the frame and configured to generate (i) surface sensor signals representative of a surface on which a marker is disposed, the marker being associated with a destination of the vehicle, and (ii) marker sensor signals representative of the marker; and

a sensor processor in communication with the at least one sensor and configured to;

estimate at least one of a position or an orientation of the at least one sensor relative to the surface on which the marker is disposed based at least in part on the surface sensor signals;

estimate at least one of a position or an orientation of the at least one sensor relative to the marker based at least in part on the marker sensor signals;

identify at least one of an edge of the marker or a corner of the marker based at least in part on the marker sensor signals;

associate at least one of the edge of the marker or the corner of the marker with the at least one of the position or the orientation of the at least one sensor relative to the surface on which the marker is disposed; and

calculate at least one of the position or the orientation of the vehicle relative to the marker based at least in part on the at least one of the position or the orientation of the at least one sensor relative to the surface and the at least one of the position or the orientation of the at least one sensor relative to the marker.

16. The vehicle as recited in claim 15, wherein the at least one sensor comprises a first imager and a second imager, and the sensor processor comprises an image processor configured to calculate the at least one of the position or the orientation of the vehicle relative to the marker based at least in part on the surface sensor signals representative of the surface on which the marker is disposed.

17. The vehicle as recited in claim 16, wherein the first imager and the second imager form at least a portion of a stereo camera system configured to generate signals representative of three-dimensional space.

18. The vehicle as recited in claim 15, wherein the sensor processor is configured to align a virtual representation of the at least one of the edge of the marker or the corner of the marker with a virtual surface representative of the surface on which the marker is disposed.

19. The vehicle as recited in claim 18, wherein the sensor processor is configured to discontinue aligning the virtual representation with the virtual surface as the vehicle approaches the marker.

20. The vehicle as recited in claim 19, wherein the sensor processor is configured to:

generate a virtual surface plane based at least in part on the surface sensor signals;

generate a virtual marker based at least in part on the marker sensor signals, the virtual marker comprising at least one of the edge of the marker or the corner of the marker;

project the virtual marker onto the virtual surface plane such that at least one of the edge of the marker or the corner of the marker lies in the virtual surface plane; and

calculate at least one of the position or the orientation of the vehicle relative to the marker based at least in part on the projection of the virtual marker onto the virtual surface plane.