Collaborative airborne object tracking systems and methods

• US 10,510,158 B1
• Filed: 11/13/2017
• Issued: 12/17/2019
• Est. Priority Date: 11/13/2017
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Kamon, Ishay
• Primary Examiner(s): Mehta, Bhavesh M
• Assistant Examiner(s): Lemieux, Ian L
• Application Number:
  US15/811,557
• Time in Patent Office: 764 Days
• CPC Class Codes: B64C 39/024 of the remote controlled vehicle type, i.e. RPV
• CPC Class Codes: B64D 47/08 Arrangements of cameras
• CPC Class Codes: B64U 10/13 Flying platforms
• CPC Class Codes: B64U 2101/30 for imaging, photography or videography
• CPC Class Codes: B64U 2201/00 UAVs characterised by their flight controls
• CPC Class Codes: B64U 2201/10 autonomous, i.e. by navigating independently from ground or air stations, e.g. by using inertial navigation systems [INS]
• CPC Class Codes: G06T 2200/28 involving image processing hardware
• CPC Class Codes: G06T 2207/10032 Satellite or aerial image; Remote sensing
• CPC Class Codes: G06T 2207/30232 Surveillance
• CPC Class Codes: G06T 2207/30261 Obstacle
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: H04N 23/61 based on recognised objects
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums

Abstract

Systems and methods related to identifying locations and/or ranges to objects using airborne imaging devices are disclosed. An object tracking system may include a plurality of aerial vehicles having associated imaging devices and a control station. Information related to positions and orientations of aerial vehicles and associated imaging devices may be received. In addition, imaging data may be received and processed to identify optical rays associated with objects within the imaging data. Further, a three-dimensional mapping of the identified optical rays may be generated, and locations or ranges of the objects relative to the aerial vehicles may be determined based on any intersections of optical rays within the three-dimensional mapping.

Claims

1. A computer-implemented method to track airborne objects, comprising:

receiving, from a first aerial vehicle of a plurality of aerial vehicles;

first location information associated with the first aerial vehicle;

first pose information associated with the first aerial vehicle;

first optical axis information associated with a first imaging device of the first aerial vehicle; and

first imaging data from the first imaging device of the first aerial vehicle;

identifying a first object within the first imaging data;

determining a first pixel location of the first object within the first imaging data;

determining a first optical ray associated with the first pixel location of the first object;

receiving, from a second aerial vehicle of the plurality of aerial vehicles;

second location information associated with the second aerial vehicle;

second pose information associated with the second aerial vehicle;

second optical axis information associated with a second imaging device of the second aerial vehicle; and

second imaging data from the second imaging device of the second aerial vehicle;

identifying a second object within the second imaging data;

determining a second pixel location of the second object within the second imaging data;

determining a second optical ray associated with the second pixel location of the second object;

determining that the first optical ray and the second optical ray intersect with a threshold degree of confidence, such that the first object and the second object are a same object;

determining at least one of a location or a range of the same object relative to at least one of the first aerial vehicle or the second aerial vehicle; and

instructing the at least one of the first aerial vehicle or the second aerial vehicle to navigate to avoid the same object based at least in part on the determined at least one of the location or the range of the same object relative to the at least one of the first aerial vehicle or the second aerial vehicle.

2. The computer-implemented method of claim 1, wherein the first location information includes first latitude, longitude, and altitude information, and the second location information includes second latitude, longitude, and altitude information.

3. The computer-implemented method of claim 1, wherein the first pose information includes first roll, pitch, and yaw information, and the second pose information includes second, roll, pitch, and yaw information.

4. The computer-implemented method of claim 1, wherein the first optical axis information includes a first direction of a first optical axis of the first imaging device relative to a first frame of the first aerial vehicle, and the second optical axis information includes a second direction of a second optical axis of the second imaging device relative to a second frame of the second aerial vehicle.

5. A computer-implemented method to track objects, comprising:

receiving, from a plurality of aerial vehicles, imaging data from respective imaging devices of individual aerial vehicles;

identifying objects within the imaging data;

determining pixel locations of the objects within the imaging data;

determining optical rays associated with the pixel locations of the objects;

determining that at least two optical rays intersect with a threshold degree of confidence based on at least one of location, pose, or optical axis information associated with individual aerial vehicles;

determining that at least two objects associated with the at least two intersecting optical rays are a same object;

determining at least one of a location or a range of the same object relative to at least one of the plurality of aerial vehicles; and

instructing the at least one of the plurality of aerial vehicles to navigate to avoid the same object based at least in part on the determined at least one of the location or the range of the same object relative to the at least one of the plurality of aerial vehicles.

6. The computer-implemented method of claim 5, further comprising:

receiving, from the plurality of aerial vehicles, the at least one of location, pose, or optical axis information associated with individual aerial vehicles.

7. The computer-implemented method of claim 6, wherein the location information includes latitude, longitude, and altitude information received from at least one of a global positioning system (GPS) sensor or an altimeter associated with individual aerial vehicles.

8. The computer-implemented method of claim 6, wherein the pose information includes roll, pitch, and yaw information received from at least one of an inertial measurement unit, an accelerometer, or a gyroscope associated with individual aerial vehicles.

9. The computer-implemented method of claim 6, wherein the optical axis information includes directions of fixed optical axes associated with respective imaging devices of individual aerial vehicles.

10. The computer-implemented method of claim 6, wherein the optical axis information includes directions of movable optical axes received from at least one of a motor controller or a position encoder associated with respective imaging devices of individual aerial vehicles.

11. The computer-implemented method of claim 5, wherein the determining pixel locations of the objects within the imaging data further comprises:

determining a pixel location associated with an object identified within the imaging data received from a respective imaging device relative to an optical axis associated with the respective imaging device.

12. The computer-implemented method of claim 5, wherein the determining optical rays associated with the pixel locations of the objects further comprises:

determining an optical ray associated with a pixel location of an object identified within the imaging data received from a respective imaging device based on at least one characteristic of the respective imaging device.

13. The computer-implemented method of claim 5, wherein the determining that at least two optical rays intersect with a threshold degree of confidence based on at least one of location, pose, or optical axis information associated with individual aerial vehicles further comprises:

generating a three-dimensional mapping of the optical rays based on the at least one of location, pose, or optical axis information associated with individual aerial vehicles; and

determining that the at least two optical rays intersect with the threshold degree of confidence based on the three-dimensional mapping.

14. The computer-implemented method of claim 13, wherein the determining at least one of a location or a range of the same object relative to at least one of the plurality of aerial vehicles further comprises:

determining the at least one of the location or the range of the same object relative to the at least one of the plurality of aerial vehicles based on the three-dimensional mapping.

15. A system to track objects, comprising:

a plurality of aerial vehicles having respective imaging devices; and

a control system in communication with the plurality of aerial vehicles and having a processor configured to at least;

receive, from the plurality of aerial vehicles, imaging data from respective imaging devices of individual aerial vehicles;

identify objects within the imaging data;

determine pixel locations of the objects within the imaging data;

determine optical rays associated with the pixel locations of the objects;

determine that at least two optical rays intersect with a threshold degree of confidence based on at least one of location, pose, or optical axis information associated with individual aerial vehicles;

determine that at least two objects associated with the at least two intersecting optical rays are a same object;

determine at least one of a location or a range of the same object relative to at least one of the plurality of aerial vehicles; and

instruct the at least one of the plurality of aerial vehicles to navigate to avoid the same object based at least in part on the determined at least one of the location or the range of the same object relative to the at least one of the plurality of aerial vehicles.

16. The system of claim 15, wherein the control system is at least one of a ground-based control system, an air-based control system, or a water-based control system.

17. The system of claim 15, wherein the control system is at least one of fixed or mobile.

18. The system of claim 15, wherein the control system is comprised in at least one of the plurality of aerial vehicles.

19. The system of claim 15, wherein the control system includes at least one of an image processing module or a three-dimensional mapping module.