OBJECT TRACKING BY AN UNMANNED AERIAL VEHICLE USING VISUAL SENSORS

• US 20180158197A1
• Filed: 11/30/2017
• Published: 06/07/2018
• Est. Priority Date: 12/01/2016
• Status: Active Grant

• Current Assignee: Skydio, Inc.
• Original Assignee: Skydio, Inc.
• Inventors: Dasgupta, Saumitro, Martirosyan, Hayk, Koppula, Hema, Kendall, Alex, Stone, Austin, Donahoe, Matthew, Bachrach, Abraham, Bry, Adam
• Granted Patent: US 11,295,458 B2
• Time in Patent Office: Days
• CPC Class Codes: B64C 39/024 of the remote controlled vehicle type, i.e. RPV
• CPC Class Codes: B64U 10/14 with four distinct rotor axes, e.g. quadcopters
• CPC Class Codes: B64U 2101/00 UAVs specially adapted for particular uses or applications
• CPC Class Codes: B64U 2101/30 for imaging, photography or videography
• CPC Class Codes: B64U 2101/31 for surveillance
• CPC Class Codes: B64U 2201/10 autonomous, i.e. by navigating independently from ground or air stations, e.g. by using inertial navigation systems [INS]
• CPC Class Codes: G05D 1/0011 associated with a remote control arrangement
• CPC Class Codes: G05D 1/0094 involving pointing a payload, e.g. camera, weapon, sensor, towards a fixed or moving target
• CPC Class Codes: G06F 18/2431 Multiple classes
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/10032 Satellite or aerial image; Remote sensing
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 2207/20088 Trinocular vision calculations; trifocal tensor
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 2207/30241 Trajectory
• CPC Class Codes: G06T 3/60 Rotation of whole images or parts thereof
• CPC Class Codes: G06T 7/10 Segmentation; Edge detection motion-based segmentation G06T7/215
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/20 : Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/292 : Multi-camera tracking
• CPC Class Codes: G06T 7/579 : from motion
• CPC Class Codes: G06T 7/75 : involving models
• CPC Class Codes: G06V 10/82 : using neural networks
• CPC Class Codes: G06V 20/13 : Satellite images
• CPC Class Codes: G06V 20/17 : taken from planes or by drones
• CPC Class Codes: G06V 30/274 : Syntactic or semantic context, e.g. balancing
• CPC Class Codes: H04N 13/239 : using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/243 : using three or more 2D image sensors
• CPC Class Codes: H04N 13/282 : for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems
• CPC Class Codes: H04N 2013/0081 : Depth or disparity estimation from stereoscopic image signals
• CPC Class Codes: H04N 2013/0085 : Motion estimation from stereoscopic image signals
• CPC Class Codes: H04N 2013/0092 : Image segmentation from stereoscopic image signals

Abstract

Systems and methods are disclosed for tracking objects in a physical environment using visual sensors onboard an autonomous unmanned aerial vehicle (UAV). In certain embodiments, images of the physical environment captured by the onboard visual sensors are processed to extract semantic information about detected objects. Processing of the captured images may involve applying machine learning techniques such as a deep convolutional neural network to extract semantic cues regarding objects detected in the images. The object tracking can be utilized, for example, to facilitate autonomous navigation by the UAV or to generate and display augmentative information regarding tracked objects to users.

Claims

1. A method for tracking objects in a physical environment using an autonomous vehicle based on captured images of the physical environment, the method comprising:

receiving, by a computer system, images of the physical environment, the received images captured by one or more image capture devices coupled to the autonomous vehicle;

processing, by the computer system, the received images to detect one or more objects in the physical environment associated with a particular class of objects;

processing, by the computer system, the received images to distinguish one or more instances of the detected one or more objects;

tracking, by the computer system, a particular object instance of the detected one or more objects; and

generating, by the computer system, an output based on the tracking of the particular object instance.

2. The method of claim 1, wherein the received images are processed using a deep convolutional neural network.

3. The method of claim 1, wherein processing the received images to detect one or more objects associated with the particular class of objects includes:

generating a dense per-pixel segmentation based on the received images, wherein each pixel in the dense per-pixel segmentation is associated with a value indicative of a likelihood that the pixel corresponds with the particular class of objects.

4. The method of claim 3, the dense per-pixel segmentation is one of a plurality of dense per-pixel segmentations comprising a tensor, each of the plurality of dense per-pixel segmentations associated with a different class of objects.

5. The method of claim 3, wherein processing the received images to distinguish one or more instances of the detected one or more objects includes:

analyzing the dense per-pixel segmentation generated based on the received images to associate pixels corresponding to the particular class of objects with a particular instance of the particular class of objects.

6. The method of claim 5, wherein associating pixels corresponding to the particular class of objects with the particular instance of the particular class includes:

applying a grouping process to group;

pixels that are substantially similar to other pixels associated with the particular instance;

pixels that are spatially clustered with other pixels associated with the particular instance; and

/or

pixels that fit an appearance-based model for the particular class of objects.

7. The method of claim 1, further comprising:

processing, by the computer system, the received images to extract semantic information regarding the detected one or more objects;

wherein the tracking is based on the extracted semantic information.

8. The method of claim 7, wherein the semantic information includes information regarding any of the a position, orientation, shape, size, scale, appearance, pixel segmentation, or activity of the detected one or more objects.

9. The method of claim 1, further comprising:

processing, by the computer system, the received images to predict a three-dimensional (3D) trajectory of the particular object instance;

wherein the tracking is based on the predicted 3D trajectory of the particular object instance.

10. The method of claim 1, further comprising:

receiving, by the computer system, sensor data from one or more other sensors coupled to the autonomous vehicle; and

processing, by the computer system, the received sensor data with the received images using a spatiotemporal factor graph to predict a 3D trajectory of the particular object instance;

wherein the tracking is based on the predicted 3D trajectory of the particular object instance.

11. The method of claim 1, wherein generating the output includes:

generating, by the computer system, a planned 3D trajectory of the autonomous vehicle through the physical environment based on the tracking of the particular object instance; and

generating, by the computer system, control commands configured to cause the autonomous vehicle to maneuver along the planned 3D trajectory.

12. The method of claim 1, wherein generating the output includes:

generating, by the computer system, control commands configured to cause a gimbal mechanism to adjust an orientation of the image capture device relative to the autonomous vehicle so as to keep the tracked particular object instance within a field of view of the image capture device.

13. The method of claim 1, wherein generating the output includes:

generating, by the computer system, an augmentation based on the tracking of the particular object instance; and

causing, by the computer system, the generated augmentation to be presented at an augmented reality (AR) device.

14. The method of claim 1, further comprising:

tracking, by the computer system, a second particular object instance while tracking the particular object instance; and

generating, by the computer system, an output based on the tracking of the second particular object instance.

15. The method of claim 1, wherein the autonomous vehicle is an unmanned aerial vehicle (UAV).

16. The method of claim 1, wherein the particular class of objects is selected from a list of classes of objects comprising people, animal, vehicles, buildings, landscape features, and plants.

17. An unmanned aerial vehicle (UAV) configured for autonomous flight through a physical environment, the UAV comprising:

a first image capture device;

a second image capture device; and

a tracking system configured to;

receive images of the physical environment captured by any of the first image capture device or second image capture device;

process the received images to detect one or more objects in the physical environment associated with a particular class of objects;

process the received images to distinguish one or more instances of the detected one or more objects;

track a particular object instance of the detected one or more objects; and

generate an output based on the tracking of the particular object instance.

18. The UAV of claim 17, wherein the received images are processed using a deep convolutional neural network.

19. The UAV of claim 17, wherein processing the received images to detect one or more objects associated with the particular class of objects includes:

generating a dense per-pixel segmentation of the received image, wherein each pixel in the received image is associated with a value indicative of a likelihood that the pixel corresponds with the particular class of objects.

20. The UAV of claim 17, wherein processing the received images to distinguish one or more instances of the detected one or more objects includes:

analyzing the dense per-pixel segmentation generated based on the received images to associate pixels corresponding to the particular class of objects with a particular instance of the particular class of objects.

21. The UAV of claim 20, wherein associating pixels corresponding to the particular class of objects with the particular instance of the particular class includes:

applying a grouping process to group;

pixels that are substantially similar to other pixels associated with the particular instance;

pixels that are spatially clustered with other pixels associated with the particular instance; and

/or

pixels that fit an appearance-based model for the particular class of objects.

22. The UAV of claim 17, wherein the tracking system is further configured to:

process the received images to extract semantic information regarding the detected one or more objects;

wherein the semantic information includes information regarding any of the a position, orientation, shape, size, scale, appearance, pixel segmentation, or activity of the detected one or more objects; and

wherein the tracking is based on the extracted semantic information.

23. The UAV of claim 17, wherein the tracking system is further configured to:

process the received images to predict a three-dimensional (3D) trajectory of the particular object instance;

wherein the tracking is based on the predicted 3D trajectory of the particular object instance.

24. The UAV of claim 17, further comprising:

a motion planning system configured to generate a planned 3D trajectory of the UAV through the physical environment;

wherein the tracking system is further configured to communicate the generated output based on the tracking of the particular object instance to the motion planning system to facilitate generating the planned 3D trajectory.

25. The UAV of claim 24, further comprising:

a propulsion system; and

a flight controller configured to;

receive the planned 3D trajectory from the motion planning system; and

generate control commands configured to cause the propulsion system to maneuver the UAV along the planned 3D trajectory.

26. The UAV of claim 17, wherein the first image capture device includes an array of stereoscopic cameras arranged along a perimeter of the UAV so as to provide a stereoscopic view at a plurality of angles around the UAV.

27. The UAV of claim 17, wherein the second image capture device is mounted to the UAV via a gimbal mechanism, the gimbal mechanism enabling the camera to adjust orientation relative to the UAV.

28. The UAV of claim 27, wherein the gimbal mechanism is a hybrid mechanical-digital gimbal system, the hybrid mechanical-digital gimbal system including:

mechanical gimbal configured to rotate the second image capture device about a first axis relative to the UAV; and

an image processing system configured to rotate images captured by the second image capture device about a second axis relative to the UAV.

29. The UAV of claim 27, wherein generating the output based on the tracking of the particular object instance includes:

generating control commands configured to cause the gimbal mechanism to adjust an orientation of the second image capture device relative to the UAV so as to keep the tracked particular object instance within a field of view of the second image capture device.

30. An unmanned aerial vehicle (UAV) comprising:

one or more image capture devices;

an object detection system configured to detect one or more objects in the physical environment associated with a class of objects by generating a dense per-pixel segmentation based on images received from the one or more image capture devices;

an instance segmentation system configured to analyze the dense per-pixel segmentation to associate sets of pixels corresponding to one or more instances of the class of objects;

an object tracking system configured to track one or more instances of the class of objects and generate an output based on the tracking.

31. The UAV of claim 30, wherein the object detection system and/or instance segmentation system are configured to apply a deep convolutional neural network when processing images received from the one or more image capture devices.

32. The UAV of claim 30, wherein each pixel in the dense per-pixel segmentation is associated with a value indicative of a likelihood that the pixel corresponds with the particular class of objects.

33. The UAV of claim 30, wherein the instance segmentation system is configured to associate sets of pixels corresponding to the one or more instances of the class of objects by grouping:

pixels that are substantially similar to other pixels associated with a particular instance;

pixels that are spatially clustered with other pixels associated with the particular instance; and

/or

pixels that fit an appearance-based model for the class of objects.

34. The UAV of claim 30, further comprising:

an identity recognition system configured to;

learn appearances of a plurality of instances of the class of objects by processing images received from the one or more image capture devices;

generate one or more appearance models based on the learned appearances;

compare the sets of pixels associated with the one or more instances with the generated appearance models; and

resolve unique identities for one or more of instances of the class of objects.

35. The UAV of claim 30, wherein the tracking system is configured to track the one or more instances of the class of objects by:

extracting semantic information regarding the one or more instances of the class of objects based on the images received from the one or more image capture devices;

receiving sensor data from one or more other non-visual sensors, the sensor data indicative of any of a position, orientation, or motion of the one or more instances of the class of objects in the physical environment ; and

predicting three-dimensional (3D) trajectories for the one or more instances of the class of objects by processing the extracted semantic information and received sensor data using a spatiotemporal factor graph.

36. The UAV of claim 30, further comprising:

a motion planning system configured to generate a planned 3D trajectory of the UAV through the physical environment based on the output generated from the tracking system.

37. The UAV of claim 36, further comprising:

a propulsion system; and

a flight controller configured to generate control commands configured to cause the propulsion system to maneuver the UAV along the planned 3D trajectory.