Selective processing of sensor data

• US 9,778,661 B2
• Filed: 05/27/2016
• Issued: 10/03/2017
• Est. Priority Date: 12/31/2014
• Status: Active Grant

• Current Assignee: SZ DJI Technology Co., Ltd. (d/b/a DJI) (iFlight Technology Company Limited)
• Original Assignee: SZ DJI Technology Co., Ltd. (d/b/a DJI) (iFlight Technology Company Limited)
• Inventors: Wang, Tao, Yan, Jiaqi
• Primary Examiner(s): Amin, Bhavesh V
• Application Number:
  US15/167,154
• Publication Number:
  US 20170045895A1
• Time in Patent Office: 494 Days
• CPC Class Codes: B64C 39/024 of the remote controlled vehicle type, i.e. RPV
• CPC Class Codes: B64D 47/08 Arrangements of cameras
• CPC Class Codes: B64U 10/14 with four distinct rotor axes, e.g. quadcopters
• CPC Class Codes: B64U 2101/00 UAVs specially adapted for particular uses or applications
• CPC Class Codes: B64U 2101/19 for use as targets or decoys
• CPC Class Codes: B64U 2101/30 for imaging, photography or videography
• CPC Class Codes: B64U 2101/31 for surveillance
• CPC Class Codes: B64U 2201/10 autonomous, i.e. by navigating independently from ground or air stations, e.g. by using inertial navigation systems [INS]
• CPC Class Codes: B64U 2201/20 Remote controls
• CPC Class Codes: B64U 30/20 Rotors; Rotor supports
• CPC Class Codes: G01C 11/34 Aerial triangulation
• CPC Class Codes: G01C 21/20 Instruments for performing navigational calculations G01C21/24, G01C21/26 take precedence
• CPC Class Codes: G05D 1/101 specially adapted for aircraft
• CPC Class Codes: G05D 1/106 Change initiated in response to external conditions, e.g. avoidance of elevated terrain or of no-fly zones
• CPC Class Codes: G06T 2207/10032 Satellite or aerial image; Remote sensing
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06V 10/248 by interactive preprocessing or interactive shape modelling, e.g. feature points assigned by a user
• CPC Class Codes: G06V 20/13 Satellite images
• CPC Class Codes: H04N 23/698 : for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 23/76 : by influencing the image signals
• CPC Class Codes: H04N 7/181 : for receiving images from a plurality of remote sources
• CPC Class Codes: H04N 7/185 : from a mobile camera, e.g. for remote control

Abstract

Systems and methods for navigating a vehicle within an environment are provided. In one aspect, a method comprises: (a) selecting, with aid of a processor, a subset of a plurality of sensors to be used for navigating the vehicle within the environment based on one or more predetermined criteria, wherein the plurality of sensors are arranged on the vehicle such that each sensor of the plurality of sensors is configured to obtain sensor data from a different field of view; (b) processing, with aid of the processor, the sensor data from the selected sensor(s) so as to generate navigation information for navigating the vehicle within the environment; and (c) outputting, with aid of the processor, signals for controlling the vehicle based on the navigation information.

Claims

1. A navigation system, comprising:

one or more propulsion units configured to effect movement of a vehicle and a plurality of sensors each configured to capture a plurality of sensor data, wherein the plurality of sensors comprise a plurality of imaging devices each configured to capture a plurality of images; and

one or more processors operably coupled to the plurality of sensors and individually or collectively configured to;

assess data quality of the plurality of sensor data from each of the plurality of sensors, wherein the data quality is assessed based on a feature point number of each image captured by said plurality of imaging devices, wherein said feature point number of each image is calculated using a corner detection algorithm that is a Features from Accelerated Segmented Test (FAST) algorithm;

select at least one of the plurality of sensors based on the assessed data quality; and

determine state information for the vehicle using the plurality of sensor data from the selected sensor(s), wherein the state information of the vehicle comprises information about an environment surrounding the vehicle.

2. The system of claim 1, wherein the one or more processors are further configured to:

output control signals to the one or more propulsion units for effecting the movement of the vehicle, based on the state information of the vehicle.

3. The system of claim 1, wherein the plurality of sensors further comprise one or more of an ultrasonic sensor, a lidar sensor, or a radar sensor.

4. The system of claim 1, wherein the plurality of imaging devices are each oriented in a different direction relative to the vehicle.

5. The system of claim 4, wherein the different directions are orthogonal directions.

6. The system of claim 4, wherein the different directions comprise at least four different directions.

7. The system of claim 1, wherein the data quality is based on a saliency of each image captured by said plurality of imaging devices.

8. The system of claim 1, wherein the data quality is based on at least one of an exposure level or contrast level in each image captured by said plurality of imaging devices.

9. The system of claim 1, wherein the data quality is based on a suitability of said plurality of sensor data for use in determining the state information for the vehicle.

10. The system of claim 1, wherein the one or more processors are configured to assess whether the data quality of said plurality of sensor data exceeds a predetermined threshold.

11. The system of claim 1, wherein the one or more processors are configured to identify which of the plurality of sensor data has the highest data quality.

12. The system of claim 1, wherein the state information further comprises at least one of a position, an attitude, a velocity, or an acceleration of the vehicle.

13. The system of claim 1, wherein the one or more processors are configured to repeat steps to assess the data quality, select at least one of the plurality of sensors, and determine state information during operation of the vehicle.

14. The system of claim 1, wherein the one or more processors are further configured to select at least one of the plurality of sensors based on a power consumption of the plurality of sensors.

15. A navigation system, comprising:

a plurality of imaging devices on a vehicle, wherein each imaging device is configured to capture a plurality of images, wherein the plurality of imaging devices comprise a primary imaging device and one or more secondary imaging devices; and

one or more processors operably coupled to the plurality of imaging devices and individually or collectively configured to;

(1) assess image quality of the plurality of images from the primary imaging device to determine whether said image quality meets a predetermined threshold, and (2) assess image quality of the plurality of images from the one or more secondary imaging devices if the image quality of the plurality of images from the primary imaging device does not meet the predetermined threshold, wherein the image quality of the plurality of images from the primary and/or secondary imaging devices are each assessed based on a feature point number of each image of said plurality of images, wherein said feature point number of each image is calculated using a corner detection algorithm that is a Features from Accelerated Segmented Test (FAST) algorithm;

select at least one of the one or more secondary imaging devices based on the assessment of the image quality of the plurality of images from the one or more secondary imaging devices; and

determine state information for the vehicle using the plurality of images from the selected secondary imaging device(s), wherein the state information of the vehicle comprises information about an environment surrounding the vehicle.

16. The system of claim 15, wherein the vehicle is an unmanned aerial vehicle.

17. The system of claim 15, wherein the plurality of imaging devices are arranged on the vehicle such that each imaging device of the plurality of imaging devices is configured to capture the plurality of images from a different field of view.

18. The system of claim 15, wherein the plurality of images comprises a plurality of successive image frames captured over a predetermined time interval.

19. The system of claim 18, wherein the predetermined time interval is within a range from about 0.02 seconds to about 0.1 seconds.

20. The system of claim 15, wherein the image quality of the plurality of images from the primary and/or secondary imaging devices are each based on saliency of each image of said plurality of images.

21. The system of claim 15, wherein the image quality of the plurality of images from the primary and/or secondary imaging devices are each based on at least one of an exposure level or contrast level in each image of said plurality of images.

22. The system of claim 15, wherein the state information further comprises at least one of a position, an attitude, a velocity, or an acceleration of the vehicle.

23. The system of claim 22, wherein the attitude comprises at least one of a roll orientation, a pitch orientation, or a yaw orientation of the vehicle.

24. A method for controlling a moving vehicle operably coupled thereto a plurality of imaging devices, comprising:

capturing a plurality of images with each imaging device of the plurality of imaging devices; and

with aid of one or more processors, individually or collectively,

assessing image quality of the plurality of images from each imaging device, wherein the image quality is assessed based on a feature point number of each image captured by said plurality of imaging devices, wherein said feature point number of each image is calculated using a corner detection algorithm that is a Features from Accelerated Segmented Test (FAST) algorithm;

selecting at least one of the plurality of imaging devices based on the assessment of the image quality of the plurality of images; and

determining state information for the vehicle using the plurality of images from the selected imaging device(s), wherein the state information of the vehicle comprises information about an environment surrounding the vehicle.

25. A method for assessing state information of a moving vehicle attached thereto a plurality of imaging devices, comprising:

capturing a plurality of images with each imaging device of the plurality of imaging devices, wherein the plurality of imaging devices comprises a primary imaging device and one or more secondary imaging devices; and

with aid of one or more processors, individual or collectively,

(1) assessing image quality of the plurality of images from the primary imaging device to determine whether said image quality meets a predetermined threshold, and (2) assessing image quality of the plurality of images from the one or more secondary imaging device if the image quality of the plurality of images from the primary imaging device does not meet the predetermined threshold, wherein the image quality of the plurality of images from the primary and/or secondary imaging devices are each assessed based on a feature point number of each image of said plurality of images, wherein the feature point number of each image is calculated using a corner detection algorithm that is a Features from Accelerated Segmented Test (FAST) algorithm;

selecting at least one of the one or more secondary imaging devices based on the assessment of the image quality of the plurality of images from the one or more secondary imaging devices; and

determining state information for the vehicle using the plurality of images from the selected secondary imaging device(s), wherein the state information of the vehicle comprises information about an environment surrounding the vehicle.