System and method for mobile platform operation

• US 10,048,687 B2
• Filed: 06/09/2016
• Issued: 08/14/2018
• Est. Priority Date: 03/31/2015
• Status: Active Grant

• Current Assignee: SZ DJI Technology Co., Ltd. (d/b/a DJI) (iFlight Technology Company Limited)
• Original Assignee: SZ DJI Technology Co., Ltd. (d/b/a DJI) (iFlight Technology Company Limited)
• Inventors: Zhao, Cong, Tang, Ketan, Zhou, Guyue
• Primary Examiner(s): Mawari, Redhwan K
• Assistant Examiner(s): King, Rodney P
• Application Number:
  US15/178,268
• Publication Number:
  US 20160291594A1
• Time in Patent Office: 796 Days
• CPC Class Codes: G01S 15/86 Combinations of sonar systems with lidar systems; Combinations of sonar systems with systems not using wave reflection
• CPC Class Codes: G01S 15/93 for anti-collision purposes
• CPC Class Codes: G01S 17/36 with phase comparison between the received signal and the contemporaneously transmitted signal
• CPC Class Codes: G01S 17/58 Velocity or trajectory determination systems; Sense-of-movement determination systems
• CPC Class Codes: G01S 17/86 Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
• CPC Class Codes: G01S 17/894 3D imaging with simultaneous measurement of time-of-flight at a 2D array of receiver pixels, e.g. time-of-flight cameras or flash lidar
• CPC Class Codes: G01S 17/933 of aircraft or spacecraft
• CPC Class Codes: G01S 7/4808 Evaluating distance, position or velocity data
• CPC Class Codes: G01S 7/497 Means for monitoring or calibrating
• CPC Class Codes: G05D 1/0088 characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours using knowledge based models G06N5/00
• CPC Class Codes: G05D 1/0094 involving pointing a payload, e.g. camera, weapon, sensor, towards a fixed or moving target
• CPC Class Codes: G05D 1/0255 using acoustic signals, e.g. ultra-sonic singals sonar systems designed for anti-collision purposes G01S15/93
• CPC Class Codes: G05D 1/101 specially adapted for aircraft
• CPC Class Codes: G05D 1/102 specially adapted for vertical take-off of aircraft

Abstract

A system for operating a mobile platform using distance and speed information and methods for making and using same. The system includes a time-of-flight sensor and an ultrasound sensor for measuring a distance between the mobile platform and an obstacle and a processor for integrating the sensed measurements and controlling the mobile platform. The distance measured by the time-of-flight sensor can be determined via a phase shift method. The time-of-flight sensor can also be used to measure a speed of the mobile platform by imaging a reference point at different times and using stereopsis to ascertain the displacement. Distances and speeds measured using the time-of-flight and ultrasound sensors can be integrated to improve measurement accuracy. The systems and methods are suitable for use in controlling any type of mobile platform, including unmanned aerial vehicles and advantageously can be applied for avoiding collisions between the mobile platform and the obstacle.

Claims

1. A method for operating a mobile platform, comprising:

measuring a distance between the mobile platform and an obstacle via time-of-flight sensing or ultrasound sensing, including;

performing, by a time-of-flight sensor, a first measurement on the distance via time-of-flight sensing;

performing, by an ultrasound sensor, a second measurement on the distance via ultrasound sensing;

determining, by a processor, at least one of the first measurement or the second measurement is determinate;

determining, by a processor, the distance based on the at least one of the first measurement or the second measurement that is determinate;

measuring, by a sensor, a speed of the mobile platform; and

controlling, by a processor, a movement of the mobile platform according to the measured distance and the measured speed, a controlling method for controlling the movement of the mobile platform depending on which one or both of the time-of-flight sensor and the ultrasound sensor are used for measuring the distance.

2. The method of claim 1, wherein said measuring the distance via time-of-flight sensing comprises measuring the distance via a phase-shift process.

3. The method of claim 2, wherein said measuring the distance via the phase-shift process comprises:

emitting a light wave toward the obstacle;

detecting the light wave after reflection by the obstacle as a reflected light wave;

measuring a phase shift between the emitted light wave and the reflected light wave; and

determining the distance according to the phase shift.

4. The method of claim 1, wherein said measuring the distance via time-of-flight sensing comprises:

generating a time-of-flight depth map of an operating environment of the mobile platform; and

measuring the distance according to the time-of-flight depth map.

5. The method of claim 1, wherein:

both the first measurement and the second measurement are determinate, and

said measuring the distance further comprises;

determining the distance as being the smaller of the first measurement and the second measurement.

6. The method of claim 1, wherein:

both the first measurement and the second measurement are determinate, and

said measuring the distance further comprises;

determining an ambient light level adjacent to the mobile platform; and

determining the distance by weighting the first measurement relative to the second measurement according to the ambient light level.

7. The method of claim 6, wherein said determining the ambient lighting comprises determining the ambient lighting via time-of-flight sensing.

8. The method of claim 7, wherein determining the ambient lighting via time-of-flight sensing comprises determining a constant bias of a cross-correlation between an emitted light wave and a corresponding received light wave.

9. The method of claim 1, wherein said measuring the distance further comprises:

generating a time-of-flight depth map of surroundings of the mobile platform;

generating an ultrasound depth map of the surroundings;

combining the time-of-flight depth map and the ultrasound depth map into a combined depth map; and

determining the distance via the combined depth map.

10. The method of claim 1, wherein said measuring the speed comprises measuring the speed via stereopsis.

11. The method of claim 10, wherein said measuring the speed via stereopsis comprises:

acquiring a first image of an operating environment of the mobile platform;

acquiring a second image of the operating environment subsequent to said acquiring the first image; and

measuring the speed based on the first image and the second image.

12. The method of claim 11, wherein:

said acquiring the first image comprises acquiring a first gray-scale image map via time-of-flight sensing; and

said acquiring the second image comprises acquiring a second gray-scale image map via time-of-flight sensing.

13. The method of claim 11, wherein said measuring the speed based on the first image and the second image comprises:

determining a disparity between the first image and the second image;

determining a displacement between the first image and the second image according to the disparity; and

determining the speed according to the displacement.

14. The method of claim 13, wherein said determining the disparity comprises:

selecting a plurality of reference points of the first image;

locating a plurality of points of the second image corresponding to the reference points; and

determining the disparity according to position changes of the reference points.

15. The method of claim 1, wherein said measuring the speed comprises measuring the speed via an inertial measurement unit.

16. The method of claim 1, wherein said controlling the mobile platform comprises automatically adjusting a speed of the mobile platform to avoid the obstacle according the measured distance and the measured speed.

17. A system for operating a mobile platform, comprising:

a time-of-flight sensor for sensing an operating environment of the mobile platform;

an ultrasound sensor for sensing the operating environment; and

a processor configured to;

determine a distance between the mobile platform and an obstacle in the operating environment using at least one of the time-of-flight sensor and the ultrasound sensor, including;

performing, by a time-of-flight sensor, a first measurement on the distance via time-of-flight sensing;

performing, by an ultrasound sensor, a second measurement on the distance via ultrasound sensing;

determining, by a processor, at least one of the first measurement or the second measurement is determinate;

determining, by a processor, the distance based on the at least one of the first measurement or the second measurement that is determinate; and

control a movement of the mobile platform according to the measured distance and a speed of the mobile platform, a controlling method for controlling the movement of the mobile platform depending on which one or both of the time-of-flight sensor and the ultrasound sensor are used for measuring the distance.

18. The system of claim 17, wherein said processor is configured to determine the distance via a phase-shift process.

19. The system of claim 17, wherein said processor is configured to determine the distance as being the smaller of a time-of-flight measurement and an ultrasound measurement.

20. The system of claim 17, wherein:

both the first measurement and the second measurement are determinate, and

said processor is configured to determine the distance by;

determining an ambient light level in the operating environment; and

determining the distance by weighting the first measurement relative to the second measurement according to the ambient light level.

21. The system of claim 17, wherein said processor is configured to measure the distance by:

generating a time-of-flight depth map of the operating environment;

generating an ultrasound depth map of the operating environment;

combining the time-of-flight depth map and the ultrasound depth map into a combined depth map; and

determining the distance via the combined depth map.

22. The system of claim 17, wherein said processor is configured to measure the speed via stereopsis.

23. The system of claim 22, wherein said processor is configured to measure the speed via stereopsis by:

acquiring a first image of the operating environment;

acquiring a second image of the operating environment subsequent to said acquiring the first image; and

detecting the speed based on the first image and the second image.

24. The system of claim 23, wherein said processor is configured to measure the speed by:

determining a disparity between the first image and the second image;

determining a displacement between the first image and the second image according to the disparity; and

determining the speed according to the displacement.

25. The method of claim 24, wherein said processor is configured to determine the disparity by:

selecting a plurality of reference points of the first image;

locating a plurality of points of the second image corresponding to the reference points; and

determining the disparity according to position changes of the reference points.

26. The system of claim 17, wherein said processor is configured to measure the speed via an inertial measurement unit mounted aboard the mobile platform.

27. The system of claim 17, wherein the time-of-flight sensor and the ultrasound sensor are mounted aboard the mobile platform.

28. The system of claim 17, wherein the mobile platform is an unmanned aerial vehicle.

29. An apparatus, comprising:

one or more processors configured for;

determining a distance between a mobile platform and an obstacle via time-of-flight sensing or ultrasound sensing, including;

performing, by a time-of-flight sensor, a first measurement on the distance via time-of-flight sensing;

performing, by an ultrasound sensor, a second measurement on the distance via ultrasound sensing;

determining, by a processor, at least one of the first measurement or the second measurement is determinate;

determining, by a processor, the distance based on the at least one of the first measurement or the second measurement that is determinate;

determining a speed of the mobile platform via stereopsis; and

automatically adjusting the speed of the mobile platform to avoid the obstacle according the determined distance and the determined speed, a controlling method for controlling a movement of the mobile platform depending on which one or both of the time-of-flight sensor and the ultrasound sensor are used for measuring the distance.

30. A non-transitory readable medium with instructions stored thereon that, when executed by a processor, perform the steps comprising:

determining a distance between a mobile platform and an obstacle via time-of-flight sensing or ultrasound sensing, including;

performing, by a time-of-flight sensor, a first measurement on the distance via time-of-flight sensing;

performing, by an ultrasound sensor, a second measurement on the distance via ultrasound sensing;

determining, by a processor, at least one of the first measurement or the second measurement is determinate;

determining, by a processor, the distance based on the at least one of the first measurement or the second measurement that is determinate;

determining a speed of the mobile platform; and

controlling a movement of the mobile platform according to the determined distance and the determined speed, a controlling method for controlling the movement of the mobile platform depending on which one or both of the time-of-flight sensor and the ultrasound sensor are used for measuring the distance.