Multi-sensor environmental mapping

• US 10,429,839 B2
• Filed: 07/30/2015
• Issued: 10/01/2019
• Est. Priority Date: 09/05/2014
• Status: Expired due to Fees

• Current Assignee: SZ DJI Technology Co., Ltd. (d/b/a DJI) (iFlight Technology Company Limited)
• Original Assignee: SZ DJI Technology Co., Ltd. (d/b/a DJI) (iFlight Technology Company Limited)
• Inventors: Liu, Ang, Mo, Weiyu, Ling, Yonggen
• Primary Examiner(s): Amin, Bhavesh V
• Application Number:
  US14/814,159
• Publication Number:
  US 20160070265A1
• Time in Patent Office: 1,524 Days
• CPC Class Codes: B64C 39/024 of the remote controlled vehicle type, i.e. RPV
• CPC Class Codes: B64U 2201/10 autonomous, i.e. by navigating independently from ground or air stations, e.g. by using inertial navigation systems [INS]
• CPC Class Codes: G01C 21/28 with correlation of data from several navigational instruments
• CPC Class Codes: G01C 21/3848 Data obtained from both position sensors and additional sensors
• CPC Class Codes: G01S 19/42 Determining position
• CPC Class Codes: G05D 1/0088 characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours using knowledge based models G06N5/00
• CPC Class Codes: G05D 1/102 specially adapted for vertical take-off of aircraft
• CPC Class Codes: G05D 1/228 Command input arrangements located on-board unmanned vehicles
• CPC Class Codes: G05D 1/652 Take-off delivering or retrieving payloads G05D1/667

Abstract

Systems and methods for controlling a movable object within an environment are provided. In one aspect, a method may comprise: determining, using at least one of a plurality of sensors carried by the movable object, an initial location of the movable object; generating a first signal to cause the movable object to navigate within the environment; receiving, using the at least one of the plurality of sensors, sensing data pertaining to the environment; generating, based on the sensing data, an environmental map representative of at least a portion of the environment; receiving an instruction to return to the initial location; and generating a second signal to cause the movable object to return to the initial location, based on the environmental map.

Claims

1. A method for controlling an unmanned aerial vehicle within an environment, the method comprising:

determining, using at least one of a plurality of sensors carried by the unmanned aerial vehicle, an initial location of the unmanned aerial vehicle, wherein the plurality of sensors comprise (1) at least one vision sensor configured to capture image data to generate a first sensing signal and (2) at least one proximity sensor configured to obtain distance data to generate a second sensing signal; and

with aid of one or more processors onboard the unmanned aerial vehicle;

(a) assessing suitability of the first and second sensing signals for generating various portions of an environmental map, wherein said assessment is based on distinct characteristics of the various portions of the environment and distinct capabilities of the at least one vision sensor and the at least one proximity sensor;

(b) based on the assessing of (a), (i) selecting the first sensing signal for generating a first portion of the environmental map, and (ii) selecting the second sensing signal for generating a second portion of the environmental map, wherein the first sensing signal is assessed to be more suitable than the second sensing signal for generating the first portion of the environmental map, and wherein the second sensing signal is assessed to be more suitable than the first sensing signal for generating the second portion of the environmental map;

(c) based on the selecting of (b), (i) processing the image data captured by the vision sensor to generate a first set of depth images including depth information for the first portion of the environmental map, and (ii) processing the distance data obtained by the proximity sensor to generate a second set of depth images including depth information for the second portion of the environmental map;

(d) combining the first and second sets of depth images to generate the environmental map; and

(e) effecting the unmanned aerial vehicle to autonomously return to the initial location or fly to a specified location utilizing the environmental map.

2. The method of claim 1, wherein the effecting step of (e) comprises:

determining, using at least one of the plurality of sensors, a current location of the unmanned aerial vehicle;

determining, based on the environmental map, a path from the current location to the initial location or the specified location; and

effecting the unmanned aerial vehicle to move along the path to return to the initial location or the specified location.

3. The method of claim 1, wherein the processing and combining steps of (c) and (d) comprise:

identifying a first plurality of feature points in the first set of depth images;

identifying a corresponding second plurality of feature points in the second set of depth images;

determining a correspondence between the first and second pluralities of feature points; and

combining the first and second sets of depth images based on the determined correspondence to generate the environmental map.

4. The method of claim 2, wherein determining the path comprises determining a path from the current location to the initial location or the specified location that avoids one or more obstacles within the environment.

5. The method of claim 2, wherein the path includes one or more portions previously traveled by the unmanned aerial vehicle.

6. The method of claim 2, wherein the path is different from another path previously traveled by the unmanned aerial vehicle, or the path is the shortest path from the current location to the initial location or the specified location.

7. The method of claim 2, wherein the path is a flight path of the unmanned aerial vehicle including spatial location and orientation of the unmanned aerial vehicle.

8. The method of claim 2, wherein the path comprises a plurality of waypoints corresponding to previously traveled positions of the unmanned aerial vehicle, the plurality of waypoints being recorded by at least one of the plurality of sensors as the unmanned aerial vehicle travels within the environment.

9. The method of claim 1, wherein the first portion of the environmental map is different from the second portion of the environmental map.

20. The method of claim 1, comprising:

generating, using the one or more processors onboard the unmanned aerial vehicle, a first signal to cause the unmanned aerial vehicle to travel within the environment, wherein the first signal is generated based on instructions received from a remote terminal in communication with the unmanned aerial vehicle;

recording, using at least one of the sensors carried by the unmanned aerial vehicle, a plurality of traveled positions of the unmanned aerial vehicle as the unmanned aerial vehicle is traveling within the environment;

receiving, from the remote terminal in communication with the unmanned aerial vehicle or from the one or more processors onboard the unmanned aerial vehicle, an instruction for the unmanned aerial vehicle to return to the initial location or a previously recorded position of the plurality of traveled positions; and

effecting autonomous return of the unmanned aerial vehicle along a path defined by the plurality of traveled positions unless an obstacle situated along the path is detected, in which case autonomous avoidance of said obstacle takes place.

21. The method of claim 20, wherein the path is a flight path of the unmanned aerial vehicle including spatial location and orientation of the unmanned aerial vehicle.

22. The method of claim 20, wherein the unmanned aerial vehicle is configured to detect the presence of a restricted flight region and not fly within a predetermined distance of the restricted flight region.

23. The method of claim 20, wherein the plurality of sensors comprise one or more of a global positioning system (GPS) sensor, a vision sensor, or a proximity sensor.

24. The method of claim 23, wherein the plurality of sensors comprise the proximity sensor, and wherein the proximity sensor comprises at least one of the following:

a lidar sensor, an ultrasonic sensor, or a time-of-flight camera sensor.

25. The method of claim 20, wherein the environmental map comprises a topological map.

26. The method of claim 20, wherein the environmental map comprises a metric map.

27. The method of claim 1, wherein the UAV comprises a receiver configured to receive an instruction to return to the initial location or the specified location, wherein said instruction is transmitted from (1) a remote terminal in communication with the UAV or (2) from one or more processors onboard the unmanned aerial vehicle, and wherein the effecting step of (e) is performed in response to the instruction to return to the initial location or the specified location.

29. The method of claim 20, wherein the autonomous avoidance of said obstacle is effected based on a location of said obstacle as determined from the environmental map.

30. The method of claim 3, wherein the correspondence between the first and second pluralities of feature points is determined by using one or more transformations to map the first set of depth images generated by the vision sensor to the second set of depth images generated by the proximity sensor.

31. The method of claim 1, wherein the first and second portions of the environmental map represent different portions of the environment that are located at different distances from the unmanned aerial vehicle.

32. The method of claim 1, wherein the distinct characteristics of the various portions of the environment are based on two or more of the following:

(1) weather conditions, (2) object density and distribution, and (3) visual or physical properties of objects located in the various portions of the environment.

33. The method of claim 1, further comprising:

generating a plurality of various possible flight paths based on the environmental map, and selecting at least one flight path out of the plurality of flight paths based on a set of operational criteria,

wherein the unmanned aerial vehicle is configured to autonomously return to the initial location or to fly to the specified location along the selected flight path, and

wherein the set of operational criteria is designed to minimize at least one of the following;

flight time, flight distance, energy consumption, number of obstacles along each path, distance of the unmanned aerial vehicle from the obstacles along each path, or altitude range of the unmanned aerial vehicle.

34. The method of claim 1, wherein the unmanned aerial vehicle is effected to autonomously return to the initial location or fly to the specified location based on occurrence of one or more conditions, wherein said conditions include (i) power level falling below a predetermined threshold, (ii) detection of malfunction of the unmanned aerial vehicle, or (iii) loss in communication between the UAV and a remote controller that controls the unmanned aerial vehicle.

10. A system for controlling an unmanned aerial vehicle within an environment, the system comprising:

a plurality of sensors carried by the unmanned aerial vehicle, wherein the plurality of sensors comprise (1) at least one vision sensor configured to capture image data to generate a first sensing signal and (2) at least one proximity sensor configured to obtain distance data to generate a second sensing signal; and

one or more processors onboard the unmanned aerial vehicle, individually or collectively configured to;

(a) determine, using at least one of the plurality of sensors, an initial location of the unmanned aerial vehicle;

(b) assess suitability of the first and second sensing signals for generating various portions of an environmental map, wherein said assessment is based on distinct characteristics of the various portions of the environment and distinct capabilities of the at least one vision sensor and the at least one proximity sensor;

(c) based on the assessed suitability of (a), (i) select the first sensing signal for generating a first portion of the environmental map, and (ii) select the second sensing signal for generating a second portion of the environmental map, wherein the first sensing signal is assessed to be more suitable than the second sensing signal for generating the first portion of the environmental map, and wherein the second sensing signal is assessed to be more suitable than the first sensing signal for generating the second portion of the environmental map;

(d) based on the selected sensing signals of (c), (i) process the image data captured by the vision sensor to generate a first set of depth images including depth information for the first portion of the environmental map, and (ii) process the distance data obtained by the proximity sensor to generate a second set of depth images including depth information for the second portion of the environmental map;

(e) combine the first and second sets of depth images to generate the environmental map; and

(f) effect the unmanned aerial vehicle to autonomously return to the initial location or fly to a specified location utilizing the environmental map.

11. The system of claim 10, wherein the one or more processors are configured to perform the effect step of (f) by:

determining, using at least one of the plurality of sensors, a current location of the unmanned aerial vehicle;

determining, based on the environmental map, a path from the current location to the initial location or the specified location; and

effecting the unmanned aerial vehicle to move along the path to return to the initial location or the specified location.

12. The system of claim 10, wherein the process and combine steps of (d) and (e) comprise:

identifying a first plurality of feature points in the first set of depth images;

identifying a corresponding second plurality of feature points in the second set of depth images;

determining a correspondence between the first and second pluralities of feature points; and

combining the first and second sets of depth images based on the determined correspondence to generate the environmental map.

13. The system of claim 11, wherein determining the path comprises determining a path from the current location to the initial location or the specified location that avoids one or more obstacles within the environment.

14. The system of claim 11, wherein the path includes one or more portions previously traveled by the unmanned aerial vehicle.

15. The system of claim 11, wherein the path is different from another path previously traveled by the unmanned aerial vehicle, or the path is the shortest path from the current location to the initial location or the specified location.

16. The system of claim 11, wherein the path is a flight path of the unmanned aerial vehicle including spatial location and orientation of the unmanned aerial vehicle.

17. The system of claim 11, wherein the path comprises a plurality of waypoints corresponding to previously traveled positions of the unmanned aerial vehicle, the plurality of waypoints being recorded by at least one of the plurality of sensors as the unmanned aerial vehicle travels within the environment.

18. The system of claim 10, wherein the first portion of the environmental map is different from the second portion of the environmental map.

28. The system of claim 10, wherein the UAV comprises a receiver configured to receive an instruction to return to the initial location or the specified location, wherein said instruction is transmitted from (1) a remote terminal in communication with the UAV or (2) from one or more processors onboard the unmanned aerial vehicle, and wherein the effect step of (f) is performed in response to the instruction to return to the initial location or the specified location.

19. One or more non-transitory computer-readable storage media having stored thereon executable instructions that, when executed by one or more processors onboard an unmanned aerial vehicle within an environment, cause the one or more processors to:

(a) determine, using at least one of a plurality of sensors carried by the unmanned aerial vehicle, an initial location of the unmanned aerial vehicle, wherein the plurality of sensors comprise (1) at least one vision sensor configured to capture image data to generate a first sensing signal and (2) at least one proximity sensor configured to obtain distance data to generate a second sensing signal;

(b) assess suitability of the first and second sensing signals for generating various portions of an environmental map, wherein said assessment is based on distinct characteristics of the various portions of the environment and distinct capabilities of the at least one vision sensor and the at least one proximity sensor;

(c) based on the assessed suitability of (b), (i) select the first sensing signal for generating a first portion of the environmental map, and (ii) select the second sensing signal for generating a second portion of the environmental map, wherein the first sensing signal is assessed to be more suitable than the second sensing signal for generating the first portion of the environmental map, and wherein the second sensing signal is assessed to be more suitable than the first sensing signal for generating the second portion of the environmental map;

(d) based on the selected sensing signals of (c), (i) process the image data captured by the vision sensor to generate a first set of depth images including depth information for the first portion of the environmental map, and (ii) process the distance data obtained by the proximity sensor to generate a second set of depth images including depth information for the second portion of the environmental map;

(e) combine the first and second sets of depth images to generate the environmental map; and

(f) effect the unmanned aerial vehicle to autonomously return to the initial location or fly to a specified location utilizing the environmental map.