Sensor trajectory planning for a vehicle

• US 10,754,350 B2
• Filed: 12/12/2017
• Issued: 08/25/2020
• Est. Priority Date: 06/09/2016
• Status: Active Grant

• Current Assignee: Intrinsic Innovation LLC (Alphabet Inc.)
• Original Assignee: X Development LLC (Alphabet Inc.)
• Inventors: Russell, Jared
• Primary Examiner(s): Trivedi, Atul
• Application Number:
  US15/838,437
• Publication Number:
  US 20180113468A1
• Time in Patent Office: 987 Days
• Field of Search: 701 25
• CPC Class Codes: B25J 9/162 Mobile manipulator, movable base with manipulator arm mounted on it
• CPC Class Codes: G05D 1/0094 involving pointing a payload, e.g. camera, weapon, sensor, towards a fixed or moving target
• CPC Class Codes: G05D 1/0242 using non-visible light signals, e.g. IR or UV signals
• CPC Class Codes: G05D 1/0246 using a video camera in combination with image processing means
• CPC Class Codes: G05D 1/0251 extracting 3D information from a plurality of images taken from different locations, e.g. stereo vision stereoscopic image analysis H04N13/00; depth recovery from images G06T7/593
• CPC Class Codes: G05D 1/0257 using a radar radar systems designed for anti-collision purposes between land vehicles or between land vehicle and fixed obstacles G01S13/931
• CPC Class Codes: G05D 1/0297 by controlling means in a control room
• CPC Class Codes: G06Q 10/08 Logistics, e.g. warehousing, loading or distribution; Inventory or stock management
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G06V 2201/06 Recognition of objects for industrial automation

Abstract

An example system includes a vehicle and a sensor connected to the vehicle. The system may receive a predetermined path for the vehicle to follow. The system may also receive a plurality of objectives, associated with a corresponding set of sensor data, for which to collect sensor data. The system may determine, for each of the plurality of objectives, a portion of the environment for the sensor to scan to acquire the corresponding set of sensor data. The system may determine, based on the portion of the environment determined for each of the plurality of objectives, a sensor trajectory through which to move the sensor. The system may cause the sensor to move through the determined sensor trajectory and scan portions of the environment corresponding to the determined sensor trajectory as the vehicle moves along the predetermined path.

Claims

1. A system comprising:

a robotic device;

a sensor connected to the robotic device, wherein the sensor is configured to move with respect to the robotic device to control a portion of an environment observable by the sensor, and wherein a parameter of the sensor is adjustable to control a quality with which the portion of the environment is scanned by the sensor; and

a control system configured to;

determine a sensor trajectory through which to move the sensor with respect to the robotic device while the robotic device moves along a path through the environment, wherein a plurality of portions of the environment are observable by the sensor as the sensor moves along the determined sensor trajectory;

determine a plurality of candidate sequences of sensor parameter values, wherein each candidate sequence of sensor parameter values defines, for each respective portion of the plurality of portions of the environment, a candidate sensor parameter value to which the parameter of the sensor is adjustable when the sensor scans the respective portion of the environment;

select, from the plurality of candidate sequences of sensor parameter values a sequence of sensor parameter values that defines, for each respective portion of the plurality of portions of the environment, a sensor parameter value with which to scan the respective portion of the environment based on an objective for which the respective portion of the environment is scanned; and

as the sensor moves along the sensor trajectory, adjust the parameter of the sensor to scan each portion of the plurality of portions of the environment according to the selected sequence of sensor parameter values.

2. The system of claim 1, wherein the control system is configured to determine each candidate sequence of sensor parameter values of the plurality of candidate sequences of sensor parameter values by:

determining, for each respective portion of the environment, the candidate sensor parameter value based on (i) a candidate sensor parameter value determined for a prior portion of the environment preceding the respective portion of the environment along the determined sensor trajectory and (ii) a rate at which the parameter of the sensor is adjustable.

3. The system of claim 1, wherein the control system is configured to determine the plurality of candidate sequences of sensor parameter values by:

determining a plurality of time points along the path, wherein each time point of the plurality of time points corresponds to a different position of the robotic device along the path;

for each time point of the plurality of time points, determining a plurality of candidate sensor parameter values based on (i) a rate at which the parameter of the sensor is adjustable and (ii) one or more candidate sensor parameter values at a prior time point; and

determining the plurality of candidate sequences of sensor parameter values by selecting, for each time point of the plurality of time points, a corresponding sensor parameter value from the plurality of candidate sensor parameter values.

4. The system of claim 3, wherein each time point of the plurality of time points is associated with a corresponding portion of the environment of the plurality of portions of the environment that are observable by the sensor as the sensor moves along the determined sensor trajectory.

5. The system of claim 1, wherein the control system is configured to select the sequence of sensor parameter values by:

selecting a first sensor parameter value for the respective portion of the environment from a plurality of candidate sensor parameter values defined for the respective portion of the environment by the plurality of candidate sequences of sensor parameter values; and

selecting, based on the first sensor parameter value, a second sensor parameter value for a successive portion of the environment subsequent to the respective portion of the environment such that the parameter of the sensor is adjustable from the first sensor parameter value to the second sensor parameter value between respective scans of the respective portion of the environment and the successive portion of the environment.

6. The system of claim 1, wherein the control system is further configured to:

determine a plurality of objectives for which to collect sensor data as the robotic device moves along the path through the environment, wherein each respective portion of the plurality of portions of the environment is scanned to collect sensor data for one or more objectives of the plurality of determined objectives.

7. The system of claim 6, wherein the control system is further configured to:

determine a plurality of priority levels corresponding to the plurality of objectives; and

select, from the plurality of candidate sequences of sensor parameter values, the sequence of sensor parameter values based on the corresponding priority level of the one or more objectives for which the respective portion of the environment is scanned.

8. The system of claim 6, wherein the plurality of objectives comprise two or more of (i) detection of obstacles on the path through the environment, (ii) localization of the robotic device within the environment, and (iii) detection of objects of interests in regions of the environment adjacent to the path through the environment.

9. The system of claim 1, wherein the control system is further configured to:

determine, for each respective portion of the plurality of portions of the environment, conditions expected to be associated with the respective portion of the environment; and

select the sequence of sensor parameter values that defines the sensor parameter value with which to scan the respective portion of the environment based on the conditions expected to be associated with the respective portion of the environment.

10. The system of claim 1, wherein the control system is further configured to:

determine, for each respective portion of the plurality of portions of the environment, expected content of the respective portion of the environment; and

select the sequence of sensor parameter values that defines the sensor parameter value with which to scan the respective portion of the environment based on the expected content of the respective portion of the environment.

11. The system of claim 1, wherein the parameter of the sensor comprise one or more of (i) a level of magnification of the sensor, (ii) a level of gain of the sensor, (iii) an exposure time of the sensor, (iv) a frame rate at which the sensor generates sensor data, (v) a resolution of the sensor, or (vi) a white balance of the sensor.

12. The system of claim 1, wherein the objective for which the respective portion of the environment is scanned is associated with a quality of sensor data, and wherein the control system is configured to select the sequence of sensor parameter values that defines the sensor parameter value with which to scan the respective portion of the environment based on the quality of sensor data associated with the respective portion of the environment.

13. The system of claim 1, wherein the control system is configured to determine the sensor trajectory by:

determining a plurality of candidate sensor trajectories; and

selecting, based on the plurality of portions of the environment to be scanned while the robotic device moves along the path through the environment, the sensor trajectory from the plurality of candidate sensor trajectories.

14. A method comprising:

determining, by a control system, a sensor trajectory through which to move a sensor with respect to a robotic device while the robotic device moves along a path through an environment, wherein the sensor is connected to the robotic device and configured to move with respect to the robotic device to control a portion of the environment observable by the sensor, wherein a plurality of portions of the environment are observable by the sensor as the sensor moves along the determined sensor trajectory, and wherein a parameter of the sensor is adjustable to control a quality with which the portion of the environment is scanned by the sensor;

determining, by the control system, a plurality of candidate sequences of sensor parameter values, wherein each candidate sequence of sensor parameter values defines, for each respective portion of the plurality of portions of the environment, a candidate sensor parameter value to which the parameter of the sensor is adjustable when the sensor scans the respective portion of the environment;

selecting, by the control system, from the plurality of candidate sequences of sensor parameter values, a sequence of sensor parameter values that defines, for each respective portion of the plurality of portions of the environment, a sensor parameter value with which to scan the respective portion of the environment based on an objective for which the respective portion of the environment is scanned; and

as the sensor moves along the sensor trajectory, adjusting, by the control system, the parameter of the sensor to scan each portion of the plurality of portions of the environment according to the selected sequence of sensor parameter values.

15. The method of claim 14, wherein determining each candidate sequence of sensor parameter values of the plurality of candidate sequences of sensor parameter values comprises:

determining, for each respective portion of the environment, the candidate sensor parameter value based on (i) a candidate sensor parameter value determined for a prior portion of the environment preceding the respective portion of the environment along the determined sensor trajectory and (ii) a rate at which the parameter of the sensor is adjustable.

16. The method of claim 14, wherein determining the plurality of candidate sequences of sensor parameter values comprises:

determining a plurality of time points along the path, wherein each time point of the plurality of time points corresponds to a different position of the robotic device along the path;

for each time point of the plurality of time points, determining a plurality of candidate sensor parameter values based on (i) a rate at which the parameter of the sensor is adjustable and (ii) one or more candidate sensor parameter values at a prior time point; and

determining the plurality of candidate sequences of sensor parameter values by selecting, for each time point of the plurality of time points, a corresponding sensor parameter value from the plurality of candidate sensor parameter values.

17. The method of claim 14, further comprising:

determining a plurality of objectives for which to collect sensor data as the robotic device moves along the path through the environment, wherein each respective portion of the plurality of portions of the environment is scanned to collect sensor data for one or more of the plurality of determined objectives.

18. A non-transitory computer readable storage medium having stored thereon instructions that, when executed by a computing device, cause the computing device to perform operations comprising:

determining a sensor trajectory through which to move a sensor with respect to a robotic device while the robotic device moves along a path through an environment, wherein the sensor is connected to the robotic device and configured to move with respect to the robotic device to control a portion of the environment observable by the sensor, wherein a plurality of portions of the environment are observable by the sensor as the sensor moves along the determined sensor trajectory, and wherein a parameter of the sensor is adjustable to control a quality with which the portion of the environment is scanned by the sensor;

determining a plurality of candidate sequences of sensor parameter values, wherein each candidate sequence of sensor parameter values defines, for each respective portion of the plurality of portions of the environment, a candidate sensor parameter value to which the parameter of the sensor is adjustable when the sensor scans the respective portion of the environment;

selecting, from the plurality of candidate sequences of sensor parameter values, a sequence of sensor parameter values that defines, for each respective portion of the plurality of portions of the environment, a sensor parameter value with which to scan the respective portion of the environment based on an objective for which the respective portion of the environment is scanned; and

as the sensor moves along the sensor trajectory, adjusting the parameter of the sensor to scan each portion of the plurality of portions of the environment according to the selected sequence of sensor parameter values.

19. The non-transitory computer readable storage medium of claim 18, wherein determining each candidate sequence of sensor parameter values of the plurality of candidate sequences of sensor parameter values comprises:

determining, for each respective portion of the environment, the candidate sensor parameter value based on (i) a candidate sensor parameter value determined for a prior portion of the environment preceding the respective portion of the environment along the determined sensor trajectory and (ii) a rate at which the parameter of the sensor is adjustable.

20. The non-transitory computer readable storage medium of claim 18, wherein determining the plurality of candidate sequences of sensor parameter values comprises:

determining a plurality of time points along the path, wherein each time point of the plurality of time points corresponds to a different position of the robotic device along the path;

for each time point of the plurality of time points, determining a plurality of candidate sensor parameter values based on (i) a rate at which the parameter of the sensor is adjustable and (ii) one or more candidate sensor parameter values at a prior time point; and

determining the plurality of candidate sequences of the sensor parameter values by selecting, for each time point of the plurality of time points, a corresponding sensor parameter value from the plurality of candidate sensor parameter values.