Using laser sensors to augment stereo sensor readings for robotic devices

• US 9,746,852 B1
• Filed: 08/17/2015
• Issued: 08/29/2017
• Est. Priority Date: 08/17/2015
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: X Development LLC (Alphabet Inc.)
• Inventors: Watts, Kevin William, Konolige, Kurt
• Primary Examiner(s): Moyer, Dale
• Application Number:
  US14/828,437
• Time in Patent Office: 743 Days
• CPC Class Codes: G01S 17/86 Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
• CPC Class Codes: G01S 17/87 Combinations of systems using electromagnetic waves other than radio waves
• CPC Class Codes: G01S 17/931 of land vehicles
• CPC Class Codes: G05D 1/0248 in combination with a laser lasers per se H01S
• CPC Class Codes: G05D 1/0251 extracting 3D information from a plurality of images taken from different locations, e.g. stereo vision stereoscopic image analysis H04N13/00; depth recovery from images G06T7/593

Abstract

An example system includes one or more laser sensors on a robotic device, where the one or more laser sensors are configured to produce laser sensor data indicative of a first area within a first distance in front of the robotic device. The system further includes one or more stereo sensors on the robotic device, where the stereo sensors on the robotic device are configured to produce stereo sensor data indicative of a second area past a second distance in front of the robotic device. The system also includes a controller configured to receive the laser sensor data, receive the stereo sensor data, detect one or more objects in front of the robotic device based on at least one of the laser sensor data and the stereo sensor data, and provide instructions for the robotic device to navigate based on the one or more detected objects.

Claims

1. A system comprising:

one or more laser sensors on a robotic device, wherein the one or more laser sensors are configured to produce laser sensor data indicative of a first area within a first distance in front of the robotic device, and wherein the one or more laser sensors comprise a plurality of laser sensors spaced across the robotic device so that consecutive laser sensors are closer together than a predetermined minimum obstacle size;

one or more stereo sensors on the robotic device, wherein the one or more stereo sensors on the robotic device are configured to produce stereo sensor data indicative of a second area past a second distance in front of the robotic device, wherein the second distance is closer to the robotic device than the first distance; and

a controller configured to;

receive the laser sensor data indicative of the first area within the first distance in front of the robotic device from the one or more laser sensors;

receive the stereo sensor data indicative of the second area past the second distance in front of the robotic device from the one or more stereo sensors;

detect one or more objects in front of the robotic device based on at least one of the laser sensor data and the stereo sensor data; and

provide instructions for the robotic device to navigate based on the one or more detected objects.

2. The system of claim 1, wherein the one or more laser sensors comprise a plurality of laser sensors that project respective narrow laser beams in front of the robotic device.

3. The system of claim 2, wherein the plurality of laser sensors are aligned horizontally at a certain height on a front surface of the robotic device.

4. The system of claim 2, wherein the plurality of laser sensors are oriented parallel to a direction of travel of the robotic device.

5. The system of claim 1, wherein the second distance comprises a distance from the robotic device at which a field of view of the one or more stereo sensors encompasses a width larger than a width of the robotic device.

6. The system of claim 1, wherein the one or more laser sensors comprise a plurality of laser sensors spaced across the front surface of the robotic device to detect obstacles of at least a predetermined minimum size at least up to the second distance from the robotic device.

7. The system of claim 1, wherein the one or more laser sensors comprise a plurality of horizontal rows of laser sensors aligned at a plurality of different heights on the robotic device, and wherein each laser sensor projects respective narrow laser beams.

8. The system of claim 1, wherein:

a first stereo sensor of the one or more stereo sensors comprises a first primary optical element and a secondary optical element such that the stereo sensor data from the first stereo sensor is indicative of a third area within the second area;

a second stereo sensor of the one or more stereo sensors comprises a second primary optical element and the secondary optical element such that the stereo sensor data from the second stereo sensor is indicative of a fourth area within the second area; and

the first primary optical element and the secondary optical element are positioned closer together on the robotic device than the second primary optical element and the secondary optical element.

9. The system of claim 8, wherein the optical elements of the first stereo sensor are spaced in a narrow baseline arrangement, and wherein the optical elements of the second stereo sensor are spaced in a wide baseline arrangement.

10. A method comprising:

receiving laser sensor data from a plurality of laser sensors Off spaced across a robotic device so that consecutive laser sensors are closer together than a predetermined minimum obstacle size, wherein the laser sensor data is indicative of a first area within a first distance of a direction of travel of the robotic device;

receiving stereo sensor data from one or more stereo sensors on the robotic device, wherein the stereo sensor data is indicative of a second area past a second distance in the direction of travel of the robotic device, wherein the second distance is closer to the robotic device than the first distance;

monitoring the first area based on the laser sensor data and the second area based on the stereo sensor data;

detecting one or more objects in at least one of the first area and second area; and

providing instructions for the robotic device to navigate based at least on the one or more detected objects.

11. The method of claim 10, wherein the plurality of laser sensors are configured to project respective narrow laser beams at least up to the first distance in the direction of travel of the robotic device.

12. The method of claim 10, wherein the second distance comprises a distance from the robotic device at which a field of view of the one or more stereo sensors encompasses a width larger than a width of the robotic device.

13. The method of claim 10, wherein the plurality of laser sensors are spaced across the robotic device to detect obstacles of at least a predetermined minimum size at least up to the second distance from the robotic device.

14. The method of claim 10, wherein:

a first stereo sensor of the one or more stereo sensors comprises a first primary optical element and a secondary optical element such that the stereo sensor data from the first stereo sensor is indicative of a third area within the second area;

a second stereo sensor of the one or more stereo sensors comprises a second primary optical element and the secondary optical element such that the stereo sensor data from the second stereo sensor is indicative of a fourth area within the second area; and

the first primary optical element and the secondary optical element are closer together on the robotic device than the second primary optical element and the secondary optical element.

15. A robotic device comprising:

at least one motion component configured to cause motion of the robotic device in a direction of travel;

a plurality of laser sensors on the robotic device, wherein the plurality of laser sensors are horizontally aligned at a certain height on the robotic device, wherein the plurality of laser sensors are oriented parallel to the direction of travel of the robotic device, and wherein the plurality of laser sensors are configured to;

project respective narrow laser beams in front of the robotic device in the direction of travel of the robotic device;

detect reflected laser beams at the robotic device from at least some of the projected narrow laser beams; and

detect objects within a first area up to a first distance in the direction of travel of the robotic device based on the reflected laser beams; and

one or more stereo sensors on the robotic device, wherein the one or more stereo sensors on the robotic device are configured to;

receive respective images captured by at least two optical elements from at least two respective viewpoints;

identify corresponding features between the respective images captured by the at least two optical elements; and

detect objects within a second area past a second distance in the direction of travel of the robotic device based on the corresponding features, wherein the second distance is closer to the robotic device than the first distance.

16. The robotic device of claim 15, wherein the second distance comprises a distance from the robotic device at which a field of view of the one or more stereo sensors encompasses a width larger than the width of the robotic device.

17. The robotic device of claim 15, wherein the plurality of laser sensors are spaced across a front surface of the robotic device to detect obstacles of at least a predetermined minimum size at least up to the second distance from the robotic device.

18. The robotic device of claim 15, wherein:

a first stereo sensor of the one or more stereo sensors comprises a first primary optical element and a secondary optical element such that the stereo sensor data from the first stereo sensor is indicative of a third area within the second area;

a second stereo sensor of the one or more stereo sensors comprises a second primary optical element and the secondary optical element such that the stereo sensor data from the second stereo sensor is indicative of a fourth area within the second area; and

the first primary optical element and the secondary optical element are closer together on the robotic device than the second primary optical element and the secondary optical element.