Ground plane detection to verify depth sensor status for robot navigation

• US 9,886,035 B1
• Filed: 08/17/2015
• Issued: 02/06/2018
• Est. Priority Date: 08/17/2015
• Status: Active Grant

• Current Assignee: X Development LLC (Alphabet Inc.)
• Original Assignee: X Development LLC (Alphabet Inc.)
• Inventors: Watts, Kevin William, Konolige, Kurt
• Primary Examiner(s): Ziaeianmehdizadeh, Navid
• Application Number:
  US14/828,445
• Time in Patent Office: 904 Days
• Field of Search: None
• CPC Class Codes: G01S 17/88 Lidar systems specially adapted for specific applications
• CPC Class Codes: G05D 1/0088 characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours using knowledge based models G06N5/00
• CPC Class Codes: G05D 1/0251 extracting 3D information from a plurality of images taken from different locations, e.g. stereo vision stereoscopic image analysis H04N13/00; depth recovery from images G06T7/593
• CPC Class Codes: G05D 1/0276 using signals provided by a source external to the vehicle involving a plurality of vehicles G05D1/0287; automatically controlling vehicle speed responsive to externally generated signals B60K31/0058
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads

Abstract

An example method includes determining a target area of a ground plane in an environment of a mobile robotic device, where the target area of the ground plane is in front of the mobile robotic device in a direction of travel of the mobile robotic device. The method further includes receiving depth data from a depth sensor on the mobile robotic device. The method also includes identifying a portion of the depth data representative of the target area. The method additionally includes determining that the portion of the depth data lacks information representing at least one section of the target area. The method further includes providing an output signal identifying at least one zone of non-traversable space for the mobile robotic device in the environment, where the at least one zone of non-traversable space corresponds to the at least one section of the target area.

Claims

1. A method comprising:

determining a target area of a ground plane in an environment of a mobile robotic device, wherein the mobile robotic device is travelling on the ground plane in the environment and the target area of the ground plane is in front of the mobile robotic device in a direction of travel of the mobile robotic device;

receiving orientation data indicative of a current steering orientation of the mobile robotic device;

determining a width of the target area of the ground plane based on the current steering orientation of the mobile robotic device;

receiving depth data from a depth sensor on the mobile robotic device, wherein the depth data is indicative of the environment of the mobile robotic device in the direction of travel of the mobile robotic device;

identifying a portion of the depth data representative of the target area of the ground plane in the environment in front of the mobile robotic device;

determining that the portion of the depth data lacks information representing at least one section of the target area of the ground plane in the environment in front of the mobile robotic device; and

controlling the mobile robotic device to avoid at least one zone of non-traversable space for the mobile robotic device in the environment, wherein the at least one zone of non-traversable space corresponds to the at least one section of the target area of the ground plane in the environment in front of the mobile robotic device.

2. The method of claim 1, wherein the depth sensor is a stereo sensor that determines the depth data by locating matching features between images captured by two or more optical elements.

3. The method of claim 1, wherein the target area of the ground plane in the environment extends from a first distance in front of the mobile robotic device in the direction of travel of the mobile robotic device to a second distance in front of the mobile robotic device in the direction of travel of the mobile robotic device.

4. The method of claim 3, further comprising:

receiving speed data indicative of a current speed of the mobile robotic device; and

determining the second distance for the target area of the ground plane based on the current speed of the mobile robotic device.

5. The method of claim 1, wherein the target area of the ground plane in the environment has a width that is greater than a width of the mobile robotic device, wherein the width of the target area increases in the direction of travel of the mobile robotic device.

6. The method of claim 1, further comprising:

receiving navigation instructions for the mobile robotic device; and

determining the target area of the ground plane in the environment based on the navigation instructions.

7. The method of claim 1, further comprising:

receiving speed data indicative of a current speed of the mobile robotic device; and

verifying that the depth data includes information representing the target area of the ground plane in the environment in front of the mobile robotic device at a chosen time interval, wherein the chosen time interval is determined based on the current speed of the mobile robotic device.

8. The method of claim 1, wherein controlling the mobile robotic device comprises causing the mobile robotic device to stop moving in the direction of travel based on the at least one identified zone of non-traversable space.

9. The method of claim 1, further comprising:

determining that a particular section of the at least one section of the target area of the ground plane is above a threshold size; and

controlling the mobile robotic device to avoid the at least one zone of non-traversable space in response to determining that the particular section is above the threshold size.

10. The method of claim 1, further comprising:

causing at least one texture projector on the mobile robotic device to project a texture pattern toward the target area of the ground plane in the environment in front of the mobile robotic device in the direction of travel of the mobile robotic device; and

identifying the portion of the depth data representing the target area of the ground plane in the environment based on detection of the texture pattern by the depth sensor on the mobile robotic device.

11. A mobile robotic device, comprising:

a depth sensor on the mobile robotic device; and

a control system, comprising a processor, wherein the control system is configured to;

determine a target area of a ground plane in an environment of the mobile robotic device, wherein the mobile robotic device is travelling on the ground plane in the environment and the target area of the ground plane is in front of the mobile robotic device in a direction of travel of the mobile robotic device;

receive orientation data indicative of a current steering orientation of the mobile robotic device;

determine a width of the target area of the ground plane based on the current steering orientation of the mobile robotic device;

receive depth data from the depth sensor on the mobile robotic device, wherein the depth data is indicative of the environment of the mobile robotic device in the direction of travel of the mobile robotic device;

identify a portion of the depth data representative of the target area of the ground plane in the environment in front of the mobile robotic device;

determine that the portion of the depth data lacks information representing at least one section of the target area of the ground plane in the environment in front of the mobile robotic device; and

control the mobile robotic device to avoid at least one zone of non-traversable space for the mobile robotic device in the environment, wherein the at least one zone of non-traversable space corresponds to the at least one section of the target area of the ground plane in the environment in front of the mobile robotic device.

12. The mobile robotic device of claim 11, wherein the depth sensor is a stereo sensor that determines the depth data by locating matching features between images captured by two or more optical elements.

13. The mobile robotic device of claim 12, wherein the stereo sensor is oriented toward the ground plane in the environment so that a field of view of the stereo sensor extends from a first distance in front of the mobile robotic device in the direction of travel of the mobile robotic device;

and

the target area of the ground plane in the environment extends from the first distance to a second distance in front of the mobile robotic device in the direction of travel of the mobile robotic device.

14. The mobile robotic device of claim 12, further comprising at least one texture projector on the mobile robotic device, wherein the control system is further configured to:

cause the at least one texture projector on the mobile robotic device to project a texture pattern toward the target area of the ground plane in the environment in front of the mobile robotic device in the direction of travel of the mobile robotic device; and

identify the portion of the depth data representing the target area of the ground plane in the environment based on detection of the pattern by the stereo sensor on the mobile robotic device.

15. The mobile robotic device of claim 14, wherein the control system is further configured to:

determine that the projected texture pattern failed to reach at least one additional section of the target area of the ground plane in the environment in front of the mobile robotic device; and

control the mobile robotic device to avoid at least one additional zone of non-traversable space for the mobile robotic device in the environment, wherein the at least one additional zone of non-traversable space corresponds to the at least one additional section of the target area of the ground plane in the environment in front of the mobile robotic device.

16. A non-transitory computer readable medium having stored therein instructions, that when executed by a control system of a mobile robotic device, cause the control system to perform functions comprising:

determining a target area of a ground plane in an environment of the mobile robotic device, wherein the mobile robotic device is travelling on the ground plane in the environment and the target area of the ground plane is in front of the mobile robotic device in a direction of travel of the mobile robotic device;

receiving orientation data indicative of a current steering orientation of the mobile robotic device;

determining a width of the target area of the ground plane based on the current steering orientation of the mobile robotic device;

receiving depth data from a depth sensor on the mobile robotic device, wherein the depth data is indicative of the environment of the mobile robotic device in the direction of travel of the mobile robotic device;

identifying a portion of the depth data representative of the target area of the ground plane in the environment in front of the mobile robotic device;

determining that the portion of the depth data lacks information representing at least one section of the target area of the ground plane in the environment in front of the mobile robotic device; and

controlling the mobile robotic device to avoid at least one zone of non-traversable space for the mobile robotic device in the environment, wherein the at least one zone of non-traversable space corresponds to the at least one section of the target area of the ground plane in the environment in front of the mobile robotic device.

17. The non-transitory computer readable medium of claim 16, wherein the depth sensor is a stereo sensor that determines the depth data by locating matching features between images captured by two or more optical elements.

18. The non-transitory computer readable medium of claim 16, wherein the target area of the ground plane in the environment extends from a first distance in front of the mobile robotic device in the direction of travel of the mobile robotic device to a second distance in front of the mobile robotic device in the direction of travel of the mobile robotic device;

and

wherein the target area of the ground plane in the environment has a width that is greater than a width of the mobile robotic device, wherein the width of the target area increases in the direction of travel of the mobile robotic device.

19. The non-transitory computer readable medium of claim 16, wherein the target area of the ground plane in the environment extends from a first distance in front of the mobile robotic device in the direction of travel of the mobile robotic device to a second distance in front of the mobile robotic device in the direction of travel of the mobile robotic device, the functions further comprising:

receiving speed data indicative of a current speed of the mobile robotic device; and

determining the second distance for the target area based on the current speed of the mobile robotic device.