Machine control system utilizing stereo disparity density

• US 8,320,627 B2
• Filed: 06/17/2010
• Issued: 11/27/2012
• Est. Priority Date: 06/17/2010
• Status: Active Grant

• Current Assignee: Caterpillar Incorporated
• Original Assignee: Caterpillar Incorporated
• Inventors: McDaniel, Michael S., Dutta, Susmita
• Primary Examiner(s): AZARIAN, SEYED H
• Application Number:
  US12/817,530
• Publication Number:
  US 20110311102A1
• Time in Patent Office: 894 Days
• Field of Search: 382/100, 382103-107, 382/154, 382/162, 382/168, 382/172, 382/173, 382/181, 382/199, 382/232, 382/254, 382/274, 382/276, 382/291, 382/305, 382/312, 348/14.02, 348/14.1, 348/51
• US Class Current: 382/104
• CPC Class Codes: G05D 1/0251 extracting 3D information from a plurality of images taken from different locations, e.g. stereo vision stereoscopic image analysis H04N13/00; depth recovery from images G06T7/593
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/30168 Image quality inspection
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/0002 Inspection of images, e.g. flaw detection

Abstract

A control system for a mobile machine is disclosed. The control system may have a first sensor mounted on the mobile machine and configured to capture a first image of a region near the mobile machine, a second sensor mounted on the mobile machine and configured to capture a second image of the region, and a controller in communication with the first and second sensors. The controller may be configured to generate a stereo image from the first and second images, compute a disparity map of the stereo image, and generate an output to affect operation of the machine when a density of the disparity map is less than a threshold density.

Claims

1. A control system for a mobile machine, comprising:

a first sensor mounted on the mobile machine and configured to capture a first image of a region at a worksite of the mobile machine;

a second sensor mounted on the mobile machine and configured to capture a second image of the region; and

a controller in communication with the first and second sensors, the controller being configured to;

generate a stereo image from the first and second images;

compute a disparity map of the stereo image; and

generate an output to affect operation of the mobile machine when a density of the disparity map is less than a threshold density, wherein the threshold density is adjustable based on at least one of a speed of the mobile machine, a type of the mobile machine, a condition of the mobile machine, a worksite condition, or an operator preference.

2. The control system of claim 1, wherein the output is an automated command that alters at least one of a speed or a direction of the mobile machine.

3. The control system of claim 1, wherein the output is a message directed to an operator of the mobile machine.

4. The control system of claim 1, wherein the output is a recommendation directed to an operator of the mobile machine to change a control parameter of the mobile machine.

5. The control system of claim 1, further including a display device located inside the mobile machine, wherein the controller is further configured to display at least one of the stereo image or a representation of the disparity map on the display device.

6. The control system of claim 1, wherein the controller is further configured to:

divide the disparity map into multiple zones;

compute the density of each of the multiple zones; and

generate the output when the density for at least one of the multiple zones is less than the threshold density.

7. The control system of claim 6, wherein the threshold density is different for different zones of the multiple zones.

8. The control system of claim 7, wherein at least one of the multiple zones is classified as a critical zone and has a threshold density higher than threshold densities of other zones of the multiple zones.

9. The control system of claim 8, wherein the controller is further configured to receive from another mobile machine co-located at a common worksite portions of a replacement stereo image corresponding with zones of the multiple zones having a density less than a corresponding threshold density.

10. The control system of claim 6, wherein at least one of a size or a number of zones is adjustable based on at least one of a speed of the mobile machine, a type of the mobile machine, a condition of the mobile machine, a worksite condition, or an operator preference.

11. The control system of claim 1, wherein the controller is configured to generate a visibility map associated with a worksite at which the mobile machine operates based on the disparity map.

12. The control system of claim 11, further including a communication device configured to communicate at least one of the stereo image, the disparity map, or the visibility map to an offboard device.

13. A method of controlling a mobile machine, comprising:

capturing a first image of a region at a worksite of the mobile machine from a first location onboard the mobile machine;

capturing a second image of the region from a second location onboard the mobile machine;

generating a stereo image from the first and second images;

computing a disparity map of the stereo image; and

generating an output to affect operation of the mobile machine when a density of the disparity map is less than a threshold density, wherein the threshold density is adjustable based on at least one of a speed of the mobile machine, a type of the mobile machine, a condition of the mobile machine, a worksite condition, or an operator preference.

14. The method of claim 13, wherein the output is at least one of an automated command that reduces a speed of the mobile machine, a visibility warning directed to an operator of the mobile machine, and a recommendation directed to the operator to change a control parameter of the mobile machine.

15. The method of claim 13, further including dividing the disparity map into multiple zones, wherein:

computing includes computing the density of each of the multiple zones;

generating the output includes generating the output when the density for at least one of the multiple zones is less than the threshold density; and

the threshold density is different for different zones of the multiple zones.

16. The method of claim 15, further including classifying at least one of the multiple zones as a critical zone, wherein the threshold density for the critical zone is higher than threshold densities of other zones of the multiple zones.

17. The method of claim 16, further including receiving from another mobile machine co-located at a common worksite portions of a replacement stereo image corresponding with zones of the multiple zones having a density less than a corresponding threshold density.

18. A mobile machine, comprising:

a body;

a power source;

traction devices connected to the body and driven by the power source to propel the mobile machine;

a first camera mounted on the body and configured to capture a first image of a region at a worksite of the mobile machine;

a second camera mounted on the body and configured to capture a second image of the region; and

a controller in communication with the first and second cameras, the controller being configured to;

generate a stereo image from the first and second images;

compute a disparity map of the stereo image; and

generate at least one of an automated command that reduces a speed of the traction devices, a visibility warning directed to an operator of the mobile machine, and a recommendation directed to the operator to change a control parameter of the mobile machine when a density of the disparity map is less than a threshold density, wherein the threshold density is adjustable based on at least one of a speed of the mobile machine, a type of the mobile machine, a condition of the mobile machine, a worksite condition, or an operator preference.