Perception Based Loading

• US 20140146167A1
• Filed: 11/27/2012
• Published: 05/29/2014
• Est. Priority Date: 11/27/2012
• Status: Active Grant

• Current Assignee: Caterpillar Incorporated
• Original Assignee: Caterpillar Incorporated
• Inventors: Friend, Paul R., Reiners, Eric A., Chen, Qi
• Granted Patent: US 9,131,119 B2
• Time in Patent Office: Days
• US Class Current: 348/118
• CPC Class Codes: G01S 17/06 Systems determining position data of a target
• CPC Class Codes: G01S 17/74 Systems using reradiation of electromagnetic waves other than radio waves, e.g. IFF, i.e. identification of friend or foe
• CPC Class Codes: G01S 17/86 Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
• CPC Class Codes: G01S 5/16 using electromagnetic waves other than radio waves
• CPC Class Codes: G05D 1/0225 involving docking at a fixed facility, e.g. base station or loading bay parking aids B62D15/027
• CPC Class Codes: G05D 1/024 in combination with a laser lasers per se H01S
• CPC Class Codes: G05D 1/0246 using a video camera in combination with image processing means
• CPC Class Codes: H04N 7/18 Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast

Abstract

Systems and methods for assisted positioning of a haul machine at a destination position for receiving materials from a loading machine are provided. The method includes obtaining from at least one perception device information reflecting a location or orientation of a first fiducial attached to the loading machine; determining a location or position of the first fiducial based on the information provided by the at least one perception device; determining the destination position based on the location or position of the first fiducial; and issuing guidance to an operator of the haul machine indicating an adjustment in steering angle based on the determined destination position.

Claims

1. A system for assisted positioning of a haul machine at a destination position for receiving materials from a loading machine, the system comprising:

at least one perception device included with the haul machine, configured to obtain and provide information reflecting a location or orientation of a first fiducial attached to the loading machine;

a communication interface located in the haul machine for conveying positioning guidance to a human operator of the haul machine; and

a processor configured to;

determine a location or position of the first fiducial based on the information provided by the perception device,

determine the destination position based on the determined location or position of the first fiducial, and

issue guidance via the communication interface indicating an adjustment in steering angle based on the determined destination position.

2. The system of claim 1, wherein the perception device includes a camera and a LIDAR, and the processor is further configured to:

obtain an image of the first fiducial captured by the camera,

detect the fiducial and determine an angular position of the first fiducial relative to the haul machine based on the image,

obtain distance information for the first fiducial from the LIDAR, and

determine a location or position of the first fiducial based on the obtained distance and determined angular position.

3. The system of claim 1, wherein the a least one perception device includes a camera and a LIDAR, and the processor is further configured to:

obtain an image of the first fiducial captured by the camera,

detect the first fiducial and determine an angular position of the first fiducial relative to the haul machine based on the image,

obtain distance information for the loading machine from the LIDAR,

determine a location or position of the loading machine based on the obtained distance and determined angular position, and

determine the destination position based on the location or position of the loading machine.

4. The system of claim 1, further comprising:

a second fiducial attached to the haul machine; and

a wireless data communication device,

wherein the processor is further configured to;

transmit, via the wireless data communication device, information regarding the location or position of the first fiducial to the loading machine,

receive, via the wireless data communication device, information regarding a location or position of the second fiducial determined by the loading machine, and

determine the destination position based on the location or position of the first fiducial and the information regarding a location or position of the second fiducial.

5. The system of claim 1, wherein the first fiducial is on or near a bucket used by the loading machine to transfer the materials to the haul machine, and the processor is further configured to determine the location of the first fiducial as a location for the destination position.

6. The system of claim 5, wherein the perception device includes a camera, and the processor is further configured to:

obtain a first image of the first fiducial captured by the camera,

decode an identifier for the first fiducial from the obtained image,

obtain a second image of a second fiducial captured by the camera, the second fiducial being attached to the loading machine, and

distinguish the first fiducial from the second fiducial as being on or near the bucket based on the decoded identifier.

7. The system of claim 1, wherein the communication interface includes a graphical display area, and the processor is further configured to:

display and manage a graphical user interface on the graphical display area,

display, in the graphical user interface, a top-down view of the haul machine and a graphic element indicating the destination position relative to a current position of the haul machine,

display, in the graphical user interface, a left turning navigation indicator if the adjustment requires the operator to rotate a steering wheel counterclockwise, and

display, in the graphical user interface, a right turning navigation indicator if the adjustment requires the operator to rotate a steering wheel clockwise.

8. The system of claim 1, wherein the perception device is further configured to obtain and provide information reflecting a location or orientation of a second fiducial at a fixed reference location, and the processor is further configured to determine a location or position of the haul machine relative to the second fiducial based on the information reflecting a location or orientation of the second fiducial.

9. The system of claim 1, wherein:

the perception device is further configured to obtain and provide information reflecting a location or orientation of a second fiducial at a fixed reference location,

the information reflecting a location or orientation of a second fiducial is obtained prior to the information reflecting a location or orientation of the first fiducial at time that the first fiducial is not viewable by the at least one perception device, and

the processor is further configured to;

determine a location or position of the second fiducial based on the information reflecting a location or orientation of a second fiducial,

determine a second destination position based on the determined location or position of the second fiducial, and

issue guidance via the communication interface indicating an adjustment in steering angle based on the determined second destination position.

10. A method for assisted positioning of a haul machine at a destination position for receiving materials from a loading machine, the method comprising:

obtaining from at least one perception device information reflecting a location or orientation of a first fiducial attached to the loading machine;

determining a location or position of the first fiducial based on the information provided by the at least one perception device;

determining the destination position based on the determined location or position of the first fiducial; and

issuing guidance to an operator of the haul machine indicating an adjustment in steering angle based on the determined destination position.

11. The method of claim 10, wherein the perception device includes a camera and a LIDAR, and the method further comprises:

obtaining an image of the first fiducial captured by the camera,

detecting the fiducial and determining an angular position of the first fiducial relative to the haul machine based on the image,

obtaining distance information for the first fiducial from the LIDAR, and

determining a location or position of the first fiducial based on the obtained distance and determined angular position.

12. The method of claim 10, wherein the perception device includes a camera and a LIDAR, and the method further comprises:

obtaining an image of the first fiducial captured by the camera,

detecting the first fiducial and determine an angular position of the first fiducial relative to the haul machine based on the image,

obtaining distance information for the loading machine from the LIDAR,

determining a location or position of the loading machine based on the obtained distance and determined angular position, and

determining the destination position based on the location or position of the loading machine.

13. The method of claim 10, further comprising:

attaching a fiducial to the haul machine;

transmitting, via a wireless data communication device, information regarding the location or position of the first fiducial to the loading machine;

receiving, via the wireless data communication device, information regarding a location or position of the second fiducial determined by the loading machine; and

determining the destination position based on the location or position of the first fiducial and the information regarding a location or position of the second fiducial.

14. The method of claim 10, wherein the first fiducial is on or near a bucket used by the loading machine to transfer the materials to the haul machine, and the method further comprises determining the location of the first fiducial as a location for the destination position.

15. The method of claim 14, wherein the perception device includes a camera, and the method further comprises:

obtaining a first image of the first fiducial captured by the camera,

decoding an identifier for the first fiducial from the obtained image,

obtaining a second image of a second fiducial captured by the camera, the second fiducial being attached to the loading machine, and

distinguishing the first fiducial from the second fiducial as being on or near the bucket based on the decoded identifier.

16. The method of claim 10, further comprising:

displaying, in a graphical user interface, a top-down view of the haul machine and a graphic element indicating the destination position relative to a current position of the haul machine;

displaying, in the graphical user interface, a left turning navigation indicator if the adjustment requires the operator to rotate a steering wheel counterclockwise; and

displaying, in the graphical user interface, a right turning navigation indicator if the adjustment requires the operator to rotate a steering wheel clockwise.

17. A system for assisted positioning of a haul machine at a destination position for receiving materials from a loading machine, the system comprising:

perception means for obtaining and providing information reflecting a location or orientation of a first fiducial attached to the loading machine;

a communication interface located in the haul machine for conveying positioning guidance to a human operator of the haul machine;

locating means for determining a location or position of the first fiducial based on the information provided by the at least one perception device;

positioning means for determining the destination position based on the determined location or position of the first fiducial; and

guidance means for issuing guidance via the communication interface indicating an adjustment in steering angle based on the determined destination position.

18. The system of claim 17, wherein the perception means includes a camera and a LIDAR, and the system further comprises:

imaging means for obtaining an image of the first fiducial captured by the camera;

detection means for detecting the fiducial and determining an angular position of the first fiducial relative to the haul machine based on the image; and

ranging means for obtaining distance information for the first fiducial from the LIDAR,

wherein the locating means determine the location or position of the first fiducial based on the obtained distance and determined angular position.

19. The system of claim 17, wherein the communication interface includes a graphical display area, and the system further comprises:

user interface means for displaying and managing a graphical user interface on the graphical display area; and

navigation means for;

displaying, in the graphical user interface, a top-down view of the haul machine and a graphic element indicating the destination position relative to a current position of the haul machine,

displaying, in the graphical user interface, a left turning navigation indicator if the adjustment requires the operator to rotate a steering wheel counterclockwise, and

displaying, in the graphical user interface, a right turning navigation indicator if the adjustment requires the operator to rotate a steering wheel clockwise.

20. The system of claim 17, further comprising:

a second fiducial attached to the haul machine;

transmitting means for transmitting information regarding the location or position of the first fiducial to the loading machine; and

receiving means for receiving information regarding a location or position of the second fiducial determined by the loading machine,

wherein the positioning means determine the destination position based on the determined location or position of the first fiducial and the information regarding a location or position of the second fiducial.