Vision system and method of monitoring surroundings of machine

• US 9,667,875 B2
• Filed: 01/21/2015
• Issued: 05/30/2017
• Est. Priority Date: 01/21/2015
• Status: Active Grant

• Current Assignee: Caterpillar Incorporated
• Original Assignee: Caterpillar Incorporated
• Inventors: Sanchez, Rodrigo, Petrany, Peter
• Primary Examiner(s): Aynalem, Nathnael
• Application Number:
  US14/601,320
• Publication Number:
  US 20160212352A1
• Time in Patent Office: 860 Days
• CPC Class Codes: B60R 1/00 Optical viewing arrangements; Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles
• CPC Class Codes: B60R 1/23 with a predetermined field of view
• CPC Class Codes: B60R 2300/303 using joined images, e.g. multiple camera images
• CPC Class Codes: B60R 2300/602 with an adjustable viewpoint
• CPC Class Codes: G06T 3/00 Geometric image transformations in the plane of the image
• CPC Class Codes: H04N 23/69 Control of means for changing angle of the field of view, e.g. optical zoom objectives or electronic zooming
• CPC Class Codes: H04N 5/265 Mixing

Abstract

A vision system for a machine system includes a first set of cameras mounted on the first section and a second set of cameras mounted on the second section of the machine. A controller is coupled to the first set of cameras and the second set of cameras. The controller includes a first and a second image transformation modules to convert a first and a second images captured by the first and the second set of cameras into a first and a second transformed image based at least on a view type, respectively. An image combination module combines the first transformed image and the second transformed image to a combined image with reference to the articulation joint. The image combination module communicates with a display device to display the combined image. The combined image is indicative of surroundings of the first section and the second section of the machine.

Claims

1. A vision system for a machine having a first section, a second section and an articulation joint pivotally connecting the first section to the second section, the vision system comprising:

a first set of cameras mounted on the first section of the machine;

a second set of cameras mounted on the second section of the machine;

a display device configured to display an image;

a controller communicably coupled to the first set of cameras, the second set of cameras, and the display device, the controller being configured to;

stitch together images captured by the first set of cameras to form a first image;

convert the first image captured by the first set of cameras into a first transformed image based at least on a view type;

stitch together images captured by the second set of cameras to form a second image;

convert the second image captured by the second set of cameras into a second transformed image based at least on a view type;

after forming the first and second images, receive the first transformed image and the second transformed image;

after receiving the first transformed image and the second transformed image, combine the first transformed image and the second transformed image to obtain a combined image with reference to the articulation joint, wherein the combined image is indicative of surroundings of the first section and the second section of the machine; and

display the combined image on the display device.

2. The vision system of claim 1, wherein the view type is one of a three dimensional view, a perspective view, a two dimensional view, a top surround view and a side view.

3. The vision system of claim 2, wherein the controller is further configured to:

select the view type based on a user input; and

select the view type based on a user input.

4. The vision system of claim 2, wherein the controller is further configured to combine the first image and the second image with reference to the articulation joint based on the view type.

5. The vision system of claim 1, wherein the first set of cameras comprises:

a front camera mounted adjacent to a front end of the first section; and

a first side camera mounted adjacent to each of the sides of the first section.

6. The vision system of claim 1, wherein the second set of cameras comprises:

a rear camera mounted adjacent to a rear end of the second section; and

a second side camera mounted adjacent to each of the sides of the second section.

7. The vision system of claim 1, wherein the controller is further configured to:

correct the first image captured by the first set of cameras based on a calibration parameter associated with the first set of cameras; and

correct the second image captured by the second set of cameras based on a calibration parameter associated with the second set of cameras.

8. The vision system of claim 1, wherein the display device is disposed within an operator cab of the machine.

9. A machine comprising:

a first section comprising a first set of ground engaging members;

a second section comprising a second set of ground engaging members;

an articulation joint configured to pivotally couple the first section to the second section; and

a vision system comprising;

a first set of cameras mounted on the first section;

a second set of cameras mounted on the second section;

a display device configured to display an image;

a controller communicably coupled to the first set of cameras, the second set of cameras, and the display device, the controller being configured to;

stitch together images captured first set of cameras to form a first image;

convert the first image captured by the first set of cameras into a first transformed image based at least on a view type;

stitch together images captured by the second set of cameras to form a second image;

convert the second image captured by the second set of cameras into a second transformed image based at least on a view type;

after forming the first and second images, receive the first transformed image and the second transformed image;

after receiving the first transformed image and the second transformed image, combine the first transformed image and the second transformed image to obtain a combined image with reference to the articulation joint, wherein the combined image is indicative of surroundings of the first section and the second section of the machine; and

display the combined image on the display device.

10. The machine of claim 9, wherein the view type is one of a three dimensional view, a perspective view, a two dimensional view, a top surround view and a side view.

11. The machine of claim 10, wherein the controller is further configured to:

select the view type based on a user input; and

select the view type based on a user input.

12. The machine of claim 10, wherein the controller is further configured to combine the first image and the second image with reference to the articulation joint based on the view type.

13. The machine of claim 9, wherein the first set of cameras comprises:

a front camera mounted adjacent to a front end of the first section; and

a first side camera mounted adjacent to each of the sides of the first section.

14. The machine of claim 9, wherein the second set of cameras comprises:

a rear camera mounted adjacent to a rear end of the second section; and

a second side camera mounted adjacent to each of the sides of the second section.

15. The machine of claim 9, wherein the controller is further configured:

correct the first image captured by the first set of cameras based on a calibration parameter associated with the first set of cameras; and

correct the second image captured by the second set of cameras based on a calibration parameter associated with the second set of cameras.

16. The machine of claim 9, wherein the first section further comprises an operator cab, and wherein the display device is disposed within the operator cab.

17. A method of monitoring surroundings of a machine having a first section, a second section and an articulation joint pivotally connecting the first section to the second section, the method comprising:

capturing, via a first set of cameras, a first image, wherein the first set of cameras is mounted on the first section of the machine;

stitching together images captured by the first set of camera to form a first image;

capturing, via a second set of cameras, a second image, wherein the second set of cameras is mounted on the second section of the machine;

stitching together images captured by the second set of cameras to form a second image;

after forming the first image, converting the first image captured by the first set of cameras into a first transformed image based at least on a view type;

after forming the first image, converting the second image captured by the second set of cameras into a second transformed image based at least on a view type;

after converting the first and second images to define the first and second transformed images, respectively, combining the first transformed image and the second transformed image to obtain a combined image with reference to the articulation joint, wherein the combined image is indicative of surroundings of the first section and the second section of the machine; and

displaying, via a display device, the combined image.

18. The method of claim 17 further comprising:

correcting the first image captured by the first set of cameras based on a calibration parameter associated with the first set of cameras; and

correcting the second image by the second set of cameras based on a calibration parameter associated with the second set of cameras.

19. The method of claim 17 further comprising selecting the view type from one of a three dimensional view, a perspective view, a two dimensional view, a top surround view and a side view.