Processing-target image creating device, processing-target image creating method, and operation assisting system

• US 10,621,743 B2
• Filed: 10/08/2015
• Issued: 04/14/2020
• Est. Priority Date: 04/24/2013
• Status: Active Grant

• Current Assignee: Sumitomo Heavy Industries Limited
• Original Assignee: Sumitomo Heavy Industries Limited
• Inventors: Kiyota, Yoshihisa
• Primary Examiner(s): Perungavoor, Sath V
• Assistant Examiner(s): Dobbs, Kristin
• Application Number:
  US14/878,150
• Publication Number:
  US 20160035110A1
• Time in Patent Office: 1,650 Days
• Field of Search: 348148
• CPC Class Codes: B60R 1/00 Optical viewing arrangements; Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles
• CPC Class Codes: B60R 1/27 providing all-round vision, e.g. using omnidirectional cameras
• CPC Class Codes: B60R 2300/607 from a bird's eye viewpoint
• CPC Class Codes: G06T 2200/32 involving image mosaicing
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 3/00 Geometric image transformations in the plane of the image
• CPC Class Codes: G06T 3/12 Panospheric to cylindrical image transformations
• CPC Class Codes: G06T 7/33 using feature-based methods
• CPC Class Codes: G06T 7/60 Analysis of geometric attributes
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle

Abstract

A processing-target image creating device creates a processing-target image based on a plurality of input images. The processing-target image creating device includes a coordinates-associating part configured to associate coordinates on each input image plane on which each input image is positioned, coordinates on a space model onto which each input image is projected, and coordinates on a processing-target image plane, and an image adjusting part configured to adjust the processing-target image by determining for each input image a re-projection angle of a re-projection line joining the coordinate and the coordinate. The image adjusting part determines for each input image slopes of the re-projection lines such that each coordinate on the processing-target image plane corresponding to an infinity line in each input image corresponds with each other on a dividing line between image parts corresponding to each input image on the processing-target image plane.

Claims

1. An image creating device comprising:

a memory; and

a processor coupled to the memory, and configured to

create an output image based on a plurality of input images captured by a plurality of cameras installed at different heights, the output image having a shape of a partial circle; and

associate coordinates on a plurality of input image planes in each of which a corresponding one of the plurality of input images is positioned, coordinates on a space model onto which the plurality of input images are projected, coordinates on a re-projection plane onto which the plurality of input images projected on the space model are re-projected, and coordinates on an output image plane in which the output image is positioned,

wherein the plurality of input images include respective feature images representing a predetermined feature, and a slope of a re-projection line relative to a re-projection axis that is a central axis of the space model is adjusted independently for each of the plurality of input images such that the feature images of the plurality of input images are continuous in a circumferential direction of the partial circle in the output image, the re-projection line being a straight line joining first coordinates on the space model and second coordinates on the re-projection plane corresponding to the first coordinates,

the predetermined feature is an infinity line or a white line horizontally drawn on a wall surface around a running body of a jib crane or a turning body of a shovel,

when the predetermined feature is the infinity line, the slope of the re-projection line relative to the re-projection axis is adjusted independently for each of the plurality of input images such that positions corresponding to the infinity line in a plurality of image parts of the output image are continuous, the plurality of image parts corresponding to the plurality of input images, so that a sky appears same in the plurality of image parts, and

when the predetermined feature is the white line, the slope of the re-projection line relative to the re-projection axis is adjusted independently for each of the plurality of input images such that respective image portions of the plurality of image parts of the output image, which image portions represent the white line, are continuous, the plurality of image parts corresponding to the plurality of input images.

2. The image creating device as claimed in claim 1, wherein the processor is configured to

update the slope of the re-projection line relative to the re-projection axis independently for each of the plurality of input images in response to a change in a cylindrical radius of the space model, such that a continuity of the predetermined feature is maintained through the plurality of image parts of the output image, the plurality of image parts corresponding to the plurality of input images;

and

change allocation rates of a horizontal image area and a road surface image area while maintaining the continuity of the predetermined feature in the output image.

3. An image creating device comprising:

a memory; and

a processor coupled to the memory, and configured to create an output image by synthesizing a plurality of input images captured by a plurality of cameras installed at different heights,

wherein a planar overhead image is created as the output image by processing the plurality of input images such that a viewpoint of the output image is different from viewpoints of the plurality of input images, the output image including a plurality of image parts corresponding to respective imaging ranges of the plurality of cameras,

the plurality of input images include respective feature images representing a feature extending over the plurality of input images,

an offset between the feature images on a dividing line between the plurality of image parts corresponding to the plurality of input images is reduced in the output image, and

the feature is a white line horizontally drawn on a wall surface around a running body of a jib crane or a turning body of a shovel.

4. The image creating device as claimed in claim 3, wherein

the output image having a shape of a partial circle, and

the feature is continuous in a circumferential direction of the partial circle.

5. The image creating device as claimed in claim 3, wherein

the plurality of cameras are installed at different heights,

coordinates of each of the plurality of input images are associated with coordinates of the output image via coordinates of a space model with the coordinates of the space model and the coordinates of the output image being associated by straight lines, and

inclinations of the straight lines are individually determined to be different from each other such that coordinates of the output image corresponding to an image of the feature in one of the plurality of input images coincide with coordinates of the output image corresponding to an image of the feature in another one of the plurality of input images adjacent to said one of the plurality of input images.

6. The image creating device as claimed in claim 3, wherein the processor is configured to

associate coordinates on a plurality of input image planes in each of which a corresponding one of the plurality of input images is positioned, coordinates on a space model onto which the plurality of input images are projected, coordinates on a re-projection plane onto which the plurality of input images projected on the space model are re-projected, and coordinates on an output image plane in which the output image is positioned, and

adjust the output image by determining a slope of a re-projection line relative to a re-projection axis that is a central axis of the space model independently for each of the plurality of input images, the re-projection line being a straight line joining first coordinates on the space model and second coordinates on the re-projection plane corresponding to the first coordinates.

7. The image creating device as claimed in claim 3,

wherein the plurality of cameras are attached to the running body of the jib crane, and

wherein, in the output image, an image of surroundings viewed from the running body in a horizontal direction is shown around an image of a neighborhood of the running body viewed from above the running body, and a horizon is shown in an arc shape.

8. The image creating device as claimed in claim 3, wherein

the plurality of cameras are attached to the turning body of the shovel, and

in the output image, an image of surroundings viewed from the turning body in a horizontal direction is shown around an image of a neighborhood of the turning body viewed from above the turning body, and a horizon is shown in an arc shape.

9. An image creating device comprising:

a memory; and

a processor coupled to the memory, and configured to create an output image by synthesizing a plurality of input images captured by a plurality of cameras installed at different heights,

wherein a planar overhead image is created as the output image by processing the plurality of input images such that a viewpoint of the output image is different from viewpoints of the plurality of input images, the output image including a plurality of image parts corresponding to respective imaging ranges of the plurality of cameras,

the plurality of input images include respective feature images representing a feature extending over the plurality of input images,

an offset between the feature images on a dividing line between the plurality of image parts corresponding to the plurality of input images is reduced in the output image,

the feature is an infinity line, and

an offset between positions corresponding to the infinity line in the plurality of image parts of the output image on the dividing line is reduced, so that a sky appears same in the plurality of image parts.

10. The image creating device as claimed in claim 9, wherein

the positions corresponding to the infinity line in the plurality of image parts of the output image are caused to be continuous or move based on a position of the infinity line input by an operator.

11. The image creating device as claimed in claim 9, wherein

the positions corresponding to the infinity line in the plurality of image parts of the output image are caused to be continuous or move directly by an input of a position of the infinity line by an operator.

12. The image creating device as claimed in claim 9, wherein

the plurality of input images are a left image, a back image, and a right image captured by a left camera, a back camera, and a right camera provided on a left, a back, and a right of a turning body of a shovel, respectively, and

the offset between the positions corresponding to the infinity line in the plurality of image parts of the output image on the dividing line is reduced, the plurality of image parts corresponding to the left image, the back image, and the right image.

13. The image creating device as claimed in claim 9, wherein

the output image having a shape of a partial circle, and

the feature is continuous in a circumferential direction of the partial circle.

14. The image creating device as claimed in claim 9, wherein

the plurality of cameras are installed at different heights,

coordinates of each of the plurality of input images are associated with coordinates of the output image via coordinates of a space model with the coordinates of the space model and the coordinates of the output image being associated by straight lines, and

inclinations of the straight lines are individually determined to be different from each other such that coordinates of the output image corresponding to an image of the feature in one of the plurality of input images coincide with coordinates of the output image corresponding to an image of the feature in another one of the plurality of input images adjacent to said one of the plurality of input images.

15. The image creating device as claimed in claim 9, wherein the processor is configured to

associate coordinates on a plurality of input image planes in each of which a corresponding one of the plurality of input images is positioned, coordinates on a space model onto which the plurality of input images are projected, coordinates on a re-projection plane onto which the plurality of input images projected on the space model are re-projected, and coordinates on an output image plane in which the output image is positioned, and

adjust the output image by determining a slope of a re-projection line relative to a re-projection axis that is a central axis of the space model independently for each of the plurality of input images, the re-projection line being a straight line joining first coordinates on the space model and second coordinates on the re-projection plane corresponding to the first coordinates.

16. The image creating device as claimed in claim 9,

wherein the plurality of cameras are attached to a running body of a jib crane, and

wherein, in the output image, an image of surroundings viewed from the running body in a horizontal direction is shown around an image of a neighborhood of the running body viewed from above the running body, and a horizon is shown in an arc shape.

17. The image creating device as claimed in claim 9, wherein

the plurality of cameras are attached to a turning body of a shovel, and

in the output image, an image of surroundings viewed from the turning body in a horizontal direction is shown around an image of a neighborhood of the turning body viewed from above the turning body, and a horizon is shown in an arc shape.