Apparatus and method of generating top-view image

• US 10,300,854 B2
• Filed: 06/22/2017
• Issued: 05/28/2019
• Est. Priority Date: 08/31/2016
• Status: Active Grant

• Current Assignee: Hyundai Mobis Co., Ltd.
• Original Assignee: Hyundai Mobis Co., Ltd.
• Inventors: Lee, Seong Soo
• Primary Examiner(s): Teitelbaum, Michael E
• Application Number:
  US15/630,896
• Publication Number:
  US 20180056873A1
• Time in Patent Office: 705 Days
• CPC Class Codes: B60R 1/002 specially adapted for covering the peripheral part of the vehicle, e.g. for viewing tyres, bumpers or the like integrated in the windows B60R1/001; combined with rear-view mirrors B60R1/08
• CPC Class Codes: G06T 2207/20212 Image combination
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 2207/30256 Lane; Road marking
• CPC Class Codes: G06T 3/60 Rotation of whole images or parts thereof
• CPC Class Codes: G06T 7/33 using feature-based methods
• CPC Class Codes: G06T 7/536 from perspective effects, e.g. by using vanishing points
• CPC Class Codes: G06T 7/73 using feature-based methods

Abstract

Provided is a method of generating a top-view image. The method includes: obtaining an input image by photographing a peripheral road surface of the vehicle by using a camera when a vehicle is driving, detecting at least two lanes from the input image, detecting a vanishing point at which the at least two lanes intersect, calculating a first rotation angle of the camera by using the vanishing point, calculating a second rotation angle of the camera for converting the input image into a top-view image represented in a top-view image coordinate system, based on the first rotation angle, and rotation-converting the input image by using the second rotation angle to generate the top-view image.

Claims

1. A method of generating a top-view image, comprising:

obtaining an input image by photographing a peripheral road surface of a vehicle by using a camera when the vehicle is driving;

detecting at least two lanes from the input image;

detecting a vanishing point at which the at least two lanes intersect;

calculating a first rotation angle of the camera by using the vanishing point;

calculating a second rotation angle of the camera for converting the input image into a top-view image represented in a top-view image coordinate system, based on the first rotation angle;

rotation-converting the input image by using the second rotation angle to generate the top-view image; and

verifying the top-view image for alignment between a vertical axis of the vehicle and the top-view image coordinate system after the generating of the top-view image;

wherein the verifying of the top-view image comprises;

detecting a lane from a previous top-view image generated from a previous frame and detecting a lane from a current top-view image generated from a current frame;

determining whether a direction of the lane detected from the previous top-view image matches a direction of the lane detected from the current top-view image;

determining whether an interval between the lane detected from the previous top-view image and the vertical axis of the vehicle matches an interval between the lane detected from the current top-view image and the vertical axis of the vehicle; and

determining the alignment as being made between the vertical axis of the vehicle and the top-view image coordinate system when the direction of the lane detected from the previous top-view image matches the direction of the lane detected from the current top-view image, and the interval between the lane detected from the previous top-view image and the vertical axis of the vehicle matches the interval between the lane detected from the current top-view image and the vertical axis of the vehicle.

2. The method of claim 1, wherein the calculating of the first rotation angle comprises calculating the first rotation angle for moving the vanishing point to a camera principal point of the camera.

3. The method of claim 1, wherein:

the calculating of the first rotation angle enables a road surface to be parallel to an XZ plane, and

the XZ plane comprises an X axis and a Z axis in a camera coordinate system that defines a pitch direction is a rotation direction where rotation is performed with respect to the X axis, a yaw direction is a rotation direction where rotation is performed with respect to a Y axis, and a roll direction is a rotation direction where rotation is performed with respect to the Z axis which is an optical axis of the camera.

4. The method of claim 1, wherein the calculating of the second rotation angle comprises:

when the top-view image coordinate system is expressed as a two-dimensional (2D) coordinate system comprising an X axis vertical to a lane and a Y axis parallel to the lane,

calculating an amount of yaw-direction rotation and an amount of pitch-direction rotation enabling the camera to face a road surface, based on an amount of pitch-direction rotation and an amount of yaw-direction rotation included in the first rotation angle; and

calculating an amount of roll-direction rotation enabling a lane direction to be parallel to the Y axis of the top-view image coordinate system, based on an amount of roll-direction rotation included in the first rotation angle.

5. An apparatus for generating a top-view image, the apparatus comprising:

a camera configured to obtain input images by photographing a peripheral road surface of a vehicle when the vehicle is driving;

a lane detector configured to detect at least two lanes from each of the input images;

a vanishing point detector configured to detect a vanishing point at which the at least two lanes intersect;

a rotation angle calculator configured to calculate a rotation angle for converting each of the input images into an image corresponding to a top-view viewpoint by using the vanishing point;

an image rotation unit configured to rotation-converting the input images by using the rotation angle;

an image synthesizer configured to synthesize the input images rotation-converted at the rotation angle to generate a top-view image; and

a top-view image verifier configured to verify the top-view image for alignment between a vertical axis of the vehicle and the top-view image coordinate system;

wherein the top-view image verifier is configured to verify whether the alignment between the vertical axis of the vehicle and the top-view image coordinate system is made, based on a previous top-view image of a previous frame and a current top-view image of a current frame.

6. The apparatus of claim 5, wherein the rotation angle calculator comprises:

a first rotation angle calculator configured to calculate a first rotation angle for moving the vanishing point to a camera principal point of the camera; and

a second rotation angle calculator configured to calculate a second rotation angle for converting each of the input images into an image represented in a top-view image coordinate system, based on the first rotation angle.

7. The apparatus of claim 6, wherein the first rotation angle calculator calculates the first rotation angle, which allows a road surface to be parallel to an XZ plane comprising an X axis and a Z axis, in a camera coordinate system which defines a pitch direction is a rotation direction where rotation is performed with respect to the X axis, a yaw direction is a rotation direction where rotation is performed with respect to a Y axis, and a roll direction is a rotation direction where rotation is performed with respect to the Z axis which is an optical axis of the camera.

8. The apparatus of claim 6, wherein when the top-view image coordinate system expressed as a two-dimensional (2D) coordinate system comprising an X axis vertical to a lane and a Y axis parallel to the lane, the second rotation angle calculator calculates an amount of yaw-direction rotation and an amount of pitch-direction rotation enabling the camera to face the road surface, based on an amount of pitch-direction rotation and an amount of yaw-direction rotation in the first rotation angle, and calculates an amount of roll-direction rotation enabling a lane direction to be parallel to the Y axis of the top-view image coordinate system, based on an amount of roll-direction rotation in the first rotation angle.

9. The apparatus of claim 5, wherein the top-view image verifier is configured to determine whether a direction of the lane detected from the previous top-view image matches a direction of a lane detected from the current top-view image and an interval between the lane detected from the previous top-view image and the vertical axis of the vehicle matches an interval between the lane detected from the current top-view image and the vertical axis of the vehicle, and verify whether the alignment between a vertical axis of the vehicle and the top-view image coordinate system is made.