Distance and direction estimation of a target point from a vehicle using monocular video camera

• US 9,802,539 B2
• Filed: 04/01/2015
• Issued: 10/31/2017
• Est. Priority Date: 04/01/2015
• Status: Active Grant

• Current Assignee: Robert Bosch GmbH, Robert Bosch LLC (Robert Bosch GmbH)
• Original Assignee: Robert Bosch GmbH
• Inventors: Gehrke, Mark
• Primary Examiner(s): Geroleo, Francis G
• Application Number:
  US14/675,862
• Publication Number:
  US 20160288711A1
• Time in Patent Office: 944 Days
• Field of Search: 348 36
• CPC Class Codes: B60R 1/00 Optical viewing arrangements; Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles
• CPC Class Codes: B60R 1/26 to the rear of the vehicle
• CPC Class Codes: B60R 2300/402 Image calibration
• CPC Class Codes: B60R 2300/8093 for obstacle warning
• CPC Class Codes: G06T 2207/30261 Obstacle
• CPC Class Codes: G06T 3/00 Geometric image transformations in the plane of the image
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads

Abstract

A method and system for determining a distance and direction between a video camera secured on a vehicle and a target point relies on an electronic control unit. The system maps and stores grid points representing a world coordinate grid onto a screen coordinate grid and displays the video image on a display using the screen coordinate grid. The system obtains a target point of an object in the video image and determines a locus of four closest grid points of the screen coordinate grid that encircle the target point. The system determines screen distances from the target point to each of the four grid points and maps the four grid points onto the world coordinate grid. The electronic control unit interpolates the location of the target point in the world coordinate grid as weighted by the screen distances. Using the video camera location in world coordinates and the target point location in world coordinates, the system determines a distance between the video camera and the target point.

Claims

1. A method of determining a distance and direction between a video camera provided on a vehicle and a target point as a point of interest of an object in a field of view of the video camera, comprising:

pre-initializing by mapping and storing in an electronic control unit a grid of points representing a world coordinate grid mapped onto a screen coordinate grid;

generating a video image with the video camera and providing the video image to an electronic control unit including a processor and a memory;

displaying the video image on a display using the screen coordinate grid;

obtaining the target point as the point of interest for the object in the video image;

determining a locus of four grid points of the screen coordinate grid that encircle the target point;

determining and storing screen distances from the target point to each of the four grid points encircling the target point;

mapping the four grid points from the screen coordinate grid encircling the target point onto the world coordinate grid;

applying interpolation to determine a location of the target point in the world coordinate grid as weighted by the screen distances; and

using a known location of the video camera in world coordinates and the determined location of the target point in world coordinates, determining a distance between the location of the video camera and the target point of the object.

2. The method according to claim 1, wherein applying interpolation to determine the location of the target point comprises applying bi-linear interpolation.

3. The method according to claim 2, wherein the bi-linear interpolation for the location of the target point comprises:

weighting each of the four grid points in the world coordinates encircling the target point by a square of an inverse of the screen distance determined for each of the four grid points in the screen coordinate grid; and

normalizing by a sum of the inverse squared distances to determine the location of the target point in world coordinates.

4. The method according to claim 1, wherein the video camera comprises a monocular panoramic video camera.

5. The method according to claim 1, wherein the pre-initializing by mapping and storing the grid of points representing the world coordinate grid onto the screen coordinate grid, comprises:

providing extrinsic video camera calibration parameters to the electronic control unit;

providing the world coordinate grid of world coordinates with known physical dimensions;

transforming the world coordinate grid to the screen coordinate grid in view of the calibration parameters; and

providing the screen coordinate grid of the screen coordinates.

6. The method according to claim 5, wherein the extrinsic video camera calibration parameters include lens properties of a lens of the video camera.

7. The method according to claim 6, wherein the video camera comprises a monocular panoramic video camera.

8. The method according to claim 1, including determining changes in distance from the video camera provided on the vehicle to the target point of the object to avoid a collision.

9. The method according to claim 1, wherein the display comprises a touch screen and obtaining the target point as the point of interest for the object shown in the video image comprises touching the touch screen at the target point of the object in the video image di splayed thereon.

10. The method according to claim 9, wherein the obtaining of the target point as the point of interest for the object shown in the video image further comprises:

enlarging and centering the object shown in the video image at the touched target point on the touch screen;

selecting an exact target point of the object in the enlarged video image by touching the touch screen at the target point a second time; and

defining the exact target point in response to the touching of the touch screen at the target point the second time.

11. A system for determining a distance and direction between a video camera provided on a vehicle and a target point as a point of interest of an object in a field of view of the video camera, comprising:

a video camera secured to a vehicle for generating video images; and

an electronic control unit including a processor and a memory, the electronic control unit configured to;

pre-initialize by mapping and storing in memory a grid of points representing a world coordinate grid mapped onto a screen coordinate grid;

receive video images from the video camera;

display the video image on a display using the screen coordinate grid;

obtain the target point as the point of interest for the object shown in the video image on the display;

determine a locus of four grid points of the screen coordinate grid that encircle the target point;

determine and store screen distances from the target point to each of the four grid points;

map the four grid points encircling the target point from the screen coordinate grid onto the world coordinate grid;

apply interpolation to determine a location of the target point in the world coordinate grid as weighted by the screen distances; and

using a known location of the video camera in world coordinates and the location of the target point in world coordinates, determine a distance and a direction from the location of the video camera to the target point of the object.

12. The system according to claim 11, wherein the interpolation to determine the location of the target point comprises bi-linear interpolation.

13. The system according to claim 12, wherein the bi-linear interpolation for the location of the target point further comprises:

weighting each of the four grid points in the world coordinates that encircle the target point by a square of an inverse of the screen distance determined for each of the four grid points of the screen coordinate grid; and

normalizing by a sum of the inverse squared screen distances to determine the location of the target point in world coordinates.

14. The system according to claim 11, wherein the video camera comprises a monocular panoramic video camera.

15. The system according to claim 11, wherein the electronic control unit is configured to pre-initialize by mapping and storing in the memory the grid of points representing the world coordinate grid mapped onto the screen coordinate grid by:

obtaining extrinsic video camera calibration parameters for the video camera;

obtaining the world coordinate grid of world coordinates with known physical dimensions;

transforming the world coordinates to screen coordinates in view of the calibration parameters; and

providing the screen coordinate grid with the transformed world coordinates.

16. The system according to claim 15, wherein the extrinsic video camera calibration parameters include lens properties of a lens of the video camera.

17. The system according to claim 16, wherein the video camera comprises a monocular panoramic video camera.

18. The system according to claim 11, wherein the electronic control unit is configured to determine changes in distance from the video camera provided on the vehicle to the target point of the object to avoid a collision.

19. The system according to claim 11, wherein the display comprises a touch screen and the electronic control unit is configured to receive signals from touch of the touch screen by an operator at the target point.

20. The system according to claim 19, wherein the electronic control unit is configured to:

enlarge and center the object shown in the video image on the touch screen at the touched target point; and

receive signals from another touch of the enlarged video image on the touch screen by an operator to obtain an exact target point of the object in the enlarged video image; and

define and store the exact target point on the enlarged video image.

21. A method of determining a distance and direction between a video camera provided on a vehicle and a target point as a point of interest of an object in a field of view of the video camera, comprising:

pre-initializing by mapping and storing in an electronic control unit a grid of points representing a world coordinate grid mapped onto a screen coordinate grid;

generating a video image with the video camera and providing the video image to an electronic control unit including a processor and a memory;

displaying the video image on a display using the screen coordinate grid;

obtaining the target point as the point of interest for the object in the video image;

determining a locus of grid points of the screen coordinate grid that encircle the target point;

determining and storing screen distances from the target point to each of the grid points encircling the target point;

mapping the grid points from the screen coordinate grid encircling the target point onto the world coordinate grid;

applying bi-linear interpolation to determine a location of the target point in the world coordinate grid as weighted by the screen distances; and

using a known location of the video camera in world coordinates and the determined location of the target point in world coordinates, determining a distance between the location of the video camera and the target point of the object.