Object position measurement with automotive camera using vehicle motion data

• US 10,423,164 B2
• Filed: 03/01/2016
• Issued: 09/24/2019
• Est. Priority Date: 04/10/2015
• Status: Active Grant

• Current Assignee: Robert Bosch GmbH
• Original Assignee: Robert Bosch GmbH
• Inventors: Baran, Matthew
• Primary Examiner(s): Alharbi, Adam M
• Application Number:
  US15/564,788
• Publication Number:
  US 20180181142A1
• Time in Patent Office: 1,302 Days
• CPC Class Codes: B60D 1/36 for facilitating connection, e.g. hitch catchers , visual guide means, signalling aids B60D1/465 takes precedence; vehicle signalling in general B60Q; optical arrangements specially adapted for viewing trailer-hitches B60R1/003
• CPC Class Codes: B60D 1/62 involving supply lines, electric circuits, or the like
• CPC Class Codes: B60R 1/003 for viewing trailer hitches
• CPC Class Codes: B60R 2300/302 combining image information with GPS information or vehicle data, e.g. vehicle speed, gyro, steering angle data
• CPC Class Codes: B60R 2300/8066 for monitoring rearward traffic
• CPC Class Codes: B60R 2300/808 for facilitating docking to a trailer
• CPC Class Codes: B62D 15/0285 Parking performed automatically
• CPC Class Codes: G01C 11/04 Interpretation of pictures
• CPC Class Codes: G01C 11/06 by comparison of two or more pictures of the same area
• CPC Class Codes: G01C 3/10 using a parallactic triangle with variable angles and a base of fixed length in the observation station, e.g. in the instrument
• CPC Class Codes: G05D 1/0253 extracting relative motion information from a plurality of images taken successively, e.g. visual odometry, optical flow determining position or orientation from images G06T7/70
• CPC Class Codes: G06T 2207/20101 Interactive definition of point of interest, landmark or seed
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/248 involving reference images or patches
• CPC Class Codes: G06T 7/579 from motion

Abstract

A method and system of locating a position of an object with an omnidirectional camera affixed to a vehicle. An electronic control unit receives movement data of the vehicle from a vehicle communication bus. Simultaneously, the electronic control unit receives a plurality of omnidirectional image frames of a stationary object with the omnidirectional camera. Based on the received omnidirectional image frames, the electronic control unit identifies an object of interest in the omnidirectional image frames and tracks the object of interest in the omnidirectional image frames while the vehicle is moving. The electronic control unit determines a change in position of the object of interest in the omnidirectional image frames as the vehicle is moving and determines a distance to the object of interest based on the change of position of the vehicle and the change in position of the object of interest in the omnidirectional image frames.

Claims

1. A method of locating a position of an object with an omnidirectional camera affixed to a vehicle, the method comprising:

receiving movement data of the vehicle from vehicle sensors indicative of a change of position of the vehicle;

receiving a plurality of omnidirectional image frames of one or more stationary objects with an omnidirectional camera;

identifying an object of interest from the one or more stationary objects in the omnidirectional image frames,

tracking the object of interest in the omnidirectional image frames, with an electronic control unit, while the vehicle is moving;

determining a change in position of the object of interest in the omnidirectional image frames as the vehicle is moving; and

determining a distance to the object of interest relative to the vehicle based on the change of position of the vehicle and the change in position of the object of interest in the omnidirectional image frames.

2. The method of locating a position of an object according to claim 1, wherein receiving movement data of the vehicle includes receiving a direction of movement and a distance of movement from a vehicle communication bus.

3. The method of locating a position of an object according to claim 1, the method further comprising:

determining a position of the vehicle based on ground positional coordinates of the vehicle in reference to a ground coordinate system;

determining a position of the object of interest based on optical positional coordinates of the object of interest in reference to an optical coordinate system;

transforming the determined position of the vehicle based on ground positional coordinates to a position of the vehicle based on optical positional coordinates in reference to the optical coordinate system.

4. The method of locating a position of an object according to claim 1, the method further comprising:

transforming the omnidirectional image frames to a plurality of rectilinear image frames;

determining a plurality of positions of the object of interest in the rectilinear image frames; and

determining a change in the positions of the object of interest in the rectilinear image frames while the vehicle is moving.

5. The method of locating a position of an object according to claim 1, wherein determining a distance to the object of interest based on the change of position of the vehicle from the movement data and the change in position of the object of interest in the omnidirectional image includes triangulating the movement data of the vehicle and the change in position of the object of interest.

6. The method of locating a position of an object according to claim 3, the method further comprising:

calibrating a relationship between the ground coordinate system and the optical coordinate system.

7. The method of locating a position of an object according to claim 1, the method further comprising transforming the omnidirectional image frames to rectilinear image frames.

8. The method of locating a position of an object according to claim 1, wherein the object of interest is classified as a plurality of optical positional coordinates, and wherein tracking the object of interest in the omnidirectional image includes tracking the optical positional coordinates, and wherein determining a change in position of the object of interest in the omnidirectional image as the vehicle is moving includes determining a change in the optical positional coordinates.

9. The method of locating a position of an object according to claim 8, further comprising performing a bundle adjustment on the coordinate points.

10. The method of locating a position of an object according to claim 1, wherein the object of interest is identified by a user input.

11. The method of locating a position of an object according to claim 1, wherein the object of interest is identified by a touchscreen.

12. The method of locating a position of an object according to claim 1, wherein the object of interest is identified by

analyzing the omnidirectional image frames for a pattern;

comparing the pattern to a predetermined set of objects; and

classifying the pattern as a specific object type from the predetermined set of objects.

13. The method of locating a position of an object according to claim 1, wherein the object of interest is one selected from the group consisting of another vehicle, a curb, a tree, a pole, and a ground plane.

14. The method of locating a position of an object according to claim 1, further comprising automatically maneuvering the vehicle based on the position of the object of interest.

15. An object locating system for locating a position of an object with an omnidirectional camera affixed to a vehicle, the system comprising:

at least one vehicle sensor configured to indicate a change of position of the vehicle;

an omnidirectional camera;

a user interface configured to display video information to a driver of the vehicle; and

an electronic control unit configured to

receive a plurality of omnidirectional image frames from the omnidirectional camera including an omnidirectional image of a one or more stationary objects,

receive movement data of the vehicle from the at least one vehicle sensor indicative of a change of position of the vehicle;

identifying an object of interest from the one or more stationary objects in the omnidirectional image frames,

tracking the object of interest in the plurality of omnidirectional image frames while the vehicle is moving;

determining a change in position of the object of interest in the omnidirectional image frames as the vehicle is moving; and

determining a distance to the object of interest based on the change of position of the vehicle and the change in position of the object of interest in the omnidirectional image frames.

16. The object locating system according to claim 15, wherein the at least one vehicle sensor includes a wheel speed sensor and a steering angle sensor.

17. The object locating system according to claim 15, wherein the electronic control unit is further configured to triangulate the change of position of the vehicle and the change in position of the object of interest in the omnidirectional image frames to determine a distance to the object of interest.

18. The object locating system according to claim 15, wherein the electronic control unit is further configured to transform the omnidirectional image to a rectilinear image before calculating distance to the object of interest.

19. The object locating system according to claim 15, further comprising a touchscreen configured to receive an input from a driver identifying the object of interest to the electronic control unit.

20. The object locating system according to claim 15, wherein the object of interest is one selected from the group consisting of another vehicle, a curb, a tree, a pole, and a ground plane.