Vehicle vision system using kinematic model of vehicle motion

• US 10,266,115 B2
• Filed: 07/10/2017
• Issued: 04/23/2019
• Est. Priority Date: 05/21/2013
• Status: Active Grant

• Current Assignee: Magna Electronics Incorporated (Magna International Incorporated)
• Original Assignee: Magna Electronics Incorporated (Magna International Incorporated)
• Inventors: Turk, James
• Primary Examiner(s): Jeanglaude, Gertrude Arthur
• Application Number:
  US15/644,987
• Publication Number:
  US 20170305347A1
• Time in Patent Office: 652 Days
• Field of Search: 701300
• CPC Class Codes: B60R 2300/105 using multiple cameras
• CPC Class Codes: B60R 2300/30 characterised by the type of image processing
• CPC Class Codes: B60R 2300/402 Image calibration
• CPC Class Codes: B60R 2300/802 for monitoring and displaying vehicle exterior blind spot views
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration

Abstract

A vision system for a vehicle includes a camera disposed at an outside rearview mirror assembly and having a field of view exterior of the vehicle. A control includes an electronic control unit having an image processor operable to process image data captured by the camera. Responsive to image processing of captured image data, the control is operable to determine objects present in the field of view of the camera. Responsive to processing at the electronic control unit of received vehicle data, the control determines a vehicle motion vector during driving of the vehicle. The control determines movement of an object relative to the vehicle via image processing of at least two frames of captured image data during driving of the vehicle. The control compares the determined relative movement of the object to the determined vehicle motion vector to determine misalignment of the camera.

Claims

1. A vision system for a vehicle, said vision system comprising:

a camera disposed at an outside rearview mirror assembly of a vehicle equipped with said vision system;

said camera having a field of view exterior of the equipped vehicle;

wherein said camera comprises a two dimensional array of a plurality of photosensing elements;

wherein said camera is operable to capture frames of image data;

a control comprising an electronic control unit, said electronic control unit comprising an image processor operable to process image data captured by said camera and provided to said electronic control unit;

wherein, responsive to image processing by said image processor of captured image data, said control is operable to determine objects present in the field of view of said camera;

wherein vehicle data is received at said electronic control unit via a communication bus of the equipped vehicle;

wherein, responsive to processing at said electronic control unit of received vehicle data, said control determines a vehicle motion vector during driving of the equipped vehicle by a driver of the equipped vehicle;

wherein the vehicle data includes speed of the equipped vehicle and at least two of (i) steering information of the equipped vehicle, (ii) vehicle geometry of the equipped vehicle and (iii) distance traveled by the equipped vehicle;

wherein said control determines movement of an object relative to the equipped vehicle via image processing by said image processor of at least two frames of captured image data during driving of the equipped vehicle by the driver of the equipped vehicle;

wherein said control compares the determined relative movement of the object to the determined vehicle motion vector;

wherein, responsive to a difference between the determined relative movement of the object and the determined vehicle motion vector, said control determines misalignment of said camera; and

wherein, responsive at least in part to determination of misalignment of said camera, calibration of said camera is adjusted so as to have location and relative movement of detected objects based on image processing coincide with their predicted location and movement based on the actual kinematic motion of the equipped vehicle.

2. The vision system of claim 1, wherein the vehicle data is communicated to said control via a CAN communication bus of the equipped vehicle.

3. The vision system of claim 1, wherein a kinematic model comprising a bicycle model is used to represent vehicle motion, and wherein said kinematic model utilizes translational vehicle motion.

4. The vision system of claim 3, wherein said kinematic model utilizes at least one of (i) wheel steering angle and (ii) wheel pulse count.

5. The vision system of claim 3, wherein said kinematic model utilizes wheel steering angle and wheel pulse count.

6. The vision system of claim 1, wherein said camera comprises one of a plurality of cameras disposed at the equipped vehicle, each camera of said plurality of cameras having a respective field of view exterior of the equipped vehicle.

7. The vision system of claim 6, wherein image data captured by at least some of said plurality of cameras is used for a surround view system of the equipped vehicle.

8. The vision system of claim 6, wherein said camera is disposed at a driver-side outside rearview mirror assembly of the equipped vehicle, and wherein said plurality of cameras further comprises (i) a rear camera disposed at a rear portion of the equipped vehicle and having at least a rearward field of view, (ii) a passenger side camera disposed at a passenger side outside rearview mirror assembly of the equipped vehicle and having at least a sideward and rearward field of view and (iii) a front camera disposed at a front portion of the equipped vehicle and having at least a forward field of view.

9. The vision system of claim 8, wherein said camera comprises a CMOS camera.

10. A vision system for a vehicle, said vision system comprising:

a camera disposed at an outside rearview mirror assembly of a vehicle equipped with said vision system;

said camera having a field of view exterior of the equipped vehicle;

wherein said camera comprises a two dimensional array of a plurality of photosensing elements;

wherein said camera is operable to capture frames of image data;

a control comprising an electronic control unit, said electronic control unit comprising an image processor operable to process image data captured by said camera and provided to said electronic control unit;

wherein, responsive to image processing by said image processor of captured image data, said control is operable to determine objects present in the field of view of said camera;

wherein vehicle data is received at said electronic control unit via a communication bus of the equipped vehicle;

wherein, responsive to processing at said electronic control unit of received vehicle data, said control determines a vehicle motion vector during driving of the equipped vehicle by a driver of the equipped vehicle;

wherein the vehicle data includes speed of the equipped vehicle and at least two of (i) steering information of the equipped vehicle, (ii) vehicle geometry of the equipped vehicle and (iii) distance traveled by the equipped vehicle;

wherein said control determines movement of an object relative to the equipped vehicle via image processing by said image processor of at least two frames of captured image data during driving of the equipped vehicle by the driver of the equipped vehicle;

wherein said control compares the determined relative movement of the object to the determined vehicle motion vector;

wherein, responsive to a difference between the determined relative movement of the object and the determined vehicle motion vector, said control determines misalignment of said camera;

wherein a kinematic model comprising a bicycle model is used to represent vehicle motion; and

wherein said kinematic model at least one of (i) utilizes lateral vehicle dynamics to develop numerical relations between wheel steering angle, vehicle heading angle and translational vehicle motion and (ii) utilizes wheel pulse counts to develop numerical relations between wheel steering angle, vehicle heading angle and translational vehicle motion.

11. The vision system of claim 10, wherein said kinematic model utilizes (i) steering angle of a front wheel of the equipped vehicle and (ii) steering angle of a rear wheel of the equipped vehicle.

12. The vision system of claim 11, wherein said kinematic model utilizes center of gravity of the equipped vehicle.

13. The vision system of claim 11, wherein said kinematic model utilizes lateral velocity of the equipped vehicle.

14. The vision system of claim 10, wherein said camera comprises a CMOS camera.

15. The vision system of claim 14, wherein the vehicle data is communicated to said control via a CAN communication bus of the equipped vehicle.

16. A vision system for a vehicle, said vision system comprising:

a camera disposed at a driver-side outside rearview mirror assembly of a vehicle equipped with said vision system;

said camera having a field of view exterior of the equipped vehicle;

wherein said camera comprises a two dimensional CMOS array of a plurality of photosensing elements;

wherein said camera is operable to capture frames of image data;

a control comprising an electronic control unit, said electronic control unit comprising an image processor operable to process image data captured by said camera and provided to said electronic control unit;

wherein, responsive to image processing by said image processor of captured image data, said control is operable to determine objects present in the field of view of said camera;

wherein vehicle data is received at said electronic control unit via a CAN communication bus of the equipped vehicle;

wherein, responsive to processing at said electronic control unit of received vehicle data, said control determines a vehicle motion vector during driving of the equipped vehicle by a driver of the equipped vehicle;

wherein the vehicle data includes speed of the equipped vehicle and at least two of (i) steering information of the equipped vehicle, (ii) vehicle geometry and (iii) distance traveled by the equipped vehicle;

wherein said control determines movement of an object relative to the equipped vehicle via image processing by said image processor of at least two frames of captured image data during driving of the equipped vehicle by the driver of the equipped vehicle;

wherein said control compares the determined relative movement of the object to the determined vehicle motion vector;

wherein, responsive to a difference between the determined relative movement of the object and the determined vehicle motion vector, said control determines misalignment of said camera;

wherein said camera comprises one of a plurality of cameras disposed at the equipped vehicle, each camera of said plurality of cameras having a respective field of view exterior of the equipped vehicle;

wherein image data captured by at least some of said plurality of cameras is used for a surround view system of the equipped vehicle; and

wherein said plurality of cameras further comprises (i) a rear camera disposed at a rear portion of the equipped vehicle and having at least a rearward field of view, (ii) a passenger side camera disposed at a passenger side outside rearview mirror assembly of the equipped vehicle and having at least a sideward and rearward field of view and (iii) a front camera disposed at a front portion of the equipped vehicle and having at least a forward field of view.

17. The vision system of claim 16, wherein said control compares the determined vehicle motion vector to an object vector determined between a first position of the object in a first frame of captured image data and a second position of the object in a second frame of captured image data.

18. The vision system of claim 17, wherein said control determines misalignment of said camera at least in part responsive to a difference between direction of the determined vehicle motion vector and direction of the determined object vector.

19. The vision system of claim 18, wherein said control determines misalignment of said camera at least in part responsive to a difference between magnitude of the determined vehicle motion vector and magnitude of the determined object vector.

20. The vision system of claim 19, comprising a kinematic model, wherein said kinematic model is used to represent vehicle motion, and wherein said kinematic model utilizes (i) steering angle of a front wheel of the equipped vehicle and (ii) steering angle of a rear wheel of the equipped vehicle.