VEHICLE VISION SYSTEM USING KINEMATIC MODEL OF VEHICLE MOTION

• US 20160082887A1
• Filed: 12/07/2015
• Published: 03/24/2016
• Est. Priority Date: 05/21/2013
• Status: Active Grant

• Current Assignee: Magna Electronics Incorporated (Magna International Incorporated)
• Original Assignee: Magna Electronics Incorporated (Magna International Incorporated)
• Inventors: Turk, James
• Granted Patent: US 9,701,246 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: B60R 1/00 Optical viewing arrangements; Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles
• CPC Class Codes: B60R 2300/105 using multiple cameras
• CPC Class Codes: B60R 2300/30 characterised by the type of image processing
• CPC Class Codes: B60R 2300/402 Image calibration
• CPC Class Codes: B60R 2300/802 for monitoring and displaying vehicle exterior blind spot views
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration

Abstract

A vision system for a vehicle includes a plurality of cameras disposed at the vehicle and respective fields of view exterior of the vehicle. An electronic control unit includes an image processor operable to process image data captured by any or all of the cameras. Responsive to image processing of captured image data, the electronic control unit is operable to determine objects present in the field of view of at least one of the cameras. Responsive to vehicle data, the electronic control unit determines a vehicle motion vector during driving of the vehicle by a driver of the vehicle. The electronic control unit compares the determined relative movement of a determined object to the determined vehicle motion vector and, responsive to a difference between the determined relative movement of the object and the determined vehicle motion vector, the electronic control unit determines a misalignment of the at least one camera.

Claims

1. A vision system for a vehicle, said vision system comprising:

a plurality of cameras disposed at a vehicle equipped with said vision system;

said plurality of cameras comprising at least (i) a first camera disposed at a rear portion of the equipped vehicle, (ii) a second camera disposed at a driver side portion of the equipped vehicle and (iii) a third camera disposed at a passenger side of the equipped vehicle;

wherein each of said first, second and third cameras has a respective field of view exterior of the equipped vehicle;

wherein each of said first, second and third cameras is operable to capture frames of image data;

an electronic control unit, said electronic control unit comprising an image processor operable to process image data captured by any or all of said first, second and third cameras;

wherein, responsive to image processing by said image processor of captured image data, said electronic control unit is operable to determine objects present in the field of view of at least one of said first, second and third cameras;

wherein, responsive to vehicle data, said electronic control unit determines a vehicle motion vector during driving of the equipped vehicle by a driver of the equipped vehicle;

wherein the vehicle data includes steering information of the equipped vehicle and at least one of (i) speed of the equipped vehicle and (ii) distance traveled by the equipped vehicle;

wherein, during driving of the equipped vehicle by the driver of the equipped vehicle, said electronic control unit determines movement of an object relative to the equipped vehicle via image processing by said image processor of at least two frames of image data captured by said at least one of said first, second and third cameras;

wherein said electronic control unit compares the determined relative movement of the object to the determined vehicle motion vector;

wherein, responsive to a difference between the determined relative movement of the object and the determined vehicle motion vector, said electronic control unit determines a misalignment of said at least one of said first, second and third cameras; and

wherein image data captured by cameras of said plurality of cameras is used by a surround view system of the equipped vehicle.

2. The vision system of claim 1, wherein the vehicle data is communicated to the electronic control unit via a communication bus of the equipped vehicle.

3. The vision system of claim 1, wherein a kinematic model comprising a bicycle model is used by said electronic control unit to represent vehicle motion.

4. The vision system of claim 3, wherein said kinematic model utilizes at least one of (i) a wheel steering angle of the equipped vehicle, (ii) a wheel pulse of the equipped vehicle, (iii) a heading angle of the equipped vehicle and (iv) translational vehicle motion of the equipped vehicle.

5. The vision system of claim 1, wherein the vehicle data includes geometry of the equipped vehicle.

6. The vision system of claim 1, wherein, responsive to a determined misalignment of said at least one of said first, second and third cameras, said electronic control unit is operable to calibrate said at least one of said first, second and third cameras.

7. The vision system of claim 1, wherein said electronic control unit compares the determined vehicle motion vector to an object vector determined between a first position of the object in a first frame of image data captured by said at least one of said first, second and third cameras and a second position of the object in a second frame of image data captured by said at least one of said first, second and third cameras.

8. The vision system of claim 7, wherein said electronic control unit determines a misalignment of said at least one of said first, second and third cameras responsive to a difference between a direction or magnitude of the determined vehicle motion vector and a direction or magnitude of the determined object vector.

9. The vision system of claim 1, wherein the vehicle data is communicated to the electronic control unit via a CAN communication bus of the equipped vehicle.

10. The vision system of claim 9, wherein the vehicle data includes geometry of the equipped vehicle.

11. The vision system of claim 1, wherein the field of view of said first camera comprises a field of view rearward of the equipped vehicle, and wherein the field of view of said second camera comprises a field of view sideward of the driver side of the equipped vehicle, and wherein the field of view of said third camera comprises a field of view sideward of the passenger side of the equipped vehicle.

12. The vision system of claim 11, wherein said at least one of said first, second and third cameras comprises said second camera and wherein said second camera is disposed at a driver side exterior rearview mirror assembly of the equipped vehicle.

13. The vision system of claim 12, wherein said second camera comprises a photosensor array having at least 1 million CMOS photosensor elements arranged in rows and columns.

14. The vision system of claim 12, wherein said at least one of said first, second and third cameras comprises said first camera, and wherein said electronic control unit is operable to stitch together image data captured at least by said first and second cameras to form a substantially seamless birds-eye view for display by a display screen of the equipped vehicle for viewing by the driver of the equipped vehicle when operating the equipped vehicle.

15. The vision system of claim 14, wherein said plurality of cameras comprises a fourth camera disposed at a front portion of the equipped vehicle, said fourth camera having a field of view exterior of the equipped vehicle, wherein the field of view of said fourth camera comprises a field of view forward of the equipped vehicle, wherein said fourth camera is operable to capture frames of image data, and wherein said image processor of said electronic control unit is operable to process image data captured by said fourth camera.

16. A vision system for a vehicle, said vision system comprising:

a plurality of cameras disposed at a vehicle equipped with said vision system;

said plurality of cameras comprising at least (i) a first camera disposed at a rear portion of the equipped vehicle, (ii) a second camera disposed at a driver side portion of the equipped vehicle, (iii) a third camera disposed at a passenger side of the equipped vehicle and (iv) a fourth camera disposed at a front portion of the equipped vehicle;

wherein each of said first, second, third and fourth cameras has a respective field of view exterior of the equipped vehicle;

wherein each of said first, second, third and fourth cameras is operable to capture frames of image data;

an electronic control unit, said electronic control unit comprising an image processor operable to process image data captured by any or all of said first, second, third and fourth cameras;

wherein, responsive to image processing by said image processor of image data captured by said second camera, said electronic control unit is operable to determine objects present in the field of view of said second camera;

wherein, responsive to vehicle data, said electronic control unit determines a vehicle motion vector during driving of the equipped vehicle by a driver of the equipped vehicle;

wherein the vehicle data includes steering information of the equipped vehicle and at least one of (i) speed of the equipped vehicle and (ii) distance traveled by the equipped vehicle;

wherein, during driving of the equipped vehicle by the driver of the equipped vehicle, said electronic control unit determines movement of an object within the field of view of said second camera relative to the equipped vehicle via image processing by said image processor of at least two frames of image data captured by said second camera;

wherein said electronic control unit compares the determined relative movement of the object to the determined vehicle motion vector; and

wherein, responsive to a difference between the determined relative movement of the object and the determined vehicle motion vector, said electronic control unit determines a misalignment of said second camera.

17. The vision system of claim 16, wherein the vehicle data is communicated to the electronic control unit via a communication bus of the equipped vehicle and wherein the vehicle data includes geometry of the equipped vehicle.

18. The vision system of claim 17, wherein, responsive to a determined misalignment of said second camera, said electronic control unit is operable to calibrate said second camera.

19. The vision system of claim 17, wherein said electronic control unit compares the determined vehicle motion vector to an object vector determined between a first position of the object in a first frame of image data captured by said second camera and a second position of the object in a second frame of image data captured by said second camera, and wherein said electronic control unit determines a misalignment of said second camera responsive to a difference between a direction or magnitude of the determined vehicle motion vector and a direction or magnitude of the determined object vector.

20. The vision system of claim 17, wherein the field of view of said first camera comprises a field of view rearward of the equipped vehicle, and wherein the field of view of said second camera comprises a field of view sideward of the driver side of the equipped vehicle, and wherein the field of view of said third camera comprises a field of view sideward of the passenger side of the equipped vehicle, and wherein the field of view of said fourth camera comprises a field of view forward of the equipped vehicle, and wherein said second camera is disposed at a driver side exterior rearview mirror assembly of the equipped vehicle, and wherein said second camera comprises a photosensor array having at least 1 million CMOS photosensor elements arranged in rows and columns.

21. The vision system of claim 20, wherein image data captured by cameras of said plurality of cameras is used by a surround view system of the equipped vehicle to generate a birds-eye view of the environment at least partially surrounding the equipped vehicle for display by a display screen of the equipped vehicle that is viewable by the driver of the equipped vehicle, and wherein said electronic control unit is operable to stitch together image data captured at least by said first and second cameras to form a substantially seamless birds-eye view for display by said display screen of the equipped vehicle for viewing by the driver of the equipped vehicle when operating the equipped vehicle.

22. The vision system of claim 21, wherein a kinematic model comprising a bicycle model is used by said electronic control unit to represent vehicle motion, and wherein said kinematic model utilizes at least one of (i) a wheel steering angle of the equipped vehicle, (ii) a wheel pulse of the equipped vehicle, (iii) a heading angle of the equipped vehicle and (iv) translational vehicle motion of the equipped vehicle.

23. A vision system for a vehicle, said vision system comprising:

a plurality of cameras disposed at a vehicle equipped with said vision system;

said plurality of cameras comprising at least (i) a first camera disposed at a rear portion of the equipped vehicle, (ii) a second camera disposed at a driver side portion of the equipped vehicle, (iii) a third camera disposed at a passenger side of the equipped vehicle and (iv) a fourth camera disposed at a front portion of the equipped vehicle;

wherein each of said first, second, third and fourth cameras has a respective field of view exterior of the equipped vehicle;

wherein each of said first, second, third and fourth cameras is operable to capture frames of image data;

an electronic control unit, said electronic control unit comprising an image processor operable to process image data captured by any or all of said first, second, third and fourth cameras;

wherein, responsive to image processing by said image processor of image data captured by said third camera, said electronic control unit is operable to determine objects present in the field of view of said third camera;

wherein, responsive to vehicle data, said electronic control unit determines a vehicle motion vector during driving of the equipped vehicle by a driver of the equipped vehicle;

wherein the vehicle data includes steering information of the equipped vehicle and at least one of (i) speed of the equipped vehicle and (ii) distance traveled by the equipped vehicle;

wherein the vehicle data includes geometry of the equipped vehicle;

wherein the vehicle data is communicated to the electronic control unit via a communication bus of the equipped vehicle;

wherein, during driving of the equipped vehicle by the driver of the equipped vehicle, said electronic control unit determines movement of an object within the field of view of said third camera relative to the equipped vehicle via image processing by said image processor of at least two frames of image data captured by said third camera;

wherein said electronic control unit compares the determined relative movement of the object to the determined vehicle motion vector;

wherein said electronic control unit compares the determined vehicle motion vector to an object vector determined between a first position of the object in a first frame of image data captured by said third camera and a second position of the object in a second frame of image data captured by said third camera, and wherein said electronic control unit determines a misalignment of said third camera responsive to a difference between a direction or magnitude of the determined vehicle motion vector and a direction or magnitude of the determined object vector;

wherein, responsive to a difference between the determined relative movement of the object and the determined vehicle motion vector, said electronic control unit determines a misalignment of said third camera; and

wherein, responsive to a determined misalignment of said third camera, said electronic control unit is operable to calibrate said third camera.

24. The vision system of claim 23, wherein the field of view of said first camera comprises a field of view rearward of the equipped vehicle, and wherein the field of view of said second camera comprises a field of view sideward of the driver side of the equipped vehicle, and wherein the field of view of said third camera comprises a field of view sideward of the passenger side of the equipped vehicle, and wherein the field of view of said fourth camera comprises a field of view forward of the equipped vehicle, and wherein said third camera comprises a photosensor array having at least 1 million CMOS photosensor elements arranged in rows and columns, and wherein said electronic control unit is operable to stitch together image data captured at least by said first and third cameras to form a substantially seamless birds-eye view for display by a display screen of the equipped vehicle for viewing by the driver of the equipped vehicle when operating the equipped vehicle.

25. The vision system of claim 23, wherein image data captured by cameras of said plurality of cameras is used by a surround view system of the equipped vehicle to generate a birds-eye view of the environment at least partially surrounding the equipped vehicle for display by a display screen of the equipped vehicle that is viewable by the driver of the equipped vehicle, and wherein said electronic control unit is operable to stitch together image data captured at least by said first and third cameras to form a substantially seamless birds-eye view for display by said display screen of the equipped vehicle for viewing by the driver of the equipped vehicle when operating the equipped vehicle.

26. The vision system of claim 25, wherein the birds-eye view displayed by said display screen includes a representation of the equipped vehicle, and wherein said third camera is disposed at a passenger side exterior rearview mirror assembly of the equipped vehicle.

27. The vision system of claim 26, wherein a kinematic model comprising a bicycle model is used by said electronic control unit to represent vehicle motion and wherein said kinematic model utilizes at least one of (i) a wheel steering angle of the equipped vehicle, (ii) a wheel pulse of the equipped vehicle, (iii) a heading angle of the equipped vehicle and (iv) translational vehicle motion of the equipped vehicle.