Vehicle vision system

• US 10,286,843 B2
• Filed: 08/21/2017
• Issued: 05/14/2019
• Est. Priority Date: 05/10/2013
• Status: Active Grant

• Current Assignee: Magna Electronics Incorporated (Magna International Incorporated)
• Original Assignee: Magna Electronics Incorporated (Magna International Incorporated)
• Inventors: Gieseke, Achim, Biemer, Michael
• Primary Examiner(s): Beaulieu, Yonel
• Application Number:
  US15/681,700
• Publication Number:
  US 20170368994A1
• Time in Patent Office: 631 Days
• CPC Class Codes: B60R 2300/105 using multiple cameras
• CPC Class Codes: B60R 2300/205 using a head-up display
• CPC Class Codes: B60R 2300/802 for monitoring and displaying vehicle exterior blind spot views
• CPC Class Codes: B60R 2300/806 for aiding parking
• CPC Class Codes: G01C 21/365 Guidance using head up displays or projectors, e.g. virtual vehicles or arrows projected on the windscreen or on the road itself
• CPC Class Codes: G01C 21/3679 Retrieval, searching and output of POI information, e.g. hotels, restaurants, shops, filling stations, parking facilities G01C21/3611 takes precedence
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06V 20/597 Recognising the driver's state or behaviour, e.g. attention or drowsiness

Abstract

A vehicular vision system includes a monitoring system operable to determine a gaze direction of a driver of the vehicle. At least one vision sensor is disposed at the vehicle and has a field of view exterior of the equipped vehicle, and at least one non-vision sensor is disposed at the vehicle and has a field of sensing exterior of the vehicle. Image data captured by the interior viewing camera, image data captured by the exterior viewing camera and sensor data sensed by the non-vision sensor are provided to a control of the vehicle. A display system includes a video screen operable to display video images for viewing by the driver of the vehicle. The video screen also displays an augmented reality overlay. Responsive at least in part to processing of captured image data and/or sensed sensor data, the vision system determines the augmented reality overlay displayed.

Claims

1. A vehicular vision system, said vision system comprising:

a monitoring system operable to determine a gaze direction of a driver of a vehicle equipped with said vision system;

wherein said monitoring system comprises an interior viewing camera disposed in an interior cabin of the equipped vehicle and having a field of view interior of the equipped vehicle that encompasses the head of the driver of the equipped vehicle;

wherein, responsive to processing of image data captured by said interior viewing camera, said monitoring system determines a gaze direction of the driver of the equipped vehicle;

at least one vision sensor disposed at the equipped vehicle, said at least one vision sensor comprising an exterior viewing camera having a field of view exterior of the equipped vehicle;

wherein said exterior viewing camera captures image data within its field of view;

at least one non-vision sensor disposed at the equipped vehicle and having a field of sensing exterior of the equipped vehicle;

wherein said non-vision sensor senses sensor data within its field of sensing;

wherein said non-vision sensor is selected from the group consisting of (i) a radar sensor, (ii) a lidar sensor and (iii) an ultrasonic sensor;

wherein image data captured by said interior viewing camera, image data captured by said exterior viewing camera and sensor data sensed by said non-vision sensor are provided to a control of the equipped vehicle;

wherein said control comprises circuitry, said circuitry comprising at least one processor operable to process data;

a display system of the equipped vehicle, said display system comprising a video display screen operable to display video images for viewing by the driver of the equipped vehicle;

wherein, responsive at least in part to processing by at least one processor of said control of image data captured by said exterior viewing camera, video images are displayed by said video display screen;

wherein said video screen also displays an augmented reality overlay; and

wherein, responsive at least in part to processing by at least one processor of said control of at least one of (a) image data captured by said exterior viewing camera and (b) sensor data sensed by said non-vision sensor, said vision system determines the augmented reality overlay displayed.

2. The vision system of claim 1, wherein the displayed video images are dependent on the determined gaze direction of the driver of the equipped vehicle.

3. The vision system of claim 2, wherein the displayed augmented reality overlay is dependent on the determined gaze direction of the driver of the equipped vehicle.

4. The vision system of claim 1, wherein the displayed augmented reality overlay pertains to at least one accessory of the equipped vehicle.

5. The vision system of claim 4, wherein the displayed augmented reality overlay is responsive at least in part to the determined gaze direction of the driver of the equipped vehicle.

6. The vision system of claim 5, wherein the augmented reality overlay displayed is responsive at least in part to a driving condition of the equipped vehicle.

7. The vision system of claim 6, wherein the driving condition comprises a parking maneuver of the equipped vehicle.

8. The vision system of claim 6, wherein the driving condition comprises a forward driving condition of the equipped vehicle.

9. The vision system of claim 8, wherein, during the forward driving condition, and responsive to determination that the driver'"'"'s gaze direction is towards the engine compartment of the equipped vehicle, the augmented reality display comprises information pertaining to at least one of (i) engine operating RPM, (ii) oil pressure, (iii) oil level, (iv) coolant temperature and (v) tire pressure.

10. The vision system of claim 1, further comprising a gesture control system operable to determine a gesture of the driver, and wherein said vision system displays video images at said video display screen responsive to the determined gesture.

11. The vision system of claim 1, further comprising a gesture control system operable to determine a gesture of the driver, and wherein said control controls at least one accessory of the equipped vehicle responsive to the determined gesture.

12. The vision system of claim 1, further comprising a gesture control system operable to determine a gesture of the driver, and wherein at least one of (i) pitch of said exterior viewing camera is controlled responsive to the determined gesture, (ii) roll of said exterior viewing camera is controlled responsive to the determined gesture and (iii) yaw of said exterior viewing camera is controlled responsive to the determined gesture.

13. The vision system of claim 1, wherein said non-vision sensor comprises a radar sensor, and wherein said radar sensor senses radar data within its field of sensing.

14. A vehicular vision system, said vision system comprising:

a monitoring system operable to determine a gaze direction of a driver of a vehicle equipped with said vision system;

wherein said monitoring system comprises an interior viewing camera disposed in an interior cabin of the equipped vehicle and having a field of view interior of the equipped vehicle that encompasses the head of the driver of the equipped vehicle;

wherein, responsive to processing of image data captured by said interior viewing camera, said monitoring system determines a gaze direction of the driver of the equipped vehicle;

at least one vision sensor disposed at the equipped vehicle, said at least one vision sensor comprising an exterior viewing camera having a field of view exterior of the equipped vehicle;

wherein said exterior viewing camera captures image data within its field of view;

at least one non-vision sensor disposed at the equipped vehicle and having a field of sensing exterior of the equipped vehicle;

wherein said non-vision sensor senses sensor data within its field of sensing;

wherein said non-vision sensor comprises one of (i) a radar sensor and (ii) a lidar sensor;

wherein image data captured by said interior viewing camera, image data captured by said exterior viewing camera and sensor data sensed by said non-vision sensor are provided to a control of the equipped vehicle;

wherein said control comprises circuitry, said circuitry comprising at least one processor operable to process data;

a display system of the equipped vehicle, said display system comprising a video display screen operable to display video images for viewing by the driver of the equipped vehicle;

wherein, responsive at least in part to processing by at least one processor of said control of image data captured by said exterior viewing camera, video images are displayed by said video display screen;

wherein said video screen also displays an augmented reality overlay;

wherein, responsive at least in part to processing by at least one processor of said control of at least one of (a) image data captured by said exterior viewing camera and (b) sensor data sensed by said non-vision sensor, said vision system determines the augmented reality overlay displayed; and

wherein the displayed augmented reality overlay is responsive at least in part to the determined gaze direction of the driver of the equipped vehicle and to a driving condition of the equipped vehicle.

15. The vision system of claim 14, further comprising a gesture control system operable to determine a gesture of the driver.

16. The vision system of claim 15, wherein said gesture control system controls at least one accessory of the equipped vehicle responsive to the determined gesture.

17. The vision system of claim 16, wherein said gesture control system is operable to determine movement of the driver'"'"'s hand, and wherein, responsive to the determined movement, a vehicle accessory is controlled.

18. A vehicular vision system, said vision system comprising:

a monitoring system operable to determine a gaze direction of a driver of a vehicle equipped with said vision system;

wherein said monitoring system comprises an interior viewing camera disposed in an interior cabin of the equipped vehicle and having a field of view interior of the equipped vehicle that encompasses the head of the driver of the equipped vehicle;

wherein, responsive to processing of image data captured by said interior viewing camera, said monitoring system determines a gaze direction of the driver of the equipped vehicle;

at least one vision sensor disposed at the equipped vehicle, said at least one vision sensor comprising an exterior viewing camera having a field of view exterior of the equipped vehicle;

wherein said exterior viewing camera captures image data within its field of view;

at least one non-vision sensor disposed at the equipped vehicle and having a field of sensing exterior of the equipped vehicle;

wherein said non-vision sensor senses sensor data within its field of sensing;

wherein said non-vision sensor comprises one of (i) a radar sensor and (ii) a lidar sensor;

wherein image data captured by said interior viewing camera, image data captured by said exterior viewing camera and sensor data sensed by said non-vision sensor are provided to a control of the equipped vehicle;

wherein said control comprises circuitry, said circuitry comprising at least one processor operable to process data;

a display system of the equipped vehicle, said display system comprising a video display screen operable to display video images for viewing by the driver of the equipped vehicle;

wherein, responsive at least in part to processing by at least one processor of said control of image data captured by said exterior viewing camera, video images are displayed by said video display screen;

wherein said video screen also displays an augmented reality overlay;

wherein, responsive at least in part to processing by at least one processor of said control of at least one of (a) image data captured by said exterior viewing camera and (b) sensor data sensed by said non-vision sensor, said vision system determines the augmented reality overlay displayed; and

a gesture control system operable to determine a gesture of the driver, and wherein at least one of (i) pitch of said exterior viewing camera is controlled responsive to the determined gesture, (ii) roll of said exterior viewing camera is controlled responsive to the determined gesture and (iii) yaw of said exterior viewing camera is controlled responsive to the determined gesture.

19. The vision system of claim 18, wherein said gesture control system is operable to determine movement of the driver'"'"'s hand, and wherein, responsive to the determined movement, a vehicle accessory is controlled.

20. The vision system of claim 19, wherein the displayed augmented reality overlay is responsive at least in part to the determined gaze direction of the driver of the equipped vehicle and to a driving condition of the equipped vehicle.