Vehicular video processor module

• US 8,405,725 B2
• Filed: 02/18/2010
• Issued: 03/26/2013
• Est. Priority Date: 07/31/2001
• Status: Expired due to Term

• Current Assignee: Magna Electronics Incorporated (Magna International Incorporated)
• Original Assignee: Donnelly Corporation (Magna International Incorporated)
• Inventors: McMahon, Martha A., Cooprider, Troy O.
• Primary Examiner(s): SHEPARD, JUSTIN E
• Application Number:
  US12/708,079
• Publication Number:
  US 20100188508A1
• Time in Patent Office: 1,132 Days
• Field of Search: 348/148
• US Class Current: 348/148
• CPC Class Codes: B60K 35/00 Instruments specially adapted for vehicles; Arrangement of instruments in or on vehicles
• CPC Class Codes: B60Q 1/08 automatically
• CPC Class Codes: B60Q 9/00 Arrangement or adaptation of signal devices not provided for in one of main groups B60Q1/00 - B60Q7/00 , e.g. haptic signalling
• CPC Class Codes: B60Q 9/008 for anti-collision purposes
• CPC Class Codes: B60R 1/27 providing all-round vision, e.g. using omnidirectional cameras
• CPC Class Codes: B60R 16/023 for transmission of signals between vehicle parts or subsystems
• CPC Class Codes: B60R 2300/105 using multiple cameras
• CPC Class Codes: B60R 2300/303 using joined images, e.g. multiple camera images
• CPC Class Codes: B60R 2300/304 using merged images, e.g. merging camera image with stored images
• CPC Class Codes: B60W 30/143 Speed control B60W30/16 takes precedence
• CPC Class Codes: G01S 13/867 Combination of radar systems with cameras
• CPC Class Codes: G05D 1/00 Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots drive control systems specially adapted for autonomous road vehicles B60W60/00
• CPC Class Codes: G05D 1/0033 by having the operator tracking the vehicle either by direct line of sight or via one or more cameras located remotely from the vehicle
• CPC Class Codes: G05D 1/0038 by providing the operator with simple or augmented images from one or more cameras located onboard the vehicle, e.g. tele-operation images analyzed by a computer and used for automatic navigation G05D1/0246
• CPC Class Codes: G05D 1/0044 by providing the operator with a computer generated representation of the environment of the vehicle, e.g. virtual reality, maps maps used for automatic navigation G05D1/0274; flight directors G01C23/005
• CPC Class Codes: H04N 21/422 Input-only peripherals , i.e. input devices connected to specially adapted client devices, e.g. global positioning system [GPS] input devices also receiving signals from specially adapted client devices H04N21/4104
• CPC Class Codes: H04N 21/42202 environmental sensors, e.g. for detecting temperature, luminosity, pressure, earthquakes
• CPC Class Codes: H04N 21/4223 Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 25/76 Addressed sensors, e.g. MOS or CMOS sensors
• CPC Class Codes: H04N 7/181 : for receiving images from a plurality of remote sources

Abstract

A video processor module suitable for use in a vehicle includes a video processor and a plurality of imaging sensor interfaces having at least a first imaging sensor interface configured for communication with a first imaging sensor of the equipped vehicle and a second imaging sensor interface configured for communication with a second imaging sensor of the equipped vehicle, with the first and second imaging sensor interfaces receiving image data captured by the first and second imaging sensors of the equipped vehicle, respectively. The video processor may include an object detection section, a camera selection section, a camera control section and/or a memory section. The video processor module may include a vehicle interface section.

Claims

1. A video processor module suitable for use in a vehicle, said video processor module comprising:

a video processor;

a plurality of imaging sensor interfaces, said plurality of imaging sensor interfaces comprising at least (a) a first imaging sensor interface configured for communication with a first imaging sensor of a vehicle equipped with said video processor module and (b) a second imaging sensor interface configured for communication with a second imaging sensor of the equipped vehicle, and wherein said first and second imaging sensor interfaces receive image data captured by the first and second imaging sensors of the equipped vehicle, respectively;

wherein said first imaging sensor interface is configured for communication with one of (a) a forward-facing imaging sensor of the equipped vehicle, (b) a rearward-facing imaging sensor of the equipped vehicle, (c) a left-side-facing imaging sensor of the equipped vehicle, (d) a right-side-facing imaging sensor of the equipped vehicle and (e) an inward-cabin-facing imaging sensor of the equipped vehicle, and wherein said second imaging sensor interface is configured for communication with another of (a) a forward-facing imaging sensor of the equipped vehicle, (b) a rearward-facing imaging sensor of the equipped vehicle, (c) a left-side-facing imaging sensor of the equipped vehicle, (d) a right-side-facing imaging sensor of the equipped vehicle and (e) an inward-cabin-facing imaging sensor of the equipped vehicle;

wherein said video processor comprises an object detection section;

wherein said video processor comprises a camera selection section;

wherein said video processor comprises a camera control section;

wherein said video processor module comprises a vehicle interface section;

wherein said video processor comprises a memory section;

wherein said camera control section integrates data by fusing data received by said first and second imaging sensor interfaces from the first and second imaging sensors of the equipped vehicle, respectively, and data received by at least one non-imaging sensor interface of said video processor module from at least one non-imaging sensor of the equipped vehicle; and

wherein said camera control section integrates said data by fusing said data into a single database in a standard format.

2. The video processor module of claim 1, further comprising at least one non-imaging sensor interface configured for communication with at least one non-imaging sensor of the equipped vehicle.

3. The video processor module of claim 2, wherein said video processor comprises a core that is operable to exchange data with said at least one non-imaging sensor interface via a communication bus of the equipped vehicle.

4. The video processor module of claim 3, wherein said core is operable to exchange said data via at least one of a CAN bus of the equipped vehicle and a LIN bus of the equipped vehicle.

5. A video processor module suitable for use in a vehicle, said video processor module comprising:

a video processor;

a plurality of imaging sensor interfaces, said plurality of imaging sensor interfaces comprising at least (a) a first imaging sensor interface configured for communication with a first imaging sensor of a vehicle equipped with said video processor module and (b) a second imaging sensor interface configured for communication with a second imaging sensor of the equipped vehicle, and wherein said first and second imaging sensor interfaces receive image data captured by the first and second imaging sensors of the equipped vehicle, respectively;

wherein said first imaging sensor interface is configured for communication with one of (a) a forward-facing imaging sensor of the equipped vehicle, (b) a rearward-facing imaging sensor of the equipped vehicle, (c) a left-side-facing imaging sensor of the equipped vehicle, (d) a right-side-facing imaging sensor of the equipped vehicle and (e) an inward-cabin-facing imaging sensor of the equipped vehicle, and wherein said second imaging sensor interface is configured for communication with another of (a) a forward-facing imaging sensor of the equipped vehicle, (b) a rearward-facing imaging sensor of the equipped vehicle, (c) a left-side-facing imaging sensor of the equipped vehicle, (d) a right-side-facing imaging sensor of the equipped vehicle and (e) an inward-cabin-facing imaging sensor of the equipped vehicle;

wherein said video processor comprises an object detection section;

wherein said video processor comprises a camera selection section;

wherein said video processor comprises a camera control section;

wherein said video processor module comprises a vehicle interface section;

wherein said video processor comprises a memory section;

at least one non-imaging sensor interface configured for communication with at least one non-imaging sensor of the equipped vehicle; and

wherein said video processor is operable to fuse data from at least two of (a) said first imaging sensor interface, (b) said second imaging sensor interface and (c) said at least one non-imaging sensor interface, and wherein said video processor is operable to fuse said data into a single database in a standard format, and wherein said video processor provides at least one output signal in response to processing said fused data of said single database, said at least one output signal comprising at least an output signal for conveying information to the driver of the equipped vehicle that is used by the driver of the equipped vehicle when the driver is normally operating the equipped vehicle.

6. The video processor module of claim 5, wherein said video processor comprises a video processor chip comprising at least one of (a) said object detection section, (b) said camera selection section and (c) said camera control section.

7. The video processor module of claim 5, wherein said object detection section is operable to detect objects in captured image data received by at least one of said first and second imaging sensor interfaces from the first and second imaging sensors, respectively.

8. The video processor module of claim 5, wherein said vehicle interface section interfaces with a communication bus of the equipped vehicle.

9. The video processor module of claim 8, wherein said vehicle interface section interfaces with at least one of a CAN bus of the equipped vehicle and a LIN bus of the equipped vehicle.

10. The video processor module of claim 5, wherein said object detection section comprises tracking of at least one object in captured images received by at least one of said first and second imaging sensor interfaces from the first and second imaging sensors, respectively.

11. The video processor module of claim 5, wherein said object detection section comprises classification of at least one object in at least one captured image received by at least one of said first and second imaging sensor interfaces from the first and second imaging sensors, respectively.

12. The video processor module of claim 11, wherein an object imaged by at least one of the first and second imaging sensors of the equipped vehicle is at least partially identified by said object classification via processing of said image data received by at least one of said first and second imaging sensor interfaces.

13. The video processor module of claim 12, wherein said object detection section classifies said at least partially identified object.

14. The video processor module of claim 13, wherein said object detection section comprises object tracking for tracking an identified object and wherein said object detection section classifies a tracked object.

15. The video processor module of claim 5, wherein said camera control section is operable to control, via at least one of said first and second imaging sensor interfaces, at least one of the first and second imaging sensors of the equipped vehicle.

16. The video processor module of claim 15, wherein said camera control section is operable to integrate image data received by said first and second imaging sensor interfaces from the first and second imaging sensors of the equipped vehicle, respectively.

17. The video processor module of claim 5, wherein said camera control section integrates data by fusing data received by said first and second imaging sensor interfaces from the first and second imaging sensors of the equipped vehicle, respectively, and data received by at least one non-imaging sensor interface of said video processor module from at least one non-imaging sensor of the equipped vehicle.

18. The video processor module of claim 5, wherein said video processor comprises a diagnostic section for performing diagnostics.

19. The video processor module of claim 5, wherein said video processor comprises a display section.

20. The video processor module of claim 19, wherein said display section is operable to generate data communicated through said vehicle interface section for use by a video display screen of the equipped vehicle.

21. The video processor module of claim 20, wherein said video processor is operable to merge image data received by said first and second imaging sensor interfaces from the first and second imaging sensors of the equipped vehicle, respectively, to generate for display on the video display screen of the equipped vehicle a video image for viewing by the driver of the equipped vehicle that exceeds the field of view of a single one of the first and second imaging sensors of the equipped vehicle.

22. The video processor module of claim 21, wherein said video processor is operable to merge image data received by said first and second imaging sensor interfaces from the first and second imaging sensors of the equipped vehicle, respectively, to generate for display on the video display screen of the equipped vehicle a panoramic video image for viewing by the driver of the equipped vehicle.

23. The video processor module of claim 20, wherein said video processor comprises a graphics section that generates graphics communicated through said vehicle interface section for displaying on the video display screen of the equipped vehicle for viewing and use by a driver of the equipped vehicle when the driver is normally operating the equipped vehicle.

24. The video processor module of claim 23, wherein said graphics section of said video processor is operable to generate a graphical overlay communicated through said vehicle interface section for display by the video display screen of the equipped vehicle.

25. The video processor module of claim 5, wherein said video processor is operable to electronically reduce image distortion in images captured by the first imaging sensor of the equipped vehicle by processing captured image data received by said first imaging sensor interface.

26. The video processor module of claim 5, wherein said video processor is operable to enhance data received by said first and second imaging sensor interfaces from the first and second imaging sensors of the equipped vehicle, respectively.

27. The video processor module of claim 5, further comprising image stabilization operable to process image data received by said first and second imaging sensor interfaces from the first and second imaging sensors of the equipped vehicle, respectively.

28. The video processor module of claim 5, wherein said video processor comprises a core that is operable to exchange data with said first and second imaging sensor interfaces.

29. The video processor module of claim 5, wherein said first imaging sensor interface is configured for communication with a forward-facing imaging sensor mounted at an interior mirror assembly of the equipped vehicle, and wherein at least one of (a) the forward-facing imaging sensor provides, at least in part, a headlamp control function, and (b) the forward-facing imaging sensor provides, at least in part, a windshield wiper control function.

30. The video processor module of claim 5, wherein said first and second imaging sensor interfaces comprise at least one of a firewire transceiver and an NTSC transceiver.

31. The video processor module of claim 5, wherein said first and second imaging sensor interfaces include at least one wireless interface operable to communicate with at least one of (a) a restricted-range radio-frequency communication device housed at an interior mirror of the equipped vehicle, (b) a restricted-range radio-frequency communication device housed at an overhead console of the equipped vehicle, (c) a restricted-range radio-frequency communication device housed at an instrument panel of the equipped vehicle, (d) a restricted-range radio-frequency communication device operable to program radio-frequency devices within the equipped vehicle, (e) a restricted-range radio-frequency communication device operable to receive user selection of vehicle parameters including vehicle parameters chosen from a garage door opener code, memory seat position, memory mirror position and exterior minor position, and (f) a restricted-range radio-frequency communication device comprising a personal display assistant device.

32. The video processor module of claim 5, wherein said video processor provides at least one of (a) detection of other vehicles exterior the equipped vehicle, (b) detection of a lane on a road at or near the equipped vehicle, (c) an adaptive speed control of the equipped vehicle, (d) a reverse aid warning to the driver of the equipped vehicle during a reversing maneuver of the equipped vehicle, (e) ambient light sensing, (f) detection of other vehicles at or near or approaching the equipped vehicle, (g) control of a state of a headlamp of the equipped vehicle, (h) detection of at least one chosen from presence, position and size of a person.

33. The video processor module of claim 32, wherein said video processor provides detection of at least one chosen from presence, position and size of an occupant inside a cabin of the equipped vehicle.

34. A video processor module suitable for use in a vehicle, said video processor module comprising:

a video processor;

a plurality of imaging sensor interfaces, said plurality of imaging sensor interfaces comprising at least (a) a first imaging sensor interface configured for communication with a first imaging sensor of a vehicle equipped with said video processor module and (b) a second imaging sensor interface configured for communication with a second imaging sensor of the equipped vehicle, and wherein said first and second imaging sensor interfaces receive image data captured by the first and second imaging sensors of the equipped vehicle, respectively;

wherein said first imaging sensor interface is configured for communication with one of (a) a forward-facing imaging sensor of the equipped vehicle, (b) a rearward-facing imaging sensor of the equipped vehicle, (c) a left-side-facing imaging sensor of the equipped vehicle, (d) a right-side-facing imaging sensor of the equipped vehicle and (e) an inward-cabin-facing imaging sensor of the equipped vehicle, and wherein said second imaging sensor interface is configured for communication with another of (a) a forward-facing imaging sensor of the equipped vehicle, (b) a rearward-facing imaging sensor of the equipped vehicle, (c) a left-side-facing imaging sensor of the equipped vehicle, (d) a right-side-facing imaging sensor of the equipped vehicle and (e) an inward-cabin-facing imaging sensor of the equipped vehicle;

at least one non-imaging sensor interface configured for communication with at least one non-imaging sensor of the equipped vehicle;

wherein said video processor comprises an object detection section;

wherein said video processor comprises a camera selection section;

wherein said video processor comprises a camera control section;

wherein said video processor module comprises a vehicle interface section;

wherein said video processor comprises a memory section;

wherein said video processor provides at least one of (a) detection of other vehicles exterior the equipped vehicle, (b) detection of a lane on a road at or near the equipped vehicle, (c) an adaptive speed control of the equipped vehicle, (d) a reverse aid warning to the driver of the equipped vehicle during a reversing maneuver of the equipped vehicle, (e) ambient light sensing, (f) detection of other vehicles at or near or approaching the equipped vehicle, (g) control of a state of a headlamp of the equipped vehicle, (h) detection of at least one chosen from presence, position and size of a person; and

wherein said video processor is operable to fuse data from at least two of (a) said first imaging sensor interface, (b) said second imaging sensor interface and (c) said at least one non-imaging sensor interface, and wherein said video processor is operable to fuse said data into a single database in a standard format, and wherein said video processor provides at least one output signal in response to processing said fused data of said single database, said at least one output signal comprising at least an output signal for conveying information to the driver of the equipped vehicle that is used by the driver of the equipped vehicle when the driver is normally operating the equipped vehicle.

35. The video processor module of claim 34, wherein said video processor provides detection of at least one chosen from presence, position and size of an occupant inside a cabin of the equipped vehicle.

36. A video processor module suitable for use in a vehicle, said video processor module comprising:

a video processor;

a plurality of imaging sensor interfaces, said plurality of imaging sensor interfaces comprising at least (a) a first imaging sensor interface configured for communication with a first imaging sensor of a vehicle equipped with said video processor module and (b) a second imaging sensor interface configured for communication with a second imaging sensor of the equipped vehicle, and wherein said first and second imaging sensor interfaces receive image data captured by the first and second imaging sensors of the equipped vehicle, respectively;

wherein said first imaging sensor interface is configured for communication with one of (a) a forward-facing imaging sensor of the equipped vehicle, (b) a rearward-facing imaging sensor of the equipped vehicle, (c) a left-side-facing imaging sensor of the equipped vehicle, (d) a right-side-facing imaging sensor of the equipped vehicle and (e) an inward-cabin-facing imaging sensor of the equipped vehicle, and wherein said second imaging sensor interface is configured for communication with another of (a) a forward-facing imaging sensor of the equipped vehicle, (b) a rearward-facing imaging sensor of the equipped vehicle, (c) a left-side-facing imaging sensor of the equipped vehicle, (d) a right-side-facing imaging sensor of the equipped vehicle and (e) an inward-cabin-facing imaging sensor of the equipped vehicle;

wherein said video processor comprises an object detection section;

wherein said video processor comprises a camera selection section;

wherein said video processor comprises a camera control section;

wherein said video processor module comprises a vehicle interface section;

wherein said video processor comprises a memory section;

wherein said video processor comprises a graphics section that generates graphics; and

wherein said video processor is operable to electronically reduce image distortion in images captured by the first imaging sensor of the equipped vehicle by processing captured image data received by said first imaging sensor interface;

at least one non-imaging sensor interface configured for communication with at least one non-imaging sensor of the equipped vehicle; and

wherein said video processor is operable to fuse data from at least two of (a) said first sensor interface, (b) said second imaging sensor interface and (c) said at least one non-imaging sensor interface, and wherein said video processor is operable to fuse said data into a single database in a standard format, and wherein said video processor provides at least one output signal in response to processing said fused data of said single database, said at least one output signal comprising at least an output signal for conveying information to the driver of the equipped vehicle that is used by the driver of the equipped vehicle when the driver is normally operating the equipped vehicle.

37. The video processor module of claim 36, wherein said graphics section generates graphics communicated through said vehicle interface section for displaying on the video display screen of the equipped vehicle for viewing and use by a driver of the equipped vehicle when the driver is normally operating the equipped vehicle.

38. The video processor module of claim 36, wherein said graphics section of said video processor is operable to generate a graphical overlay communicated through said vehicle interface section for display by the video display screen of the equipped vehicle.

39. A video processor module suitable for use in a vehicle, said video processor module comprising:

a video processor;

a plurality of imaging sensor interfaces, said plurality of imaging sensor interfaces comprising at least (a) a first imaging sensor interface configured for communication with a first imaging sensor of a vehicle equipped with said video processor module and (b) a second imaging sensor interface configured for communication with a second imaging sensor of the equipped vehicle, and wherein said first and second imaging sensor interfaces receive image data captured by the first and second imaging sensors of the equipped vehicle, respectively;

wherein said first imaging sensor interface is configured for communication with one of (a) a forward-facing imaging sensor of the equipped vehicle, (b) a rearward-facing imaging sensor of the equipped vehicle, (c) a left-side-facing imaging sensor of the equipped vehicle, (d) a right-side-facing imaging sensor of the equipped vehicle and (e) an inward-cabin-facing imaging sensor of the equipped vehicle, and wherein said second imaging sensor interface is configured for communication with another of (a) a forward-facing imaging sensor of the equipped vehicle, (b) a rearward-facing imaging sensor of the equipped vehicle, (c) a left-side-facing imaging sensor of the equipped vehicle, (d) a right-side-facing imaging sensor of the equipped vehicle and (e) an inward-cabin-facing imaging sensor of the equipped vehicle;

wherein said vehicle interface section interfaces with a communication bus of the equipped vehicle;

wherein said video processor comprises an object detection section, and wherein said object detection section is operable to detect objects in captured image data received by at least one of said first and second imaging sensor interfaces from the first and second imaging sensors, respectively;

wherein said video processor comprises a camera selection section;

wherein said video processor comprises a camera control section;

wherein said video processor module comprises a vehicle interface section;

wherein said video processor comprises a memory section;

at least one non-imaging sensor interface configured for communication with at least one non-imaging sensor of the equipped vehicle; and

wherein said video processor is operable to fuse data from at least two of (a) said first imaging sensor interface, (b) said second imaging sensor interface and (c) said at least one non-imaging sensor interface, and wherein said video processor is operable to fuse said data into a single database in a standard format, and wherein said video processor provides at least one output signal in response to processing said fused data of said single database, said at least one output signal comprising at least an output signal for conveying information to the driver of the equipped vehicle that is used by the driver of the equipped vehicle when the driver is normally operating the equipped vehicle.

40. The video processor module of claim 39, wherein said video processor comprises a video processor chip comprising at least one of (a) said object detection section, (b) said camera selection section and (c) said camera control section.

41. The video processor module of claim 39, wherein said object detection section comprises tracking of at least one object in captured images received by at least one of said first and second imaging sensor interfaces from the first and second imaging sensors, respectively.