VEHICULAR CAMERA WITH ON-BOARD MICROCONTROLLER

• US 20120265416A1
• Filed: 07/27/2010
• Published: 10/18/2012
• Est. Priority Date: 07/27/2009
• Status: Active Grant

• Current Assignee: Magna Electronics Incorporated (Magna International Incorporated)
• Original Assignee: Magna Electronics Incorporated (Magna International Incorporated)
• Inventors: Lu, Yuesheng, Gibson, Joel S., Gebauer, Duane W., Shriner, Richard D., Miller, Patrick A.
• Granted Patent: US 9,495,876 B2
• Time in Patent Office: Days
• US Class Current: 701/70
• CPC Class Codes: B60R 1/00 Optical viewing arrangements; Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles
• CPC Class Codes: B60R 2300/105 using multiple cameras
• CPC Class Codes: B60R 2300/305 merging camera image with lines or icons
• CPC Class Codes: B60R 2300/806 for aiding parking
• CPC Class Codes: B60W 2420/403 Image sensing, e.g. optical camera
• CPC Class Codes: B60W 30/09 Taking automatic action to avoid collision, e.g. braking and steering
• CPC Class Codes: B62D 15/0295 by overlaying a vehicle path based on present steering angle over an image without processing that image
• CPC Class Codes: G08G 1/168 Driving aids for parking, e.g. acoustic or visual feedback on parking space
• CPC Class Codes: G09G 2340/125 wherein one of the images is motion video
• CPC Class Codes: G09G 2380/10 Automotive applications
• CPC Class Codes: G09G 5/14 Display of multiple viewports
• CPC Class Codes: H04N 23/60 Control of cameras or camera modules
• CPC Class Codes: H04N 23/63 by using electronic viewfinders
• CPC Class Codes: H04N 25/61 the noise originating only from the lens unit, e.g. flare, shading, vignetting or "cos4"

Abstract

In one aspect of the invention, a vehicular camera is provided, comprising a lens, a housing, an imager and a microcontroller that is capable of handling certain functions, such as applying overlays to images received by the imager, dewarping the image and/or providing different viewing modes.

Claims

1. A vehicular camera, comprising:

a lens;

an image processor configured to receive images from the lens; and

a microcontroller containing flash memory, wherein the microcontroller is connected to the image processor by a first bus through which command data is communicated to the image processor from the microcontroller, wherein the command data includes application instructions to draw application data from a selected point in the flash memory, and wherein the microcontroller is connected to the image processor by a second bus through which application data is communicated to the image processor from the flash memory.

2. A vehicular camera as claimed in claim 1, wherein the first bus is an inter-integrated circuit bus, and the second bus is a serial peripheral interface bus.

3. A vehicular camera as claimed in claim 1, wherein the application data includes data relating to an overlay, and wherein the microcontroller is programmed to send position data to the image processor relating to a position at which to apply the overlay to one or more of the images.

4. A vehicular camera as claimed in claim 1, wherein the microcontroller is programmed to calculate at least one of the size and shape of an overlay and to transmit data relating to the overlay to the image processor for the image processor to apply to one or more of the images.

5. A vehicular camera, comprising:

a lens having a field of view;

an image sensor;

a housing, wherein the lens and image sensor are mounted to the housing at a selected position relative to each other, wherein the image sensor receives raw images from the lens; and

a controller, wherein for at least one raw image the controller is programmed to generate a dewarped image by stretching and compressing portions of the raw image so as to provide in the dewarped image a selected shape for a selected known element in the field of view of the lens.

6. A vehicular camera as claimed in claim 5, wherein the lens has an optical axis,

wherein the selected known element is a horizon line,

wherein the vehicular camera is mounted to a vehicle in such a way that the horizon line in the field of view of the lens approximately intersects the optical axis of the lens,

and wherein the portions of the raw image stretched and compressed are vertically extending portions of the raw image above and below the horizon line in the raw image to at least partially straighten out the horizon line appearing in the dewarped image.

7. A vehicular camera as claimed in claim 5, wherein the selected known element is an edge of a bumper on the vehicle.

8. A vehicular camera as claimed in claim 5, wherein the lens has approximately a 180 degree field of view horizontally.

9. A vehicular camera as claimed in claim 5, wherein the lens has approximately a 135 degree field of view horizontally.

10. A vehicular camera as claimed in claim 6, wherein the lens has approximately a 180 degree field of view horizontally, and has a first viewing mode and a second viewing mode,

wherein in the first viewing mode dewarped images are sent to a display in the vehicle showing approximately 180 degrees of the field of view horizontally, and the controller is programmed to stretch and compress portions of the raw image using first values to provide the selected shape for the selected known element in the dewarped image,

and wherein in the second viewing mode images are sent to a display in the vehicle showing less of the field of view horizontally than in the first viewing mode, and the controller is programmed to stretch and compress portions of the raw image using second values to provide the selected shape for the selected known element in the dewarped image.

11. A vehicular camera as claimed in claim 10, wherein in the second viewing mode dewarped images are sent to a display in the vehicle showing approximately 135 degrees of the field of view horizontally.

12. A vehicular camera as claimed in claim 5, wherein the image sensor is part of an imager, and the controller is made up of a portion of the imager and a microcontroller that is external to the imager.

13. A camera system for a vehicle, comprising:

a camera;

a distance sensor; and

a controller that determines whether there is an element in the fields of view of the camera and the distance sensor that represents a collision risk for the vehicle based on an image from the camera and input from the distance sensor, wherein upon detection of an element that represents a collision risk the controller is programmed to carry out at least one action consisting of;

informing the driver of the vehicle of the collision risk; and

control at least one vehicle component to inhibit a collision by the vehicle with the element that represents a collision risk.

14. A camera system as claimed in claim 13, wherein a display is provided in an interior of the vehicle, and wherein the controller is programmed to send to the display an image from the camera, wherein the image shows the element that represents a collision risk, and the image includes an overlay that highlights the element that represents a collision risk.

15. A camera system as claimed in claim 13, wherein upon determination that the element represents a collision risk the controller is programmed to apply a brake on the vehicle.

16. A camera system as claimed in claim 14, wherein the controller receives a depth map from the distance sensor and is configured to determine the distance to the element that represents a collision risk based at least partially on the depth map.

17. A camera system as claimed in claim 16, wherein prior to determining whether the element represents a collision risk, the controller is programmed to determine whether the element falls within a category of elements that do not represent a collision risk based on the image from the camera.

18. A camera system as claimed in claim 16, wherein the controller receives the image from the camera and wherein the distance sensor is configured to detect light at a selected wavelength and the controller receives an image from the distance sensor and is configured to determine the distance to the element that represents a collision risk partially based on differences in the position of the element in the image from the camera and the position of the element in the image from the distance sensor.

19. A camera system as claimed in claim 18, wherein the distance sensor is configured to detect infrared light and wherein a plurality of infrared light sources are positioned proximate the distance sensor.

20. A camera system as claimed in claim 18, wherein the image from the camera has a first resolution and the image from the distance sensor has a second resolution that is lower than the first resolution.

21. A camera system as claimed in claim 16, wherein prior to determining whether the element represents a collision risk, the controller is programmed to determine whether the element falls within a category of elements that represent objects of interest based on the image from the camera.

22. A camera system as claimed in claim 21, wherein the objects of interest include at least one of people, animals and other vehicles.

23. A camera system for a vehicle, comprising:

a plurality of cameras, each camera having a field of view and being positioned at selected positions about the vehicle, wherein the plurality of cameras has a combined field of view, wherein each camera includes;

a lens;

an image sensor;

a housing, wherein the lens and image sensor are mounted to the housing at a selected position relative to each other, wherein the image sensor receives raw images from the lens; and

a controller programmed to modify at least one raw image to produce a processed image;

a central controller programmed to combine the processed images from the cameras into a single combined image showing the combined field of view; and

an electrical cable connecting each camera to the central controller, wherein the electrical cable is a non-coaxial cable.

24. A camera system as claimed in claim 23, wherein the plurality of cameras is four cameras and wherein the combined field of view is substantially 360 degrees about the vehicle.

25. A camera system as claimed in claim 23, wherein the controller is programmed to dewarp the raw image to produce the processed image.

26. A camera system as claimed in claim 23, wherein the central controller is programmed to apply an overlay to the single combined image to show the vehicle.

27. A camera system as claimed in claim 23, wherein each camera is positioned at an actual viewing angle has a bird'"'"'s eye viewing mode in which the controller is programmed to modify the raw image so that the processed image appears to have been taken at an apparent viewing angle that is more vertically oriented than the actual viewing angle.

28. A camera system as claimed in claim 27, wherein the controller is programmed to compress a lower portion of the raw image and stretch an upper portion of the raw image so that the apparent viewing angle is more vertically oriented than the actual viewing angle.

29. A camera for a vehicle having a towing device, comprising:

a lens having a field of view;

an image sensor;

a housing, wherein the lens and image sensor are mounted to the housing at a selected position relative to each other, wherein the image sensor receives raw images from the lens; and

a controller programmed to modify at least one raw image to produce a processed image,

wherein the camera is positioned at an actual viewing angle such that the towing device is in the field of view of the lens, and wherein the camera has a bird'"'"'s eye viewing mode in which the controller is programmed to modify the raw image so that the processed image appears to have been taken at an apparent viewing angle that is more vertically oriented than the actual viewing angle.

30. A camera as claimed in claim 29, wherein the controller is programmed to compress a lower portion of the taw image and stretch an upper portion of the raw image so that the apparent viewing angle is more vertically oriented than the actual viewing angle.

31. A camera as claimed in claim 29, wherein the camera has a hitch viewing mode in which the processed image appears to have been taken at an apparent viewing angle that is more vertically oriented than the actual viewing angle, and in which the towing device in the processed image is magnified to a magnification level that is greater than a magnification level provided in the bird'"'"'s eye viewing mode.

32. A camera as claimed in claim 29, wherein the controller is programmed to determine a projected trajectory for the vehicle and is programmed to apply an overlay to the processed image containing a representation of the projected trajectory.

33. A camera as claimed in claim 32, wherein the controller is programmed to apply at least one indicator of distance from the towing device to the image.

34. A camera for a vehicle, comprising:

a lens having a field of view;

an image sensor;

a housing, wherein the lens and image sensor are mounted to the housing at a selected position relative to each other, wherein the imager receives raw images from the lens; and

a controller programmed to determine the position of a selected feature in the field of view of the lens and to apply an overlay on the selected feature in an initial raw image to produce an initial processed image and to thereafter hold the overlay in a fixed position in subsequent processed images regardless of movement of the selected feature in subsequent raw images.

35. A camera as claimed in claim 34, wherein the selected feature is a feature on a trailer behind the vehicle.

36. A camera as claimed in claim 34, wherein the image sensor is part of an imager and the controller is made up of a portion of the imager and a microcontroller that is external to the imager, and wherein the portion of the imager is programmed to determine the position of the selected feature in the field of view of the lens, and wherein the microcontroller is programmed to receive from the imager a position indicator indicating the position of the selected feature, wherein the microcontroller and the portion of the imager are together programmed to apply the overlay on the selected feature in the initial raw image to produce the initial processed image and to thereafter hold the overlay in the fixed position in subsequent processed images regardless of movement of the selected feature in subsequent raw images.