Vehicular camera with on-board microcontroller

• US 9,495,876 B2
• Filed: 07/27/2010
• Issued: 11/15/2016
• Est. Priority Date: 07/27/2009
• Status: Active Grant

• Current Assignee: Magna Electronics Incorporated (Magna International Incorporated)
• Original Assignee: Magna Electronics Incorporated (Magna International Incorporated)
• Inventors: Lu, Yuesheng, Gibson, Joel S., Gebauer, Duane W., Shriner, Richard D., Miller, Patrick A.
• Primary Examiner(s): Troost, Aaron L
• Application Number:
  US13/384,673
• Publication Number:
  US 20120265416A1
• Time in Patent Office: 2,303 Days
• Field of Search: 701/70, 701/119, 701/36, 701/302, 340/425.5, 348/148, 348/222.1, 348/142, 382/106, 382/260, 382/203, 382/209, 382/218, 382/254, 382276-278, 280/477
• US Class Current: 1/1
• CPC Class Codes: B60R 1/00 Optical viewing arrangements; Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles
• CPC Class Codes: B60R 2300/105 using multiple cameras
• CPC Class Codes: B60R 2300/305 merging camera image with lines or icons
• CPC Class Codes: B60R 2300/806 for aiding parking
• CPC Class Codes: B60W 2420/403 Image sensing, e.g. optical camera
• CPC Class Codes: B60W 30/09 Taking automatic action to avoid collision, e.g. braking and steering
• CPC Class Codes: B62D 15/0295 by overlaying a vehicle path based on present steering angle over an image without processing that image
• CPC Class Codes: G08G 1/168 Driving aids for parking, e.g. acoustic or visual feedback on parking space
• CPC Class Codes: G09G 2340/125 wherein one of the images is motion video
• CPC Class Codes: G09G 2380/10 Automotive applications
• CPC Class Codes: G09G 5/14 Display of multiple viewports
• CPC Class Codes: H04N 23/60 Control of cameras or camera modules
• CPC Class Codes: H04N 23/63 by using electronic viewfinders
• CPC Class Codes: H04N 25/61 the noise originating only from the lens unit, e.g. flare, shading, vignetting or "cos4"

Abstract

In one aspect of the invention, a vehicular camera is provided, comprising a lens, a housing, an imager and a microcontroller that is capable of handling certain functions, such as applying overlays to images received by the imager, dewarping the image and/or providing different viewing modes.

Claims

1. A vehicular camera system, comprising:

a rear backup camera comprising a lens and having a field of view that encompasses a rear bumper of a vehicle equipped with said vehicular camera system;

wherein said rear backup camera comprises an image sensor and a housing;

wherein the lens and the image sensor are mounted within the housing at a selected position relative to each other, and wherein the image sensor receives raw images from the lens and captures image data;

a controller, wherein for at least one raw image, the controller is operable to (i) compare captured image data of the rear bumper present in the field of view of said rear backup camera to a known shape of the rear bumper of the vehicle, (ii) responsive to the comparison, determine a distortion correction that is indicative of distortion of the rear bumper in the raw image relative to the known shape of the rear bumper and (iii) apply the distortion correction to the raw image to generate a dewarped image by stretching and compressing portions of the raw image so that the rear bumper in the dewarped image corresponds to the known shape of the rear bumper;

a display screen disposed in the vehicle and operable to display dewarped images for viewing by a driver of the vehicle while the driver is operating the vehicle;

wherein the controller is operable to identify a portion of a trailer in the captured image data and, responsive at least in part to identifying the portion of the trailer, is further operable to determine the position of the portion of a trailer in the field of view of the rear backup camera and rearward of the vehicle, and wherein the controller, responsive to the determined position of the portion of the trailer, is operable to apply a first overlay at the portion of the trailer in a first displayed dewarped image to produce an initial displayed dewarped image, and wherein the controller is operable to thereafter maintain the first overlay in a fixed position in subsequent displayed dewarped images, and wherein the portion of the trailer in subsequent displayed dewarped images is movable relative to the first overlay; and

wherein the controller, responsive at least in part to processing of captured image data, is operable to display a second overlay in displayed dewarped images, and wherein the controller is operable to thereafter move the second overlay in subsequent displayed dewarped images relative to the fixed first overlay as the vehicle is driven to assist the driver of the vehicle in reversing the vehicle towards the trailer.

2. A vehicular camera system as claimed in claim 1, wherein the lens has an optical axis,

wherein the rear backup camera is mounted to a vehicle in such a way that a horizon line in the field of view of the lens approximately intersects the optical axis of the lens,

and wherein the portions of the raw image stretched and compressed are vertically extending portions of the raw image above and below the horizon line in the raw image to at least partially straighten out the horizon line appearing in the displayed dewarped image.

3. A vehicular camera system as claimed in claim 2, wherein the lens has approximately a 180 degree field of view horizontally, and has a first viewing mode and a second viewing mode,

wherein in the first viewing mode dewarped images are sent to a display in the vehicle showing approximately 180 degrees of the field of view horizontally, and the controller is operable to stretch and compress portions of the raw image using first values so that the rear bumper in the dewarped image corresponds to the known shape of the rear bumper,

and wherein in the second viewing mode images are sent to a display in the vehicle showing less of the field of view horizontally than in the first viewing mode, and the controller is operable to stretch and compress portions of the raw image using second values so that the rear bumper in the dewarped image corresponds to the known shape of the rear bumper.

4. A vehicular camera system as claimed in claim 3, wherein in the second viewing mode dewarped images are sent to a display in the vehicle showing approximately 135 degrees of the field of view horizontally.

5. A vehicular camera system as claimed in claim 2, wherein the rear backup camera has a first viewing mode and a second viewing mode,

wherein in the first viewing mode dewarped images are sent to said display screen in the vehicle showing a field of view horizontally, and the controller is operable to stretch and compress portions of the raw image using first values so that the rear bumper in the dewarped image corresponds to the known shape of the rear bumper,

and wherein in the second viewing mode images are sent to said display screen in the vehicle showing a second field of view that is less of the field of view horizontally than in the first viewing mode, and the controller is operable to stretch and compress portions of the raw image using second values so that the rear bumper in the dewarped image corresponds to the known shape of the rear bumper.

6. A vehicular camera system as claimed in claim 1, wherein the known shape of the rear bumper comprises an edge of the rear bumper on the vehicle.

7. A vehicular camera system as claimed in claim 1, wherein the lens has approximately a 180 degree field of view horizontally.

8. A vehicular camera system as claimed in claim 1, wherein the lens has approximately a 135 degree field of view horizontally.

9. A vehicular camera system as claimed in claim 1, wherein the image sensor is part of an imager, and the controller is made up of a portion of the imager and a microcontroller that is external to the imager.

10. A vehicular camera system as claimed in claim 9, wherein the microcontroller contains flash memory, wherein the microcontroller is connected to an image processor by a first bus through which command data is communicated to the image processor from the microcontroller, wherein the command data includes application instructions to draw application data from a selected point in the flash memory, and wherein the microcontroller is connected to the image processor by a second bus through which application data is communicated to the image processor from the flash memory.

11. A vehicular camera system as claimed in claim 10, wherein the first bus is an inter-integrated circuit bus, and the second bus is a serial peripheral interface bus.

12. A vehicular camera system as claimed in claim 10, wherein the command data includes data relating to an overlay, and wherein the microcontroller is operable to send position data to the image processor relating to a position at which to apply the overlay to one or more displayed images.

13. A vehicular camera system as claimed in claim 10, wherein the microcontroller is operable to calculate at least one of the size and shape of an overlay and to transmit data relating to the overlay to the image processor for the image processor to apply to one or more displayed images.

14. A vehicular camera system as claimed in claim 1, wherein the vehicular camera system is for a vehicle having a towing device.

15. A vehicular camera system as claimed in claim 14, wherein the controller is operable to modify at least one raw image to produce a processed image, and wherein the rear backup camera is positioned at an actual viewing angle such that the towing device is in the field of view of the lens, and wherein the rear backup camera has a bird'"'"'s eye viewing mode in which the controller is operable to modify the raw image so that the processed image, when displayed on said display screen, appears to have been taken at an apparent viewing angle that is more vertically oriented than the actual viewing angle.

16. A vehicular camera system as claimed in claim 15, wherein the controller is operable to compress a lower portion of the raw image and stretch an upper portion of the raw image so that the apparent viewing angle is more vertically oriented than the actual viewing angle.

17. A vehicular camera system as claimed in claim 15, wherein the rear backup camera has a hitch viewing mode in which the processed image, when displayed on said display screen, appears to have been taken at an apparent viewing angle that is more vertically oriented than the actual viewing angle, and in which the towing device in the displayed processed image is magnified to a magnification level that is greater than a magnification level provided in the bird'"'"'s eye viewing mode.

18. A vehicular camera system as claimed in claim 15, wherein the controller is operable to determine a projected trajectory for the vehicle and is operable to apply a projected trajectory overlay to the displayed processed image containing a representation of the projected trajectory.

19. A vehicular camera system as claimed in claim 18, wherein the controller is operable to apply at least one indicator of distance from the towing device to displayed images.

20. A vehicular camera system as claimed in claim 1, wherein the controller is operable to move the second overlay relative to the first overlay to indicate to the driver of the vehicle when the trajectory of the vehicle has changed to assist the driver of the vehicle in driving the vehicle towards the trailer.

21. A vehicular camera system as claimed in claim 1, wherein said controller is operable to determine the presence of a feature of a hitch portion of a trailer behind the vehicle and wherein the controller is operable to move the second overlay relative to the first overlay to indicate to the driver of the vehicle when the trajectory of the vehicle has changed to assist the driver of the vehicle in reversing the vehicle towards the hitch portion of the trailer.

22. A vehicular camera system as claimed in claim 21, wherein the image sensor is part of an imager and the controller is made up of a portion of the imager and a microcontroller that is external to the imager, and wherein the portion of the imager is operable to determine the position of the feature of the hitch portion in the field of view of the lens, and wherein the microcontroller is operable to receive from the imager a position indicator indicating the position of the feature of the hitch portion, wherein the microcontroller and the portion of the imager are together operable to apply the first overlay at the portion of the trailer in the first image to produce the initial image and to thereafter maintain the first overlay in the fixed position in subsequent displayed images regardless of movement of the portion of the trailer in subsequent captured images.

23. A vehicular camera system as claimed in claim 1, wherein the known shape of the rear bumper comprises a known generally horizontal edge of the rear bumper and wherein said controller applies the distortion correction to straighten a curved edge of the rear bumper in the raw image so the edge of the rear bumper in the dewarped image corresponds to the known generally horizontal edge of the rear bumper.

24. A vehicular camera system as claimed in claim 1, wherein the known shape of the rear bumper comprises a known generally vertical edge of the rear bumper and wherein said controller applies the distortion correction to straighten a curved edge of the rear bumper in the raw image so the edge of the rear bumper in the dewarped image corresponds to the known generally vertical edge of the rear bumper.