Panoramic vision system and method

• US 20060017807A1
• Filed: 07/26/2004
• Published: 01/26/2006
• Est. Priority Date: 07/26/2004
• Status: Active Grant

• Current Assignee: Geo Semiconductor Incorporated
• Original Assignee: Silicon Optix (Renesas Electronics Corporation)
• Inventors: Vakili, Masoud, Lee, Louie
• Granted Patent: US 7,576,767 B2
• Time in Patent Office: Days
• US Class Current: 348/36
• CPC Class Codes: B60R 1/00 Optical viewing arrangements; Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles
• CPC Class Codes: B60R 1/27 providing all-round vision, e.g. using omnidirectional cameras
• CPC Class Codes: B60R 2300/105 using multiple cameras
• CPC Class Codes: B60R 2300/302 combining image information with GPS information or vehicle data, e.g. vehicle speed, gyro, steering angle data
• CPC Class Codes: B60R 2300/303 using joined images, e.g. multiple camera images
• CPC Class Codes: B60R 2300/8026 in addition to a rear-view mirror system
• CPC Class Codes: H04N 23/635 Region indicators; Field of view indicators
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 23/81 for suppressing or minimising disturbance in the image signal generation
• CPC Class Codes: H04N 7/147 Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals selecting H04Q
• CPC Class Codes: H04N 7/15 Conference systems
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources

Abstract

A distortion corrected panoramic vision system and method provides a visually correct composite image acquired through wide angle optics and projected onto a viewing surface. The system uses image acquisition devices to capture a scene up to 360° or 4π steradians broad. An image processor corrects for luminance or chrominance non-uniformity and applies a spatial transform to each image frame. The spatial transform is convolved by concatenating the viewing transform, acquisition geometry and optical distortion transform, and display geometry and optical transform. The distortion corrections are applied separately for red, green, and blue components to eliminate lateral color aberrations of the optics. A display system is then used to display the resulting composite image on a display device which is then projected through the projection optics and onto a viewing surface. The resulting image is visibly distortion free and matches the characteristics of the viewing surface.

Claims

1. A panoramic vision system having associated camera, display optics and geometric characteristics, said system comprising:

(a) a plurality of image acquisition devices to capture image frame data from a scene and to generate image sensor inputs, said image frame data collectively covering up to a 360°

field of view, said image acquisition devices having geometric and optical distortion parameters;

(b) a digitizer coupled to the plurality of image acquisition devices to sample and convert the image frame data and the image sensor inputs into digital image data;

(c) an image processor coupled to the digitizer comprising;

(i) an image measurement device to receive the digital image data and to measure the image luminance histogram and the ambient light level associated with the digital image data;

(ii) a luminance correction module coupled to the image measurement device to receive the digital image data along with the camera, display optics and geometric characteristics, the image luminance histogram and the ambient light level, and to correct for luminance non-uniformities and to optimize the luminance range of selected regions within the digital image data;

(iii) a convolution stage coupled to the luminance correction module to combine the geometric and optical distortion parameters including the image sensor inputs, the camera, display optics and geometric characteristics and imperfections associated therein to form convoluted distortion parameters;

(iv) a distortion correction module coupled to the convolution stage to generate and apply a distortion correction transformation based on the convoluted distortion parameters to the digital image data to generate corrected digital image data;

(v) a display controller coupled to the distortion correction module to synthesize a composite image from the corrected digital image data; and

(d) a display system coupled to said image processor to display the composite image on a viewing surface for viewing, said composite image being visually distortion free.

2. The panoramic vision system of claim 1, wherein said image processor is adapted to separately process red, green, and blue components associated with the digital image data to correct for lateral color aberration and distortions.

3. The panoramic vision system of claim 2, wherein said luminance correction module separately processes the red, green, and blue components to correct for chrominance non-uniformity.

4. The panoramic vision system of claim 1, wherein said plurality of image acquisition devices and the digitizer are implemented by a plurality of digital cameras.

5. The panoramic vision system of claim 3, wherein the display system is a light projection system, and wherein, the geometric and optical distortion parameters are concatenated in order to pre-correct for the associated geometric and optical distortions.

6. The panoramic vision system of claim 3, further comprising at least one ambient light sensor, at least one lighting signal, and at least one user dimmer switch, wherein the brightness of the display system is adjusted according to at least one of said ambient light sensor, separate lighting signal, and user dimmer switch.

7. The panoramic vision system of claim 3, further comprising a controller wherein said controller is adapted to relay a set of user parameters to the panoramic vision system to concatenate said user parameters with distortion parameters based on a set of selected viewing functions and preferences.

8. The panoramic vision system of claim 3, further comprising a control interface coupled to the image processor which is adapted to receive and relay user inputs and control parameters.

9. The panoramic vision system of claim 3, wherein said image processor is further adapted to send exposure control commands to the image acquisition devices for iris and exposure time control.

10. The panoramic vision system of claim 3, further comprising a viewing surface, wherein the position and size of the composite image on the viewing surface is adjustable according to a set of user inputs.

11. The panoramic vision system of claim 3, provided as a vehicle vision system.

12. The vehicle vision system of claim 11, wherein the display system is adapted to form a composite image in the forward field of vision of a driver positioned within the vehicle, namely on at least one of the group consisting of:

dashboard, windshield, roof, and fascia.

13. The vehicle vision system of claim 12, wherein the display system is adapted to form a composite image on two viewing surfaces on the interior corners of the windshield and fascia in order to mimic vehicle side mirrors.

14. The vehicle vision system of claim 11, wherein the focal point of the composite image falls in the forward field of focus associated with a driver positioned within the vehicle.

15. The vehicle vision system of claim 11, wherein the focal point of the display system is practically infinite.

16. The vehicle vision system of claim 11, wherein the vehicle vision system reconfigures amongst different configurations according to input from the group consisting of:

a turning signal, a transmission system, a brake system, a steering system, and a speedometer.

17. The vehicle vision system of claim 16, wherein the display system is adapted to enhance the right view from the vehicle when the vehicle turning signal is set to turn right and to enhance the left view from the vehicle when the vehicle turning signal is set to turn left.

18. The vehicle vision system of claim 16, wherein the display system is adapted to enhance the right view from the vehicle when the vehicle steering wheel is turned toward the right by a preset angle and to enhance the left view from the vehicle when the vehicle steering wheel is turned toward the left by a preset angle.

19. The vehicle vision system of claim 16, wherein the display system is adapted to enhance the rear view from the vehicle when the vehicle is operating in reverse.

20. The vehicle vision system of claim 16, wherein the vision system is adapted to enhance at least one of the front and rear corner side view from the vehicle if the direct view is obscured by adjacent objects.

21. The vehicle vision system of claim 20, further comprising at least one of a laser and a radar sensor and wherein the image processor is further adapted to use the sensors to detect distances between the exterior of the vehicle and an object, and to provide audio and visual warnings when the distance between the exterior of the vehicle and the object falls within limits defined by the speed of the vehicle.

22. The vehicle vision system of claim 21, wherein the display system is configured to show the vehicle and it'"'"'s surrounding setting as viewed from the top by capturing two dimensions via cameras and extrapolating in the third dimension according to lane and curb patterns and object look-up databases.

23. The vehicle vision system of claim 20, wherein the image processor is further adapted to detect at least one of road, lane, and curb edge markings and to provide audio and visual warnings when the exterior of the vehicle approaches said edge markings.

24. The vehicle vision system of claim 23, wherein, the image processor is further adapted to detect the position of a designated area and to provide directional indicators on the display system to guide vehicle movement to the designated area.

25. The vehicle vision system of claim 11, further comprising at least one of the group consisting of:

ambient light sensors, oncoming headlight sensors, and driver dimmer switch, and wherein the brightness and contrast associated with the display system is adjusted according to the readings of at least one of the ambient light sensors, oncoming headlight sensors, and driver dimmer switch.

26. The vehicle vision system of claim 11, further comprising a secure black box device and a trigger device for recording all the digital image data associated with a period of time defined around activation of the trigger, said trigger bring adapted to be activated by at least one of the group consisting of:

a user, a collision, a hijacking, and a theft.

27. The vehicle vision system of claim 11, further comprising an integrated GPS navigation unit which allows for road maps and roadside information to be displayed on the display system.

28. The panoramic vision system of claim 3, further comprising a plurality of surveillance sensors, said system being provided as a surveillance vision system.

29. The surveillance vision system of claim 28, wherein the image processor is further adapted to perform motion detection and object tracking using the surveillance sensors.

30. The surveillance vision system of claim 28, wherein the image processor is adapted to provide a set of visual and audio signals when motion is detected by surveillance sensors.

31. The surveillance vision system of claim 26, wherein the image processor is further adapted to define a set of designated protected areas, to track a plurality of objects, and to generate audio and visual warnings when such objects enter into designated protected areas, and wherein the display system is reconfigured to highlight said objects when such objects enter into the designated protected areas.

32. The panoramic vision system of claim 3, further including a set of audio sensors and wherein the digitizer is further adapted to convert the audio sensor inputs into digital audio data, said system being provided as a videoconferencing vision system.

33. The videoconference vision system of claim 32, wherein the image processor and the display system are further adapted to generate and display a zoomed image of a videoconferencing user according to the strength and direction of the audio signal generated by the videoconferencing user.

34. A method for providing panoramic vision using a panoramic vision system having camera, display optics and geometric characteristics as well as geometric and optical distortion parameters, to generate a composite image that covers up to 360°

or 4π

steradians, said method comprising;

(a) acquiring image frame data from a scene, said image frame data collectively covering up to 360°

or 4π

steradian field of view, and generating a set of image senor inputs;

(b) converting the image frame data and the image senor inputs into digital image data, said digital image data being associated with an image luminance histogram and an ambient light level;

(c) obtaining the camera, display optics and geometric characteristics, the image luminance histogram and the ambient light level and correcting for luminance non-uniformities to optimize the luminance range of selected regions within the digital image data;

(d) convoluting the geometric and optical distortion parameters including the image sensor inputs, the camera, display optics and geometric characteristics and imperfections associated therein to form convoluted distortion parameters;

(e) generating and applying the distortion correction transformations to the digital image data, said distortion correction transformations being based on the convoluted geometric and optical distortion parameters to generate corrected digital image data;

(f) synthesizing a composite image from the corrected digital image data; and

(g) displaying the composite image on a viewing surface for viewing, said composite image being visually distortion free.

35. The method of claim 34, further processing the red, green, and blue components of the digital image data separately to correct for lateral color aberration distortions.

36. The method of claim 35, wherein luminance adjustment is conducted separately for each of said red, green and blue components to correct for chrominance non-uniformity.

37. The method of claim 36, wherein the correction transformation is achieved by:

(i) obtaining a grid data set covering the entire space of the geometric and optical distortion parameters;

(ii) computing a correction transformation corresponding to each data set on the grid of (i), and (iii) interpolating adjacent grid transformation to fit the actual image frame transformation parameters.

38. The method of claim 37, wherein (i) and (ii) are executed once off line and the transformation grid is accessed for each image frame correction.

39. The method of claim 37, wherein (i) and (ii) are done dynamically under varying conditions.

40. The method of claim 37, wherein the pixel map associated with the transformation is represented via warp maps.

41. The method of claim 40, wherein the warp maps are parameterized in terms of one of incremental polynomials and positional polynomials.

42. The method of claim 36, further generating and relaying exposure control commands to a plurality of image acquisition devices.

43. The method of claim 36, further including receiving control and user parameters to convolute with distortion parameters.

44. The method of claim 36, further including motion detection and tracking.

45. The method of claim 43, further including adjusting brightness, contrast, and size of the composite image according to sensors and user inputs.

46. The method of claim 43, further reconfiguring the display system according to a set of control inputs.

47. The method of claim 43, further adjusting the focal length of the composite image according to a set of control parameters.

48. An image processor, for use in a panoramic vision system having associated camera, display optics and geometric characteristics as well as geometric and optical distortion parameters, said panoramic vision system using a plurality of image acquisition devices to capture image frames from a scene and to generate digital image data and image sensor inputs and a digitizer to convert the digital image data and the image sensor inputs into digital image data, said image processor comprising:

(a) an image measurement device to receive the digital image data and to measure the image luminance histogram and the ambient light level associated with the digital image data;

(b) a luminance correction module coupled to the image measurement device to receive the digital image data along with the camera, display optics and geometric characteristics, the image luminance histogram, and the ambient light level and to correct for luminance non-uniformities and to optimize the luminance range of selected regions within the digital image data;

(c) a convolution stage coupled to the luminance correction module to combine the geometric and optical distortion parameters including the image sensor inputs, the camera, display optics and geometric characteristics and imperfections associated therein to form convoluted distortion parameters;

(d) a distortion correction module coupled to the convolution stage to generate and apply a distortion correction transformation, based on the convoluted distortion parameters, to the digital image data to generate corrected digital image data; and

(e) a display controller coupled to the distortion correction module to synthesize a composite image from the corrected digital image data.

49. The image processor of claim 48, further adapted to separately process red, green, and blue components of the digital image data to correct for lateral color aberration distortions.

50. The image processor of claim 49, wherein said luminance correction module separately processes red, green, and blue components of the digital image data to correct for chrominance non-uniformity.

51. The image processor of claim 50, further adopted to generate and relay exposure control commands to image acquisition devices.

52. The image processor of claim 50, further adapted to receive control and user parameters to convolute with distortion parameters.

53. The image processor of claim 50, further adopted for motion detection and tracking components.

54. The image processor of claim 50, further adopted for adjusting the brightness, contrast, and size of the synthesized image according to a plurality of sensors and a set of user inputs.

55. The image processor of claim 52, further adapted to reconfigure the display system according to a set of control inputs.

56. The image processor of claim 52, further adapted to adjust the focal length of the composite image according to a set of control parameters.