Image processing device and method

• US 20070110331A1
• Filed: 10/13/2005
• Published: 05/17/2007
• Est. Priority Date: 10/14/2004
• Status: Active Grant

• Current Assignee: Nissan Motor Co., Ltd.
• Original Assignee: Nissan Motor Co., Ltd.
• Inventors: Satoh, Hiroshi
• Granted Patent: US 7,865,029 B2
• Time in Patent Office: Days
• US Class Current: 382/275
• CPC Class Codes: B60R 1/00 Optical viewing arrangements; Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles
• CPC Class Codes: B60R 2300/302 combining image information with GPS information or vehicle data, e.g. vehicle speed, gyro, steering angle data
• CPC Class Codes: B60R 2300/402 Image calibration
• CPC Class Codes: H04N 25/67 applied to fixed-pattern noise, e.g. non-uniformity of response

Abstract

A vehicle-mounted image-processing device and method is provided to remove fixed pattern noise from images due to the image pickup element. Images captured consecutively with a camera are superimposed and stored in an image memory. A controller extracts the high spatial frequency component from the stored images, and the images are corrected with the extracted high spatial frequency component used as the correction data for removing the fixed pattern noise.

Claims

1. An image processing device for removing fixed pattern noise in images captured by an image pickup device mounted on a vehicle, comprising:

(a) memory on which is stored a plurality of images captured by the image pickup device while the vehicle is running; and

(b) a controller operatively coupled to the memory and adapted to generate correction data by extracting high spatial frequency components from portions of the stored images, and to use the correction data to remove fixed pattern noise from images captured by the image pickup device.

2. The image processing device of claim 1, wherein the controller is further adapted to use a high pass spatial frequency filter to extract the high spatial frequency components.

3. The image processing device of claim 1, wherein the controller is further adapted to use a low pass spatial frequency filter to extract the high spatial frequency components.

4. The image processing device of claim 1, further comprising:

a sensor adapted to be mounted to the vehicle and to output a behavior signal indicating whether the vehicle is undergoing a detected type of movement;

wherein the controller is further adapted to store in memory the images captured by the image pickup device in response to the behavior signal indicating that the vehicle is undergoing the detected type of movement.

5. The image processing device of claim 4, wherein the detected type of movement is a yaw movement that is sufficient to blur the images captured by the image pickup device, and wherein the controller is further adapted to extract correction data from horizontally-oriented portions of the images.

6. The image processing device of claim 4, wherein the detected type of movement is a pitch movement that is sufficient to blur the images captured by the image pickup device, and wherein the controller is further adapted to extract correction data from vertically-oriented portions of the images.

7. The image processing device of claim 4, wherein the controller is further adapted to store the images in the memory during a time period that varies with the rate of the detected type of movement.

8. The image processing device of claim 7, wherein the controller is further adapted to store the images in memory during a time period that varies inversely with the rate of the detected type of movement.

9. The image processing device of claim 1, further comprising:

a wiper area extractor adapted to extract, from the images captured by the image pickup device, a plurality of areas that correspond to windshield wipers;

wherein the controller is further adapted to superimpose the plurality of extracted windshield wipers areas to form one or more of the stored image portions from which correction data is extracted.

10. The image processing device of claim 9, wherein the wiper extractor is adapted to calculate speed of objects in the images captured by the image pickup device, and to extract as the windshield wiper areas the objects in the images having a speed above a prescribed value.

11. The image processing device of claim 1 wherein the processor is further adapted to extract a high spatial frequency component for which the luminance output value is highest in the stored images and a high spatial frequency component for which the luminance output value is lowest in the stored images, and to adjust the gain and offset of the images captured by the image pickup means based on the extracted high spatial frequency component at high luminance and the high spatial frequency component at low luminance.

12. An image processing device for removing fixed pattern noise from images captured by an image pickup device mounted on a vehicle, comprising:

(a) storage means for storing a plurality images captured with the image pickup means;

(b) extracting means for extracting the high spatial frequency component contained in the images in the storage means and generating correction data for removing the fixed pattern noise based on the high spatial frequency component; and

(c) image correcting means for removing fixed pattern noise from the images captured with the image pickup means based on the correction data.

13. The image processing device of claim 12, wherein the storage means stores images captured by the image pickup device while the vehicle is traveling.

14. The image processing device of claim 12, further comprising:

(a) behavior detecting means for detecting the behavior of the vehicle;

(b) storage time setting means for determining a time period during which images captured by the image pickup means will be stored in the storage means based on the behavior of the vehicle detected by the behavior detecting means; and

wherein the storage means stores the images captured by the image pickup means for the duration of the storage time set by the storage time setting means.

15. The image processing device of claim 12, further comprising:

wiper area extraction means for extracting the areas in which a wiper is present from images captured by the image pickup means;

wherein the storage means stores a plurality of wiper areas that have been extracted from images by the wiper area extraction means.

16. The image processing device of claim 15, wherein the wiper area extraction means further comprises:

speed calculation means for processing the images captured by the image pickup means to calculate speed information for the images;

wherein the area on the images for which speed above a prescribed value is calculated is extracted as the wiper area based on the image speed information calculated by the speed calculation means.

17. The image processing device of claim 12, further comprising:

a gain offset means for extracting the high spatial frequency component for which the luminance output value is highest in the stored images and a high spatial frequency component for which the luminance output value is lowest in the stored images, and for correcting the gain and offset of the images captured by the image pickup means based on the extracted high spatial frequency component at high luminance and the high spatial frequency component at low luminance to improve the uniformity of the images.

18. A vehicle having an image processing device for removing fixed pattern noise in images, comprising:

(a) a vehicle having a camera mounted thereto, the camera including an image-pickup element generating as output a plurality of images;

(b) memory on which is stored the plurality of images generated by the image pickup element while the vehicle is running; and

(c) a controller operatively coupled to the memory and adapted to generate correction data by extracting high spatial frequency components from portions of the stored images, and to use the correction data to remove fixed pattern noise from images captured by the image pickup element.

19. The vehicle of claim 18, further comprising:

a sensor mounted to the vehicle and adapted to output a behavior signal indicating whether the vehicle is undergoing a detected type of movement;

wherein the controller is further adapted to store in memory the images captured by the image pickup element in response to the behavior signal indicating that the vehicle is undergoing the detected type of movement.

20. The vehicle of claim 19, wherein the detected type of movement is a yaw movement that is sufficient to blur the images generated by the image pickup element, and wherein the controller is further adapted to extract high spatial frequency components from horizontally-oriented portions of the images.

21. The vehicle of claim 19, wherein the detected type of movement is a pitch movement that is sufficient to blur the images generated by the image pickup element, and wherein the controller is further adapted to extract high spatial frequency components from vertically-oriented portions of the images.

22. The vehicle of claim 19, wherein the controller is further adapted to store the images in the memory during a time period that varies with the rate of the detected type of movement.

23. The vehicle of claim 18, further comprising:

windshield wipers and a wiper controller adapted to generate a wiper activation signal when the windshield wipers are operational;

wherein the controller is responsive to the wiper activation signal and is further adapted to extract areas of the images that correspond to the windshield wipers, and to superimpose the extracted windshield wiper areas in the memory to form one or more of the stored image portions from which the controller extracts high spatial frequency components.

24. The vehicle of claim 23 wherein the controller is further adapted to calculate speed of objects in the images generated by, and to extract as the windshield wiper areas the areas on the images having a speed above a prescribed value.

25. The vehicle of claim 18 wherein the processor is further adapted to extract a high spatial frequency component for which the luminance output value is highest in the stored images and a high spatial frequency component for which the luminance output value is lowest in the stored images, and to correct the gain and offset of the images captured by the image pickup means based on the extracted high spatial frequency component at high luminance and the high spatial frequency component at low luminance.

26. An image processing system for removing fixed pattern noise in images, comprising:

(a) a camera including an image pickup element generating as output a plurality of images;

(b) memory on which is stored the plurality of images generated by the image pickup element; and

(c) a controller operatively coupled to the memory and adapted to generate correction data by extracting high spatial frequency components from portions of the stored images, and to use the correction data to remove fixed pattern noise from images captured by the image pickup element.

27. The image processing system of claim 26, further comprising:

a sensor adapted to output a behavior signal indicating that the camera is undergoing a detected type of movement;

wherein the controller is further adapted to store in memory the images captured by the image pickup element in response to the behavior signal indicating the detected type of movement.

28. The image processing system of claim 26, wherein the detected type of movement is a yaw movement that is sufficient to blur the images captured by the image pickup element, and wherein the controller is further adapted to extract high spatial frequency components from horizontally-oriented portions of the images.

29. The image processing system of claim 26, wherein the detected type of movement is a pitch movement that is sufficient to blur the images captured by the image pickup element, and wherein the controller is further adapted to extract high spatial frequency components from vertically-oriented portions of the images.

30. The image processing system of claim 27, wherein the behavior signal indicates the rate of the detected type of movement, and the controller is further adapted to store the images in the memory during a time period that varies with the rate of the detected type of movement.

31. The image processing system of claim 26, wherein the controller is further adapted to extract areas of the images that correspond to windshield wipers, and to superimpose the extracted windshield wiper areas in the memory to form at least one of stored image portions from which the controller extracts high spatial frequency components.

32. The image processing system of claim 31, wherein the controller is further adapted to calculate speed of objects in the images generated by the image pickup element, and to extract as the windshield wiper areas the areas on the images having a speed above a prescribed value.

33. The image processing system of claim 26, wherein the processor is further adapted to extract a high spatial frequency component for which the luminance output value is highest in the stored images and a high spatial frequency component for which the luminance output value is lowest in the stored images, and to correct the gain and offset of the images captured by the image pickup means based on the extracted high spatial frequency component at high luminance and the high spatial frequency component at low luminance.

34. A image processing method for removing fixed pattern noise from images captured by an image pickup device mounted on a vehicle, comprising:

(a) storing a plurality of images captured with the image pickup means;

(b) extracting the high spatial frequency component contained in the images;

(c) generating correction data for removing the fixed pattern noise based on the high spatial frequency component; and

(d) using the correction data to remove fixed pattern noise from the images captured by the image pickup.

35. The image processing method of claim 34, wherein storing the plurality of images takes place while the vehicle is moving.

36. The image processing method of claim 34, further comprising:

detecting the behavior of the vehicle;

based on the behavior of the vehicle, determining a time period during which images captured by the image pickup means will be stored; and

storing the plurality images during the time period.

37. The image processing method of claim 34, further comprising:

extracting the area in which a wiper is present from images captured by the image pickup device;

superimposing the wiper areas to define an image having at least one portion that is black; and

extracting high spatial frequency component contained in the black portion.

38. The image processing method of claim 34, further comprising:

processing the images captured by the image pickup means to calculate speed information for the images; and

extracting as wiper areas the portions of the images for which speed above a prescribed value is calculated.

39. The image processing method of claim 34, further comprising:

extracting a high spatial frequency component for which the luminance output value is highest in the stored images and a high spatial frequency component for which the luminance output value is lowest in the stored images, and improving the uniformity of the images by correcting the gain and offset of the images based on the extracted high spatial frequency component at high luminance and the high spatial frequency component at low luminance.