Processing of mosaic images

• US 20080095466A1
• Filed: 03/07/2007
• Published: 04/24/2008
• Est. Priority Date: 11/27/2006
• Status: Active Grant

• Current Assignee: DigitalOptics Corporation (Adeia Inc.)
• Original Assignee: Dblur Technologies Ltd. (Adeia Inc.)
• Inventors: Alon, Alex, Kinrot, Uri
• Granted Patent: US 7,907,791 B2
• Time in Patent Office: Days
• US Class Current: 382/284
• CPC Class Codes: G06T 1/0007 Image acquisition
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/20192 Edge enhancement; Edge preservation
• CPC Class Codes: G06T 5/002 Denoising; Smoothing noise processing or correction adapted to be used in an image pickup device containing and electronic image sensor H04N23/81, H04N25/60 - H04N25/67
• CPC Class Codes: G06T 5/003 Deblurring; Sharpening vibration or motion blur correction for cameras comprising an electronic image sensor H04N23/682
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: G06T 5/73 Deblurring; Sharpening
• CPC Class Codes: H04N 2209/045 using mosaic colour filter
• CPC Class Codes: H04N 23/843 Demosaicing, e.g. interpolating colour pixel values
• CPC Class Codes: H04N 25/134 based on three different wavelength filter elements
• CPC Class Codes: H04N 5/21 Circuitry for suppressing or minimising disturbance, e.g. moiré or halo

Abstract

Imaging apparatus includes a mosaic image sensor, which is configured to generate a stream of input pixel values belonging to a plurality of input sub-images, each sub-image responsive to light of a different, respective color that is incident on the mosaic image sensor. An image restoration engine (IRE) is coupled to receive and digitally filter the input pixel values in each of the input sub-images so as to generate a corresponding plurality of enhanced output sub-images, and to extract auxiliary information from the input sub-images prior to digitally filtering the input pixel values. An image signal processor (ISP) is coupled to receive the plurality of the output sub-images and the auxiliary information from the IRE, and to combine the output sub-images in order to generate a color video output image while enhancing the color video output image using the auxiliary information.

Claims

1. Imaging apparatus, comprising:

a mosaic image sensor, which is configured to generate a stream of input pixel values belonging to a plurality of input sub-images, each sub-image responsive to light of a different, respective color that is incident on the mosaic image sensor;

an image restoration engine (IRE), which is coupled to receive and digitally filter the input pixel values in each of the input sub-images so as to generate a corresponding plurality of enhanced output sub-images, and to extract auxiliary information from the input sub-images prior to digitally filtering the input pixel values; and

an image signal processor (ISP), which is coupled to receive the plurality of the output sub-images and the auxiliary information from the IRE, and to combine the output sub-images in order to generate a color video output image while enhancing the color video output image using the auxiliary information.

2. The apparatus according to claim 1, wherein the IRE comprises a deconvolution filter for reducing a blur of the input sub-images.

3. The apparatus according to claim 1, wherein the mosaic image sensor comprises an array of filters that are arranged in a Bayer mosaic pattern.

4. The apparatus according to claim 1, wherein the auxiliary information comprises edge information.

5. The apparatus according to claim 4, wherein the edge information comprises an edge mask identifying edge regions in the input sub-images.

6. The apparatus according to claim 4, wherein the edge information comprises at least one edge characteristic that is determined respectively for each of a plurality of pixels in the input sub-images, wherein the at least one edge characteristic is selected from a group of characteristics consisting of an edge magnitude and an edge direction.

7. The apparatus according to claim 4, wherein the ISP is configured to enhance the color video output image by performing at least one image enhancement operation responsively to the edge information, wherein the at least one image enhancement operation is selected from a group of operations consisting of noise suppression and edge enhancement.

8. The apparatus according to claim 1, wherein the auxiliary information comprises at least one item of information selected from a group of items consisting of an identification of bad pixels and a local noise parameter.

9. The apparatus according to claim 1, wherein the auxiliary information comprises at least one type of pixel value data selected from a group of types of pixel value data consisting of the input pixel values, and differences between the input pixel values and output pixel values of the output sub-images.

10. The apparatus according to claim 1, wherein the output sub-images comprise output pixel values, and wherein the IRE is configured to convey the auxiliary information to the ISP by adding one or more auxiliary bits to the output pixel values.

11. The apparatus according to claim 10, wherein the IRE is configured to convey the output pixel values and the one or more auxiliary bits to the ISP via a serial data bus.

12. The apparatus according to claim 10, wherein the IRE is configured to convey the output pixel values and the one or more auxiliary bits to the ISP via a parallel data bus.

13. The apparatus according to claim 1, wherein the ISP is configured to perform one or more image processing and control operations selected from the group of operations consisting of demosaic processing, white balancing, and auto-exposure control, and to apply the auxiliary information in performing at least one of the one or more image processing and control operations.

14. Imaging apparatus, comprising:

a mosaic image sensor, which is configured to generate a stream of input pixel values belonging to a plurality of input sub-images, each sub-image responsive to light of a different, respective color that is incident on the mosaic image sensor;

an image restoration engine (IRE), which is coupled to receive and digitally filter the input pixel values in each of the input sub-images using filter parameters determined responsively to auxiliary information, so as to generate a corresponding plurality of enhanced output sub-images; and

an image signal processor (ISP), which is coupled to receive and combine the plurality of the output sub-images in order to generate a color video output image, while generating, responsively to the combined sub-images, the auxiliary information for input to the image restoration engine.

15. The apparatus according to claim 14, wherein the IRE is configured to detect edges in the input sub-images, and wherein the auxiliary information comprises an edge detection threshold for use in detecting the edges in the input sub-images.

16. The apparatus according to claim 14, wherein the auxiliary information comprises an indication of a filter kernel to be applied by the IRE in digitally filtering the input pixel values.

17. Imaging apparatus, comprising:

a mosaic image sensor, which is configured to generate a stream of input pixel values belonging to a plurality of sub-images, each sub-image responsive to light of a different, respective color that is incident on the mosaic image sensor;

an edge detector, which is coupled to extract edge information from one or more of the sub-images; and

an image signal processor (ISP), which is coupled to receive and combine the plurality of the sub-images in order to generate a color video output image, and to enhance the color video output image using the edge information.

18. The apparatus according to claim 17, wherein the edge information comprises an edge mask identifying edge regions in the sub-images.

19. The apparatus according to claim 17, wherein the edge information comprises at least one edge characteristic that is determined respectively for each of a plurality of pixels in the sub-images, wherein the at least one edge characteristic is selected from a group of characteristics consisting of an edge magnitude and an edge direction.

20. The apparatus according to claim 17, wherein the ISP is configured to enhance the color video output image by performing at least one image enhancement operation responsively to the edge information, wherein the at least one image enhancement operation is selected from a group of operations consisting of noise suppression and edge enhancement.

21. The apparatus according to claim 17, wherein the edge detector is configured to extract the edge information at each pixel among a plurality of pixels in at least one sub-image by processing the input pixel values of a first group of the pixels that are horizontally aligned with the pixel and of a second group of the pixels that are vertically aligned with the pixel.

22. A method for imaging, comprising:

receiving from a mosaic image sensor a stream of input pixel values belonging to a plurality of input sub-images, each sub-image responsive to light of a different, respective color that is incident on the mosaic image sensor;

digitally filtering the input pixel values in each of the input sub-images so as to generate a corresponding plurality of enhanced output sub-images;

extracting auxiliary information from the input sub-images prior to digitally filtering the input pixel values; and

combining the output sub-images in order to generate a color video output image, while enhancing the color video output image using the auxiliary information.

23. The method according to claim 22, wherein digitally filtering the input pixel values comprises applying a deconvolution filter to reduce a blur of the input sub-images.

24. The method according to claim 22, wherein the mosaic image sensor comprises an array of filters that are arranged in a Bayer mosaic pattern.

25. The method according to claim 22, wherein the auxiliary information comprises edge information.

26. The method according to claim 25, wherein the edge information comprises an edge mask identifying edge regions in the input sub-images.

27. The method according to claim 25, wherein the edge information comprises at least one edge characteristic that is determined respectively for each of a plurality of pixels in the input sub-images, wherein the at least one edge characteristic is selected from a group of characteristics consisting of an edge magnitude and an edge direction.

28. The method according to claim 25, wherein enhancing the color video output image comprises performing at least one image enhancement operation responsively to the edge information, wherein the at least one image enhancement operation is selected from a group of operations consisting of noise suppression and edge enhancement.

29. The method according to claim 22, wherein the auxiliary information comprises at least one item of information selected from a group of items consisting of an identification of bad pixels and a local noise parameter.

30. The method according to claim 22, wherein the auxiliary information comprises at least one type of pixel value data selected from a group of types of pixel value data consisting of the input pixel values, and differences between the input pixel values and output pixel values of the output sub-images.

31. The method according to claim 22, wherein the output sub-images comprise output pixel values and are combined by an image signal processor (ISP), and wherein extracting the auxiliary information comprises conveying the auxiliary information to the ISP by adding one or more auxiliary bits to the output pixel values.

32. The method according to claim 31, wherein conveying the auxiliary information comprises conveying the output pixel values and the one or more auxiliary bits to the ISP via a serial data bus.

33. The method according to claim 31, wherein conveying the auxiliary information comprises conveying the output pixel values and the one or more auxiliary bits to the ISP via a parallel data bus.

34. The method according to claim 31, wherein combining the output sub-images and enhancing the color video output image comprise performing one or more image processing and control operations selected from the group of operations consisting of demosaic processing, white balancing, and auto-exposure control, and applying the auxiliary information in performing at least one of the one or more image processing and control operations.

35. A method for imaging, comprising:

receiving from a mosaic image sensor a stream of input pixel values belonging to a plurality of input sub-images, each sub-image responsive to light of a different, respective color that is incident on the mosaic image sensor;

digitally filtering the input pixel values in each of the input sub-images using filter parameters determined responsively to auxiliary information, so as to generate a corresponding plurality of enhanced output sub-images;

combining the output sub-images in order to generate a color video output image while enhancing the color video output image; and

generating the auxiliary information responsively to the combined sub-images.

36. The method according to claim 35, wherein digitally filtering the input pixel values comprises detecting edges in the input sub-images, and wherein generating the auxiliary information comprises providing an edge detection threshold for use in detecting the edges in the input sub-images.

37. The method according to claim 35, wherein generating the auxiliary information comprises providing an indication of a filter kernel to be applied in digitally filtering the input pixel values.

38. A method for imaging, comprising:

receiving from a mosaic image sensor a stream of input pixel values belonging to a plurality of sub-images, each sub-image responsive to light of a different, respective color that is incident on the mosaic image sensor;

extracting edge information from one or more of the sub-images;

combining the output sub-images in order to generate a color video output image; and

enhancing the color video output image using the edge information.

39. The method according to claim 38, wherein the edge information comprises an edge mask identifying edge regions in the sub-images.

40. The method according to claim 38, wherein the edge information comprises at least one edge characteristic that is determined respectively for each of a plurality of pixels in the sub-images, wherein the at least one edge characteristic is selected from a group of characteristics consisting of an edge magnitude and an edge direction.

41. The method according to claim 38, wherein enhancing the color video output image comprises performing at least one image enhancement operation responsively to the edge information, wherein the at least one image enhancement operation is selected from a group of operations consisting of noise suppression and edge enhancement.

42. The method according to claim 38, wherein extracting the edge information comprises, at each pixel among a plurality of pixels in at least one sub-image, processing the input pixel values of a first group of the pixels that are horizontally aligned with the pixel and of a second group of the pixels that are vertically aligned with the pixel.

43. A computer software product, comprising a computer-readable medium in which program instructions are stored, which instructions, when read by the computer, cause the computer to receive, from a mosaic image sensor, a stream of input pixel values belonging to a plurality of sub-images, each sub-image responsive to light of a different, respective color that is incident on the mosaic image sensor, to extract edge information from one or more of the sub-images, to combine the plurality of the sub-images in order to generate a color video output image, and to enhance the color video output image using the edge information.

44. A computer software product, comprising a computer-readable medium in which program instructions are stored, which instructions, when read by the computer, cause the computer to receive, from a mosaic image sensor, a stream of input pixel values belonging to a plurality of sub-images, each sub-image responsive to light of a different, respective color that is incident on the mosaic image sensor, to digitally filter the input pixel values in each of the input sub-images so as to generate a corresponding plurality of enhanced output sub-images, and to extract auxiliary information from the input sub-images prior to digitally filtering the input pixel values, and to combine the output sub-images in order to generate a color video output image while enhancing the color video output image using the auxiliary information.