Processing of mosaic images

• US 7,907,791 B2
• Filed: 03/07/2007
• Issued: 03/15/2011
• Est. Priority Date: 11/27/2006
• Status: Active Grant

• Current Assignee: DigitalOptics Corporation (Adeia Inc.)
• Original Assignee: Tessera International Incorporated (Adeia Inc.)
• Inventors: Alon, Alex, Kinrot, Uri
• Primary Examiner(s): COUSO, YON JUNG
• Application Number:
  US11/715,862
• Publication Number:
  US 20080095466A1
• Time in Patent Office: 1,469 Days
• Field of Search: 382/254, 382/263, 382/276, 382/280, 382/284, 382/199
• US Class Current: 382/276
• CPC Class Codes: G06T 1/0007 Image acquisition
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/20192 Edge enhancement; Edge preservation
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: G06T 5/73 Deblurring; Sharpening
• CPC Class Codes: H04N 2209/045 using mosaic colour filter
• CPC Class Codes: H04N 23/843 Demosaicing, e.g. interpolating colour pixel values
• CPC Class Codes: H04N 25/134 based on three different wavelength filter elements
• CPC Class Codes: H04N 5/21 Circuitry for suppressing or minimising disturbance, e.g. moiré or halo

Abstract

Imaging apparatus includes a mosaic image sensor, which is configured to generate a stream of input pixel values belonging to a plurality of input sub-images, each sub-image responsive to light of a different, respective color that is incident on the mosaic image sensor. An image restoration engine (IRE) is coupled to receive and digitally filter the input pixel values in each of the input sub-images so as to generate a corresponding plurality of enhanced output sub-images, and to extract auxiliary information from the input sub-images prior to digitally filtering the input pixel values. An image signal processor (ISP) is coupled to receive the plurality of the output sub-images and the auxiliary information from the IRE, and to combine the output sub-images in order to generate a color video output image while enhancing the color video output image using the auxiliary information.

Claims

1. Imaging apparatus, comprising:

a mosaic image sensor, which is configured to generate a stream of input pixel values belonging to a plurality of input sub-images, each input sub-image corresponding to light of a respective color that is incident on the mosaic image sensor;

an image restoration engine (IRE), which is coupled to receive and digitally filter the input pixel values in each of the input sub-images so as to generate a corresponding plurality of enhanced output sub-images, and to extract auxiliary information from the input sub-images prior to digitally filtering the input pixel values; and

an image signal processor (ISP), which is coupled to receive the plurality of enhanced output sub-images and the auxiliary information from the IRE, and to process the enhanced output sub-images in order to generate a color video output image while enhancing the color video output image using the auxiliary information.

2. The apparatus according to claim 1, wherein the IRE comprises a deconvolution filter for reducing a blur of the input sub-images.

3. The apparatus according to claim 1, wherein the mosaic image sensor comprises an array of filters that are arranged in a Bayer mosaic pattern.

4. The apparatus according to claim 1, wherein the auxiliary information comprises edge information.

5. The apparatus according to claim 4, wherein the edge information comprises an edge mask identifying edge regions in the input sub-images.

6. The apparatus according to claim 4, wherein the edge information comprises at least one edge characteristic that is determined respectively for each of a plurality of pixels in the input sub-images, wherein the at least one edge characteristic is selected from a group of characteristics consisting of an edge magnitude and an edge direction.

7. The apparatus according to claim 4, wherein the ISP is configured to enhance the color video output image by performing at least one image enhancement operation responsively to the edge information, wherein the at least one image enhancement operation is selected from a group of operations consisting of noise suppression and edge enhancement.

8. The apparatus according to claim 1, wherein the auxiliary information comprises at least one item of information selected from a group of items consisting of an identification of bad pixels and a local noise parameter.

9. The apparatus according to claim 1, wherein the auxiliary information comprises at least one type of pixel value data selected from a group of types of pixel value data consisting of the input pixel values, and differences between the input pixel values and output pixel values of the enhanced output sub-images.

10. The apparatus according to claim 1, wherein the enhanced output sub-images comprise output pixel values, and wherein the IRE is configured to convey the auxiliary information to the ISP by adding one or more auxiliary bits to the output pixel values.

11. The apparatus according to claim 10, wherein the IRE is configured to convey the output pixel values and the one or more auxiliary bits to the ISP via a serial data bus.

12. The apparatus according to claim 10, wherein the IRE is configured to convey the output pixel values and the one or more auxiliary bits to the ISP via a parallel data bus.

13. The apparatus according to claim 1, wherein the ISP is configured to perform one or more image processing and control operations selected from the group of operations consisting of demosaic processing, white balancing, and auto-exposure control, and to apply the auxiliary information in performing at least one of the one or more image processing and control operations.

14. Imaging apparatus, comprising:

a mosaic image sensor, which is configured to generate a stream of input pixel values belonging to a plurality of input sub-images, each input sub-image corresponding to light of a respective color that is incident on the mosaic image sensor;

an image restoration engine (IRE), which is coupled to receive and digitally filter the input pixel values in each of the input sub-images using filter parameters determined responsively to auxiliary information, so as to generate a corresponding plurality of enhanced output sub-images; and

an image signal processor (ISP), which is coupled to receive and process the plurality of enhanced output sub-images in order to generate a color video output image, the ISP further generating the auxiliary information for input to the image restoration engine.

15. The apparatus according to claim 14, wherein the IRE is configured to detect edges in the input sub-images, and wherein the auxiliary information comprises an edge detection threshold for use in detecting the edges in the input sub-images.

16. The apparatus according to claim 14, wherein the auxiliary information comprises an indication of a filter kernel to be applied by the IRE in digitally filtering the input pixel values.

17. Imaging apparatus, comprising:

a mosaic image sensor, which is configured to generate a stream of input pixel values belonging to a plurality of sub-images, each sub-image corresponding to light of a respective color that is incident on the mosaic image sensor;

an edge detector, which is coupled to extract edge information from one or more of the sub-images; and

an image signal processor (ISP), which is coupled to receive and process the plurality of sub-images in order to generate a color video output image, and to enhance the color video output image using the edge information, wherein the ISP uses the edge information to determine which noise filter of a plurality of noise filters to use to filter noise from the color video output image.

18. The apparatus according to claim 17, wherein the edge information comprises an edge mask identifying edge regions in the sub-images.

19. The apparatus according to claim 17, wherein the edge information comprises at least one edge characteristic that is determined respectively for each of a plurality of pixels in the sub-images, wherein the at least one edge characteristic is selected from a group of characteristics consisting of an edge magnitude and an edge direction.

20. The apparatus according to claim 17, wherein the ISP is configured to enhance the color video output image by performing at least one image enhancement operation responsively to the edge information, wherein the at least one image enhancement operation is selected from a group of operations consisting of noise suppression and edge enhancement.

21. The apparatus according to claim 17, wherein the edge detector is configured to extract the edge information at each pixel among a plurality of pixels in at least one sub-image by processing the input pixel values of a first group of the pixels that are horizontally aligned with the pixel and of a second group of the pixels that are vertically aligned with the pixel.

45. The apparatus of claim 17, wherein the edge information is incorporated into the input pixel values belonging to the plurality of sub-images.

22. A method for imaging, comprising:

receiving from a mosaic image sensor a stream of input pixel values belonging to a plurality of input sub-images, each input sub-image corresponding to light of a respective color that is incident on the mosaic image sensor;

digitally filtering the input pixel values in each of the input sub-images so as to generate a corresponding plurality of enhanced output sub-images;

extracting auxiliary information from the input sub-images prior to digitally filtering the input pixel values; and

processing the enhanced output sub-images in order to generate a color video output image, while enhancing the color video output image using the auxiliary information.

23. The method according to claim 22, wherein digitally filtering the input pixel values comprises applying a deconvolution filter to reduce a blur of the input sub- images.

24. The method according to claim 22, wherein the mosaic image sensor comprises an array of filters that are arranged in a Bayer mosaic pattern.

25. The method according to claim 22, wherein the auxiliary information comprises edge information.

26. The method according to claim 25, wherein the edge information comprises an edge mask identifying edge regions in the input sub-images.

27. The method according to claim 25, wherein the edge information comprises at least one edge characteristic that is determined respectively for each of a plurality of pixels in the input sub-images, wherein the at least one edge characteristic is selected from a group of characteristics consisting of an edge magnitude and an edge direction.

28. The method according to claim 25, wherein enhancing the color video output image comprises performing at least one image enhancement operation responsively to the edge information, wherein the at least one image enhancement operation is selected from a group of operations consisting of noise suppression and edge enhancement.

29. The method according to claim 22, wherein the auxiliary information comprises at least one item of information selected from a group of items consisting of an identification of bad pixels and a local noise parameter.

30. The method according to claim 22, wherein the auxiliary information comprises at least one type of pixel value data selected from a group of types of pixel value data consisting of the input pixel values, and differences between the input pixel values and output pixel values of the enhanced output sub-images.

31. The method according to claim 22, wherein the enhanced output sub-images comprise output pixel values which are processed by an image signal processor (ISP), and wherein extracting the auxiliary information comprises conveying the auxiliary information to the ISP by adding one or more auxiliary bits to the output pixel values.

32. The method according to claim 31, wherein conveying the auxiliary information comprises conveying the output pixel values and the one or more auxiliary bits to the ISP via a serial data bus.

33. The method according to claim 31, wherein conveying the auxiliary information comprises conveying the output pixel values and the one or more auxiliary bits to the I8P via a parallel data bus.

34. The method according to claim 31, wherein processing the enhanced output sub-images and enhancing the color video output image comprise performing one or more image processing and control operations selected from the group of operations consisting of demosaic processing, white balancing, and auto-exposure control, and wherein the auxiliary information is applied in performing at least one of the one or more image processing and control operations.

35. A method for imaging, comprising:

receiving from a mosaic image sensor a stream of input pixel values belonging to a plurality of input sub-images, each input sub-image corresponding to light of a respective color that is incident on the mosaic image sensor;

digitally filtering the input pixel values in each of the input sub-images using filter parameters determined responsively to auxiliary information, so as to generate a corresponding plurality of enhanced output sub-images;

processing the enhanced output sub-images in order to generate a color video output image; and

generating the auxiliary information used in digitally filtering the input pixel values.

36. The method according to claim 35, wherein digitally filtering the input pixel values comprises detecting edges in the input sub-images, and wherein generating the auxiliary information comprises providing an edge detection threshold for use in detecting the edges in the input sub-images.

37. The method according to claim 35, wherein generating the auxiliary information comprises providing an indication of a filter kernel to be applied in digitally filtering the input pixel values.

38. A method for imaging, comprising:

receiving from a mosaic image sensor a stream of input pixel values belonging to a plurality of sub-images, each sub-image corresponding to light of a respective color that is incident on the mosaic image sensor;

extracting edge information from one or more of the sub-images;

processing the output sub-images in order to generate a color video output image; and

enhancing the color video output image using the edge information, wherein the edge information is used to determine which noise filter of a plurality of noise filters to use to filter noise from the color video output image.

39. The method according to claim 38, wherein the edge information comprises an edge mask identifying edge regions in the sub-images.

40. The method according to claim 38, wherein the edge information comprises at least one edge characteristic that is determined respectively for each of a plurality of pixels in the sub-images, wherein the at least one edge characteristic is selected from a group of characteristics consisting of an edge magnitude and an edge direction.

41. The method according to claim 38, wherein enhancing the color video output image comprises performing at least one image enhancement operation responsively to the edge information, wherein the at least one image enhancement operation is selected from a group of operations consisting of noise suppression and edge enhancement.

42. The method according to claim 38, wherein extracting the edge information comprises, at each pixel among a plurality of pixels in at least one sub-image, processing the input pixel values of a first group of the pixels that are horizontally aligned with the pixel and of a second group of the pixels that are vertically aligned with the pixel.

46. The method of claim 38, further comprising:

incorporating the edge information into the input pixel values belonging to the plurality of sub-images.

43. A computer software product, comprising a computer-readable medium in which program instructions are stored, which instructions, when read by the computer, cause the computer to receive, from a mosaic image sensor, a stream of input pixel values belonging to a plurality of sub-images, each sub-image corresponding to light of a respective color that is incident on the mosaic image sensor, to extract edge information from one or more of the sub-images, to process the plurality of the sub-images in order to generate a color video output image, and to enhance the color video output image using the edge information, wherein the edge information is used to determine which noise filter of a plurality of noise filters to use to filter noise from the color video output image.

47. The computer software product of claim 43, wherein the instructions further cause the computer to incorporate the edge information into the input pixel values belonging to the plurality of sub-images.

44. A computer software product, comprising a computer-readable medium in which program instructions are stored, which instructions, when read by the computer, cause the computer to receive, from a mosaic image sensor, a stream of input pixel values belonging to a plurality of input sub-images, each input sub-image corresponding to light of a respective color that is incident on the mosaic image sensor, to digitally filter the input pixel values in each of the input sub-images so as to generate a corresponding plurality of enhanced output sub-images, to extract auxiliary information from the input sub-images prior to digitally filtering the input pixel values, and to process the enhanced output sub-images in order to generate a color video output image while enhancing the color video output image using the auxiliary information.