Color selection for sparse color image reconstruction

• US 6,757,012 B1
• Filed: 01/13/2000
• Issued: 06/29/2004
• Est. Priority Date: 01/13/2000
• Status: Expired due to Term

• Current Assignee: Biomorphic Vlsi, Inc.
• Original Assignee: Biomorphic Vlsi, Inc.
• Inventors: Hubina, Thomas Eugene, Reinhart, Craig Charles, Chou, Hung Yu
• Primary Examiner(s): Garber, Wendy R.
• Assistant Examiner(s): ROSENDALE, MATTHEW L
• Application Number:
  US09/482,270
• Time in Patent Office: 1,629 Days
• Field of Search: 348/222.1, 348/272, 348/273, 348/276, 348/277, 348/280, 348/246, 348/252, 348/278, 382/300, 358/525
• US Class Current: 348/222.1
• CPC Class Codes: G06T 3/4015 Image demosaicing, e.g. colour filter arrays [CFA] or Bayer patterns
• CPC Class Codes: H04N 23/843 Demosaicing, e.g. interpolating colour pixel values
• CPC Class Codes: H04N 23/88 for colour balance, e.g. white-balance circuits or colour temperature control
• CPC Class Codes: H04N 25/133 including elements passing panchromatic light, e.g. filters passing white light
• CPC Class Codes: H04N 25/134 based on three different wavelength filter elements
• CPC Class Codes: H04N 25/135 based on four or more different wavelength filter elements

Abstract

Described herein are a system and method for reconstructing a color image from sparse image data collected at an imaging sensor which provides intensity data for exactly one color for each pixel location. A pattern for color selection preferably defines an assignment of color information to each pixel location from neighboring pixels. Thus, the color selection pattern provides an assignment of multicolor image data for each pixel location including color information collected at the pixel location in the imaging sensor and color information selected from neighboring pixels according to the color selection pattern. The color selection pattern may be formulated to minimize the effect of color artifacts in the resulting reconstructed image.

Claims

1. A method of processing data associated with an array of pixels, each pixel corresponding with one of a plurality of distinct spectral regions and being associated with a value representative of an intensity of photoexposure at a pixel location in the array, the method comprising:

capturing a single image in exactly one photoexposure period using the array of pixels;

at each pixel location being associated with a corresponding one of the plurality of spectral regions, selecting at least one neighboring pixel being associated with a spectral region which is distinct from the corresponding spectral region; and

assigning the value representative of the intensity of photoexposure associated with the selected neighboring pixel to the pixel location, the assigned value being an estimate of an intensity of photoexposure in the distinct spectral region at the pixel location.

2. The method of claim 1, the method further comprising selecting the at least one neighboring pixel based upon a directional location of the neighboring pixel relative to the pixel location.

3. The method of claim 1, the method further including:

for each pixel location associated with a corresponding one of the plurality of spectral regions, selecting neighboring pixels being associated with each of the plurality of spectral regions which is distinct from the corresponding spectral region; and

assigning to the pixel location the values representative of the intensity of photoexposure associated with the neighboring pixel for each spectral region which is distinct from the corresponding spectral region, each of the assigned values being representative of an estimate of an intensity of photoexposure in the distinct spectral region at the pixel location,

wherein the values representative of the actual intensity photoexposure in the spectral regions at the pixel locations in the imaging array and the estimates of the intensity of photoexposure in the distinct spectral regions provide data representative of a color image having a value representative of an intensity of photoexposure in each of the spectral regions at each pixel location.

4. The method of claim 1, wherein the imaging array includes alternating odd and even rows of pixels, each of the odd rows having alternating pixels associated with values representative of photoexposure in a first spectral region and a second spectral region, each of the even rows having alternating pixels responsive to the first spectral region and a third spectral region, each of the odd rows being adjacent to an even row and each of the even rows being adjacent to an odd row, the method further including:

at each pixel location associated with the first spectral region in each of the odd rows, assigning a value representative of the intensity of photoexposure in the second spectral region from an adjacent pixel in the odd row and assigning a value representative of the intensity of photoexposure in the third spectral region from a pixel in an adjacent even row;

at each pixel location associated with the first spectral region in each of the even rows, assigning a value representative of the intensity of photoexposure in the third spectral region from an adjacent pixel in the even row and assigning a value representative of the intensity of photoexposure in the second spectral region from a pixel in an adjacent odd row;

at each pixel location associated with the second spectral region in each of the odd rows, assigning a value representative of the intensity of photoexposure in the first spectral region from a pixel in a first adjacent even row and assigning a value representative of the intensity of photoexposure in the third spectral region from a pixel in a second adjacent even row; and

at each pixel location associated with the third spectral region in each of the even rows, assigning a value representative of the intensity of photoexposure in the first spectral region from a pixel in a first adjacent odd row and assigning a value representative of the intensity of photoexposure in the second spectral region from a pixel in a first adjacent odd row.

5. The method of claim 4, wherein the first and second adjacent odd rows are the same row and the first and second even rows are the same row.

6. The method of claim 4, wherein the first and second adjacent odd rows are separated by an even row and the first and second adjacent even rows are separated by an odd row.

7. The method of claim 1, wherein the imaging array includes alternating odd and even rows of pixels, each of the odd rows having alternating pixels associate with values representative of photoexposure in a first spectral region and a second spectral region, each of the even rows having alternating pixels responsive to the first region and a third spectral region, each of the odd rows being adjacent to an even row and each of the even rows being adjacent to an odd row, the method further including:

at each odd row, at each pixel location associated with the first spectral region assigning a value representative of the intensity of photoexposure in the second spectral region from an adjacent pixel in the odd row and assigning a value representative of the intensity of photoexposure in the third spectral region from a pixel in a first adjacent even row, and at each pixel location associated with the second spectral region assigning a value representative of the intensity of photoexposure in the first spectral region from an adjacent pixel in the odd row and assigning a value representative of the intensity of photoexposure in the third spectral region from a pixel in a second adjacent even row; and

at each even row, at each pixel location associated with the first spectral region assigning a value representative of the intensity of photoexposure in the third spectral region from an adjacent pixel in the even row and assigning a value representative of the intensity of photoexposure in the second spectral region from a pixel in a first adjacent odd row, and at each pixel location associated with the third spectral region assigning a value representative of the intensity of photoexposure in the first spectral region from an adjacent pixel in the even row and assigning a value representative of the intensity of photoexposure in the second spectral region from a pixel in a second adjacent odd row.

8. The method of claim 7, wherein the first and second adjacent odd rows are the same row and the first and second even rows are the same row.

9. The method of claim 7, wherein the first and second adjacent odd rows are separated by an even row and the first and second adjacent even rows are separated by an odd row.

10. The method of claim 1, wherein the array includes a plurality of odd and even rows of pixels, each row having a plurality of uniformly spaced pixels separated by boundaries, each of the odd rows being adjacent to an even row and each of the even rows being adjacent to an odd row, wherein each of the pixels in the odd rows is centered at about a boundary separating adjacent pixels in an even row and each of the pixels in the even rows is centered about a boundary separating adjacent pixels in an odd row.

11. In a camera, the camera having an imaging array of pixels and an optical system for projecting an image onto the imaging array, each pixel in the imaging array being at a location in the array and corresponding with one of a plurality of distinct spectral regions, the imaging array outputting a value representative of an intensity of photoexposure at a pixel location in the array, the improvement comprising:

logic for capturing a single image in exactly one photoexposure period using the array of pixels;

logic for selecting, at each pixel location being associated with a corresponding one of the plurality of spectral regions, at least one neighboring pixel being associated with a spectral region which is distinct from the corresponding spectral region; and

logic for assigning the value representative of the intensity of photoexposure associated with the selected neighboring pixel to the pixel location, the assigned value being an estimate of an intensity of photoexposure in the distinct spectral region at the pixel location.

12. The camera of claim 11, the camera further comprising logic for selecting the at least one neighboring pixel based upon a directional location of the nearest pixel relative to the pixel location.

13. The camera of claim 11, the camera further including:

for each pixel location associated with a corresponding one of the plurality of spectral regions, logic for selecting neighboring pixels being associated with each of the plurality of spectral regions which is distinct from the corresponding spectral region; and

logic for assigning to the pixel location the values representative of the intensity of photoexposure associated the neighboring pixels for each spectral region which is distinct from the corresponding spectral region, each of the assigned values being representative of an estimate of an intensity of photoexposure in the distinct spectral region at the pixel location,

wherein the values representative of the actual intensity photoexposure in the spectral regions at the pixel locations in the imaging array and the estimates of the intensity of photoexposure in the distinct spectral regions provide data representative of a color image having a value representative of an intensity of photoexposure in each of the spectral regions at each pixel location.

14. The camera of claim 11, wherein the imaging array includes alternating odd and even rows of pixels, each of the odd rows having alternating pixels associated with values representative of photoexposure in a first spectral region and a second spectral region, each of the even rows having alternating pixels responsive to the first spectral region and a third spectral region, each of the odd rows being adjacent to an even row and each of the even rows being adjacent to an odd row, the improvement further including:

for each pixel location associated with the first spectral region in each of the odd rows, logic for assigning a value representative of the intensity of photoexposure in the second spectral region from an adjacent pixel in the odd row and assigning a value representative of the intensity of photoexposure in the third spectral region from a pixel in an adjacent even row;

for each pixel location associated with the first spectral region in each of the even rows, logic for assigning a value representative of the intensity of photoexposure in the third spectral region from an adjacent pixel in the even row and assigning a value representative of the intensity of photoexposure in the second spectral region from a pixel in an adjacent odd row;

for each pixel location associated with the second spectral region in each of the odd rows, logic for assigning a value representative of the intensity of photoexposure in the first spectral region from a pixel in a first adjacent even row and assigning a value representative of the intensity of photoexposure in the third spectral region from a pixel in a second adjacent even row; and

for each pixel location associated with the third spectral region in each of the even rows, logic for assigning a value representative of the intensity of photoexposure in the first spectral region from a pixel in a first adjacent odd row and assigning a value representative of the intensity of photoexposure in the second spectral region from a pixel in a first adjacent odd row.

15. The camera of claim 14, wherein the first and second adjacent odd rows are the same row and the first and second even rows are the same row.

16. The camera of claim 14, wherein the first and second adjacent odd rows are separated by an even row and the first and second adjacent even rows are separated by an odd row.

17. The camera of claim 11, wherein the imaging array includes alternating odd and even rows of pixels, each of the odd rows having alternating pixels associate with values representative of photoexposure in a first spectral region and a second spectral region, each of the even rows having alternating pixels responsive to the first region and a third spectral region, each of the odd rows being adjacent to an even row and each of the even rows being adjacent to an odd row, the method further including:

for each odd row, for each pixel location associated with the first spectral region, logic for assigning a value representative of the intensity of photoexposure in the second spectral region from an adjacent pixel in the odd row and assigning a value representative of the intensity of photoexposure in the third spectral region from a pixel in a first adjacent even row, and for each pixel location associated with the second spectral region, logic for assigning a value representative of the intensity of photoexposure in the first spectral region from an adjacent pixel in the odd row and assigning a value representative of the intensity of photoexposure in the third spectral region from a pixel in a second adjacent even row; and

for each even row, for each pixel location associated with the first spectral region, logic for assigning a value representative of the intensity of photoexposure in the third spectral region from an adjacent pixel in the even row and assigning a value representative of the intensity of photoexposure in the second spectral region from a pixel in a first adjacent odd row, and for each pixel location associated with the third spectral region logic for assigning a value representative of the intensity of photoexposure in the first spectral region from an adjacent pixel in the even row and assigning a value representative of the intensity of photoexposure in the second spectral region from a pixel in a second adjacent odd row.

18. The camera of claim 17, wherein the first and second adjacent odd rows are the same row and the first and second even rows are the same row.

19. The camera of claim 17, wherein the first and second adjacent odd rows are separated by an even row and the first and second adjacent even rows are separated by an odd row.

20. The camera of claim 11, wherein the array includes a plurality of odd and even rows of pixels, each row having a plurality of uniformly spaced pixels separated by boundaries, each of the odd rows being adjacent to an even row and each of the even rows being adjacent to an odd row, wherein each of the pixels in the odd rows is centered at about a boundary separating adjacent pixels in an even row and each of the pixels in the even rows is centered about a boundary separating adjacent pixels in an odd row.

21. A computer readable medium for use in conjunction with an imaging array of pixels, each pixel being at a location in the array and corresponding with one of a plurality of distinct spectral regions, the imaging array outputting a value representative of an intensity of photoexposure at a pixel location in the array, the computer readable medium including computer readable instructions encoded thereon for performing the following:

capturing a single image in exactly one photoexposure period using the array of pixels;

at each pixel location being associated with a corresponding one of the plurality of spectral regions, selecting at least one neighboring pixel being associated with a spectral region which is distinct from the corresponding spectral pixel region; and

assigning the value representative of the intensity of photoexposure associated with the selected neighboring pixel to the pixel location, the assigned value being an estimate of an intensity of photoexposure in the distinct spectral region at the pixel location.

22. The computer readable medium of claim 21, the computer readable medium further including computer readable instructions encoded thereon for selecting the at least one neighboring pixel based upon a directional location of the neighboring pixel relative to the pixel location.

23. The computer readable medium of claim 21, the computer readable medium further including computer readable instructions encoded thereon for performing the following:

for each pixel location associated with a corresponding one of the plurality of spectral regions, selecting neighboring pixels being associated with each of the plurality of spectral regions which is distinct from the corresponding spectral region; and

assigning to the pixel location the values representative of the intensity of photoexposure associated with the neighboring pixels for each spectral region which is distinct from the corresponding spectral region, each of the assigned values being representative of an estimate of an intensity of photoexposure in the distinct spectral region at the pixel location,

wherein the values representative of the actual intensity photoexposure in the spectral regions at the pixel locations in the imaging array and the estimates of the intensity of photoexposure in the distinct spectral regions provide data representative of a color image having a value representative of an intensity of photoexposure in each of the spectral regions at each pixel location.

24. The computer readable medium of claim 21, wherein the imaging array includes alternating odd and even rows of pixels, each of the odd rows having alternating pixels associated with values representative of photoexposure in a first spectral region and a second spectral region, each of the even rows having alternating pixels responsive to the first spectral region and a third spectral region, each of the odd rows being adjacent to an even row and each of the even rows being adjacent to an odd row, the computer readable medium further including computer readable instructions encoded thereon for performing the following:

at each pixel location associated with the first spectral region in each of the odd rows, assigning a value representative of the intensity of photoexposure in the second spectral region from an adjacent pixel in the odd row and assigning a value representative of the intensity of photoexposure in the third spectral region from a pixel in an adjacent even row;

at each pixel location associated with the first spectral region in each of the even rows, assigning a value representative of the intensity of photoexposure in the third spectral region from an adjacent pixel in the even row and assigning a value representative of the intensity of photoexposure in the second spectral region from a pixel in an adjacent odd row;

at each pixel location associated with the second spectral region in each of the odd rows, assigning a value representative of the intensity of photoexposure in the first spectral region from a pixel in a first adjacent even row and assigning a value representative of the intensity of photoexposure in the third spectral region from a pixel in a second adjacent even row; and

at each pixel location associated with the third spectral region in each of the even rows, assigning a value representative of the intensity of photoexposure in the first spectral region from a pixel in a first adjacent odd row and assigning a value representative of the intensity of photoexposure in the second spectral region from a pixel in a first adjacent odd row.

25. The computer readable medium of claim 24, wherein the first and second adjacent odd rows are the same row and the first and second even rows are the same row.

26. The computer readable medium of claim 24, wherein the first and second adjacent odd rows are separated by an even row and the first and second adjacent even rows are separated by an odd row.

27. The computer readable medium of claim 21, wherein the imaging array includes alternating odd and even rows of pixels, each of the odd rows having alternating pixels associate with values representative of photoexposure in a first spectral region and a second spectral region, each of the even rows having alternating pixels responsive to the first region and a third spectral region, each of the odd rows being adjacent to an even row and each of the even rows being adjacent to an odd row, the computer readable medium further including computer readable instructions encoded thereon for performing the following:

at each odd row, at each pixel location associated with the first spectral region assigning a value representative of the intensity of photoexposure in the second spectral region from an adjacent pixel in the odd row and assigning a value representative of the intensity of photoexposure in the third spectral region from a pixel in a first adjacent even row, and at each pixel location associated with the second spectral region assigning a value representative of the intensity of photoexposure in the first spectral region from an adjacent pixel in the odd row and assigning a value representative of the intensity of photoexposure in the third spectral region from a pixel in a second adjacent even row; and

at each even row, at each pixel location associated with the first spectral region assigning a value representative of the intensity of photoexposure in the third spectral region from an adjacent pixel in the even row and assigning a value representative of the intensity of photoexposure in the second spectral region from a pixel in a first adjacent odd row, and at each pixel location associated with the third spectral region assigning a value representative of the intensity of photoexposure in the first spectral region from an adjacent pixel in the even row and assigning a value representative of the intensity of photoexposure in the second spectral region from a pixel in a second adjacent odd row.

28. The computer readable medium of claim 27, wherein the first and second adjacent odd rows are the same row and the first and second even rows are the same row.

29. The computer readable medium of claim 27, wherein the first and second adjacent odd rows are separated by an even row and the first and second adjacent even rows are separated by an odd row.

30. The computer readable medium of claim 21, wherein the array includes a plurality of odd and even rows of pixels, each row having a plurality of uniformly spaced pixels separated by boundaries, each of the odd rows being adjacent to an even row and each of the even rows being adjacent to an odd row, wherein each of the pixels in the odd rows is centered at about a boundary separating adjacent pixels in an even row and each of the pixels in the even rows is centered about a boundary separating adjacent pixels in an odd row.

31. An image processor for use in conjunction with an imaging array of pixels, each pixel being at a location in the array and corresponding with one of a plurality of spectral regions, the imaging array outputting a value representative of an intensity of photoexposure at a pixel location in the array, the image processor comprising:

logic for capturing a single image in exactly one photoexposure period using the array of pixels;

logic for selecting, at each pixel location being associated with a corresponding one of the plurality of spectral regions, at least one neighboring pixel being associated with a spectral region which is distinct from the corresponding spectral pixel region; and

logic for assigning the value representative of the intensity of photoexposure associated with the selected neighboring pixel to the pixel location, the assigned value being an estimate of an intensity of photoexposure in the distinct spectral region at the pixel location.

32. The image processor of claim 31, the image processor further comprising logic for selecting the at least one neighboring pixel based upon a directional location of the neighboring pixel relative to the pixel location.

33. The image processor of claim 31, the image processor further including:

for each pixel location associated with a corresponding one of the plurality of spectral regions, logic for selecting neighboring pixels being associated with each of the plurality of spectral regions which is distinct from the corresponding spectral region; and

logic for assigning to the pixel location the values representative of the intensity of photoexposure associated with neighboring pixels for each spectral region which is distinct from the corresponding spectral region, each of the assigned values being representative of an estimate of an intensity of photoexposure in the distinct spectral region at the pixel location,

wherein the values representative of the actual intensity photoexposure in the spectral regions at the pixel locations in the array and the estimates of the intensity of photoexposure in the spectral regions provide data representative of a color image having a value representative of an intensity of photoexposure in each of the spectral regions at each pixel location.

34. The image processor of claim 31, wherein the imaging array includes alternating odd and even rows of pixels, each of the odd rows having alternating pixels associated with values representative of photoexposure in a first spectral region and a second spectral region, each of the even rows having alternating pixels responsive to the first spectral region and a third spectral region, each of the odd rows being adjacent to an even row and each of the even rows being adjacent to an odd row, the image processor further including:

for each pixel location associated with the first spectral region in each of the odd rows, logic for assigning a value representative of the intensity of photoexposure in the second spectral region from an adjacent pixel in the odd row and assigning a value representative of the intensity of photoexposure in the third spectral region from a pixel in an adjacent even row;

for each pixel location associated with the first spectral region in each of the even rows, logic for assigning a value representative of the intensity of photoexposure in the third spectral region from an adjacent pixel in the even row and assigning a value representative of the intensity of photoexposure in the second spectral region from a pixel in an adjacent odd row;

for each pixel location associated with the second spectral region in each of the odd rows, logic for assigning a value representative of the intensity of photoexposure in the first spectral region from a pixel in a first adjacent even row and assigning a value representative of the intensity of photoexposure in the third spectral region from a pixel in a second adjacent even row; and

for each pixel location associated with the third spectral region in each of the even rows, logic for assigning a value representative of the intensity of photoexposure in the first spectral region from a pixel in a first adjacent odd row and assigning a value representative of the intensity of photoexposure in the second spectral region from a pixel in a first adjacent odd row.

35. The image processor of claim 34, wherein the first and second adjacent odd rows are the same row and the first and second even rows are the same row.

36. The image processor of claim 34, wherein the first and second adjacent odd rows are separated by an even row and the first and second adjacent even rows are separated by an odd row.

37. The image processor of claim 31, wherein the imaging array includes alternating odd and even rows of pixels, each of the odd rows having alternating pixels associate with values representative of photoexposure in a first spectral region and a second spectral region, each of the even rows having alternating pixels responsive to the first region and a third spectral region, each of the odd rows being adjacent to an even row and each of the even rows being adjacent to an odd row, the method further including:

for each odd row, for each pixel location associated with the first spectral region, logic for assigning a value representative of the intensity of photoexposure in the second spectral region from an adjacent pixel in the odd row and assigning a value representative of the intensity of photoexposure in the third spectral region from a pixel in a first adjacent even row, and for each pixel location associated with the second spectral region, logic for assigning a value representative of the intensity of photoexposure in the first spectral region from an adjacent pixel in the odd row and assigning a value representative of the intensity of photoexposure in the third spectral region from a pixel in a second adjacent even row; and

for each even row, for each pixel location associated with the first spectral region, logic for assigning a value representative of the intensity of photoexposure in the third spectral region from an adjacent pixel in the even row and assigning a value representative of the intensity of photoexposure in the second spectral region from a pixel in a first adjacent odd row, and for each pixel location associated with the third spectral region logic for assigning a value representative of the intensity of photoexposure in the first spectral region from an adjacent pixel in the even row and assigning a value representative of the intensity of photoexposure in the second spectral region from a pixel in a second adjacent odd row.

38. The image processor of claim 37, wherein the first and second adjacent odd rows are the same row and the first and second even rows are the same row.

39. The image processor of claim 37, wherein the first and second adjacent odd rows are separated by an even row and the first and second adjacent even rows are separated by an odd row.

40. The image processor of claim 31, wherein the array includes a plurality of odd and even rows of pixels, each row having a plurality of uniformly spaced pixels separated by boundaries, each of the odd rows being adjacent to an even row and each of the even rows being adjacent to an odd row, wherein each of the pixels in the odd rows is centered at about a boundary separating adjacent pixels in an even row and each of the pixels in the even rows is centered about a boundary separating adjacent pixels in an odd row.