Four-color mosaic pattern for depth and image capture

• US 20040174446A1
• Filed: 02/28/2003
• Published: 09/09/2004
• Est. Priority Date: 02/28/2003
• Status: Abandoned Application

• Current Assignee: Tahoe Research Limited (f/k/a Learndale Limited) (Vector Capital Corporation)
• Original Assignee: Tahoe Research Limited (f/k/a Learndale Limited) (Vector Capital Corporation)
• Inventors: Acharya, Tinku
• Application Number:
  US10/376,156
• Publication Number:
  US 20040174446A1
• Time in Patent Office: Days
• US Class Current: 348/279
• CPC Class Codes: H04N 23/11 for generating image signals from visible and infrared light wavelengths
• CPC Class Codes: H04N 23/20 for generating image signals from infrared radiation only
• CPC Class Codes: H04N 23/843 Demosaicing, e.g. interpolating colour pixel values
• CPC Class Codes: H04N 25/131 including elements passing infrared wavelengths
• CPC Class Codes: H04N 25/134 based on three different wavelength filter elements
• CPC Class Codes: H04N 25/135 based on four or more different wavelength filter elements
• CPC Class Codes: H04N 5/33 Transforming infrared radiation cameras or camera modules for generating image signals from infrared radiation H04N23/20; circuitry of SSIS for transforming infrared radiation into image signals H04N25/20

Abstract

A sensor for color and depth information capture is disclosed. A filter passes selected wavelengths according to a predetermined pattern to the sensor. The sensor measures light intensities passed by the filter. In one embodiment, the wavelengths passed by the filter correspond to red, green, blue and infrared light. The intensity values can be used for interpolation operations to provide intensity values for areas not captured by the sensor. For example, in an area corresponding to a pixel for which an intensity of red light is captured, interpolation operations using neighboring intensity values can be used to provide an estimation of blue, green and infrared intensities. Red, green and blue intensity values, whether captured or interpolated, are used to provide visible color image information. Infrared intensity values, whether captured or interpolated, are used to provide depth and/or surface texture information.

Claims

1. An apparatus comprising:

a sensor unit to capture wavelength intensity data for a plurality of pixel locations wherein the sensor generates a value corresponding to an intensity of light from a selected range of wavelengths for the pixel locations and further wherein infrared intensity values are generated for a subset of the pixel locations; and

an interpolation unit coupled with the sensor unit to interpolate intensity data to estimate intensity values not generated by the sensor.

2. The apparatus of claim 1 further comprising:

a red pixel buffer coupled with the interpolation unit to store red intensity data;

a green pixel buffer coupled with the interpolation unit to store green intensity data;

a blue pixel buffer coupled with the interpolation unit to store blue intensity data; and

an infrared pixel buffer coupled with the interpolation unit to store infrared intensity data.

3. The apparatus of claim 2 further comprising a signal processing unit coupled to the red pixel data buffer, the green pixel data buffer, the blue pixel data buffer and the infrared pixel data buffer.

4. The apparatus of claim 1 wherein the sensor unit captures intensity data according to a predetermined pattern comprising:

5. The apparatus of claim 4 wherein the red, green, blue and infrared intensity information are captured substantially contemporaneously.

6. The apparatus of claim 4, wherein for a pixel in the predetermined pixel pattern in a location (m,n) where m indicates a row and n indicates a column and X(m,n) is the intensity corresponding to the pixel in the location (m,n), if m and n are both odd integers, the infrared intensity corresponding to the location (m,n) is given by

7. The apparatus of claim 6, wherein if the pixel at location (m,n) is red, the blue intensity corresponding to the location (m,n) is given by

8. The apparatus of claim 4, wherein for a pixel in the predetermined pixel pattern in a location (m,n) where m indicates a row and n indicates a column and X(m,n) is the intensity corresponding to the pixel in the location (m,n), if m is an odd integer and n is an even integer, the infrared intensity corresponding to the location (m,n) is given by

9. The apparatus of claim 8, wherein if the pixel at location (m,n−

1) is red, the blue intensity corresponding to the location (m,n) is given by B=X(m, n+1) and the red intensity corresponding to the location (m,n) is given by R=X(m, n−

1), and if the pixel at location (m,n−

1) is blue, the red intensity corresponding to the location (m,n) is given by R=X(m, n+1) and the blue intensity corresponding to the location (m,n) is given by B=X(m, n−

1).

10. The apparatus of claim 4, wherein for a pixel in the predetermined pixel pattern in a location (m,n) where m indicates a row and n indicates a column and X(m,n) is the intensity corresponding to the pixel in the location (m,n), if m is an even integer and n is an odd integer, the infrared intensity corresponding to the location (m,n) is given by

11. The apparatus of claim 10, wherein if the pixel at location (m−

1,n) is red, the blue intensity corresponding to the location (m,n) is given by B=X(m+1, n) and the red intensity corresponding to the location (m,n) is given by R=X(m−

1, n), and if the pixel at location (m−

1,n) is blue, the red intensity corresponding to the location (m,n) is given by R=X(m+1, n) and the blue intensity corresponding to the location (m,n) is given by B=X(m−

1, n).

12. The apparatus of claim 4, wherein for a pixel in the predetermined pixel pattern in a location (m,n) where m indicates a row and n indicates a column and X(m,n) is the intensity corresponding to the pixel in the location (m,n), if m and n are both even integers, the infrared intensity corresponding to the location (m,n) is given by IR X(m, n), and the green intensity corresponding to the location (m,n) is given by

13. The apparatus of claim 12, wherein if the pixel at location (m−

1,n+1) is red, the blue intensity corresponding to the location (m,n) is given by

14. An apparatus comprising:

a complementary metal-oxide semiconductor (CMOS) sensor to capture an array of pixel data; and

a color filter array (CFA) to pass selected wavelength ranges to respective pixel locations of the CMOS sensor according to a predetermined pattern, wherein the wavelength ranges include at least infrared wavelengths for one or more pixel locations.

15. The apparatus of claim 14 wherein the predetermined pattern comprises:

16. The apparatus of claim 15 wherein the red, green, blue and infrared intensity information is captured substantially contemporaneously.

17. The apparatus of claim 15 further comprising an interpolation unit coupled with the CMOS sensor to interpolate color information to determine multiple color intensities for one or more of the pixel locations.

18. The apparatus of claim 15, wherein for a pixel in the predetermined pixel pattern in a location (m,n) where m indicates a row and n indicates a column and X(m,n) is the intensity corresponding to the pixel in the location (m,n), if m and n are both odd integers, the infrared intensity corresponding to the location (m,n) is given by

19. The apparatus of claim 18, wherein if the pixel at location (m,n) is red, the blue intensity corresponding to the location (m,n) is given by

20. The apparatus of claim 15, wherein for a pixel in the predetermined pixel pattern in a location (m,n) where m indicates a row and n indicates a column and X(m,n) is the intensity corresponding to the pixel in the location (m,n), if m is an odd integer and n is an even integer, the infrared intensity corresponding to the location (m,n) is given by

21. The apparatus of claim 20, wherein if the pixel at location (m,n−

1) is red, the blue intensity corresponding to the location (m,n) is given by B=X(m, n+1) and the red intensity corresponding to the location (m,n) is given by R=X(m, n−

1), and if the pixel at location (m,n−

1) is blue, the red intensity corresponding to the location (m,n) is given by R=X(m, n+1) and the blue intensity corresponding to the location (m,n) is given by B=X(m, n−

1).

22. The apparatus of claim 15, wherein for a pixel in the predetermined pixel pattern in a location (m,n) where m indicates a row and n indicates a column and X(m,n) is the intensity corresponding to the pixel in the location (m,n), if m is an even integer and n is an odd integer, the infrared intensity corresponding to the location (m,n) is given by

23. The apparatus of claim 22, wherein if the pixel at location (m−

1,n) is red, the blue intensity corresponding to the location (m,n) is given by B=X(m+1, n) and the red intensity corresponding to the location (m,n) is given by R=X(m−

1, n), and if the pixel at location (m−

1,n) is blue, the red intensity corresponding to the location (m,n) is given by R=X(m+1, n) and the blue intensity corresponding to the location (m,n) is given by B=X(m−

1, n).

24. The apparatus of claim 15, wherein for a pixel in the predetermined pixel pattern in a location (m,n) where m indicates a row and n indicates a column and X(m,n) is the intensity corresponding to the pixel in the location (m,n), if m and n are both even integers, the infrared intensity corresponding to the location (m,n) is given by IR X(m, n), and the green intensity corresponding to the location (m,n) is given by

25. The apparatus of claim 24, wherein if the pixel at location (m−

1,n+1) is red, the blue intensity corresponding to the location (m,n) is given by

26. A method comprising:

receiving pixel data representing color intensity values for a plurality of pixel locations according to a predetermined pattern, wherein one or more of the color intensity values corresponds to intensity of light having infrared wavelengths;

generating intensity values for multiple color intensities corresponding to a single pixel location by interpolating intensity values corresponding to neighboring pixel locations;

promoting one or more of the generated intensity values to a user-accessible state.

27. The method of claim 26 wherein promoting the one or more of the generated intensity values to a user-accessible state comprises storing the received intensity values and the generated intensity values on a computer-readable storage device.

28. The method of claim 26 wherein promoting the one or more of the generated intensity values to a user-accessible state comprises:

generating an output image with the received intensity values and the generated intensity values; and

displaying the output image on a display device.

29. The method of claim 26 wherein promoting the one or more of the generated intensity values to a user-accessible state comprises:

generating an output image with the received intensity values and the generated intensity values; and

printing the output image.

30. The method of claim 26 wherein the predetermined pattern comprises:

31. The method of claim 30, wherein for a pixel in the predetermined pixel pattern in a location (m,n) where m indicates a row and n indicates a column and X(m,n) is the intensity corresponding to the pixel in the location (m,n), if m and n are both odd integers, the infrared intensity corresponding to the location (m,n) is given by

32. The method of claim 31, wherein if the pixel at location (m,n) is red, the blue intensity corresponding to the location (m,n) is given by

33. The method of claim 30, wherein for a pixel in the predetermined pixel pattern in a location (m,n) where m indicates a row and n indicates a column and X(m,n) is the intensity corresponding to the pixel in the location (m,n), if m is an odd integer and n is an even integer, the infrared intensity corresponding to the location (m,n) is given by

34. The method of claim 33, wherein if the pixel at location (m,n−

1) is red, the blue intensity corresponding to the location (m,n) is given by B=X(m, n+1) and the red intensity corresponding to the location (m,n) is given by R=X(m, n−

1), and if the pixel at location (m,n−

1) is blue, the red intensity corresponding to the location (m,n) is given by R=X(m, n+1) and the blue intensity corresponding to the location (m,n) is given by B=X(m,n−

1).

35. The method of claim 30, wherein for a pixel in the predetermined pixel pattern in a location (m,n) where m indicates a row and n indicates a column and X(m,n) is the intensity corresponding to the pixel in the location (m,n), if m is an even integer and n is an odd integer, the infrared intensity corresponding to the location (m,n) is given by

36. The method of claim 35, wherein if the pixel at location (m−

1,n) is red, the blue intensity corresponding to the location (m,n) is given by B=X(m+1, n) and the red intensity corresponding to the location (m,n) is given by R=X(m−

1, n), and if the pixel at location (m−

1,n) is blue, the red intensity corresponding to the location (m,n) is given by R=X(m+1, n) and the blue intensity corresponding to the location (m,n) is given by B=X(m<sup>−

1</sup>, n).

37. The method of claim 30, wherein for a pixel in the predetermined pixel pattern in a location (m,n) where m indicates a row and n indicates a column and X(m,n) is the intensity corresponding to the pixel in the location (m,n), if m and n are both even integers, the infrared intensity corresponding to the location (m,n) is given by IR=X(m, n), and the green intensity corresponding to the location (m,n) is given by

38. The method of claim 37, wherein if the pixel at location (m−

1,n+1) is red, the blue intensity corresponding to the location (m,n) is given by

39. A sensor that receives pixel data representing color intensity values for a plurality of pixel locations of a scene to be captured according to a predetermined pattern, wherein one or more of the color intensity values corresponds to intensity of light having infrared wavelengths.

40. The sensor of claim 39 wherein the predetermined pattern comprises:

41. The sensor of claim 40 wherein the red intensity values, the green intensity values, the blue intensity values and the infrared intensity values are capture substantially contemporaneously.

42. The sensor of claim 39 further comprising a color filter array (CFA) to pass selected wavelength ranges to respective pixel locations of the sensor according to the predetermined pattern.