Devices, methods, and systems for expanded-field-of-view image and video capture

• US 9,609,190 B2
• Filed: 10/30/2013
• Issued: 03/28/2017
• Est. Priority Date: 10/31/2012
• Status: Active Grant

• Current Assignee: InVisage Technologies, Inc. (Apple Inc.)
• Original Assignee: InVisage Technologies, Inc. (Apple Inc.)
• Inventors: Lee, Jess Jan Young, Hepp, Michael, Sargent, Edward Hartley
• Primary Examiner(s): Nguyen, Luong T
• Application Number:
  US14/067,745
• Publication Number:
  US 20140118584A1
• Time in Patent Office: 1,245 Days
• US Class Current: 1/1
• CPC Class Codes: G03B 37/04 with cameras or projectors providing touching or overlapping fields of view
• CPC Class Codes: H01L 27/14605 Structural or functional details relating to the position of the pixel elements, e.g. smaller pixel elements in the center of the imager compared to pixel elements at the periphery
• CPC Class Codes: H01L 27/14612 involving a transistor
• CPC Class Codes: H01L 27/14623 Optical shielding
• CPC Class Codes: H01L 27/1464 Back illuminated imager structures
• CPC Class Codes: H01L 27/14643 Photodiode arrays; MOS imagers
• CPC Class Codes: H04N 2209/045 using mosaic colour filter
• CPC Class Codes: H04N 23/45 for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
• CPC Class Codes: H04N 23/69 Control of means for changing angle of the field of view, e.g. optical zoom objectives or electronic zooming
• CPC Class Codes: H04N 25/134 based on three different wavelength filter elements
• CPC Class Codes: H04N 25/53 Control of the integration time
• CPC Class Codes: H04N 25/75 Circuitry for providing, modifying or processing image signals from the pixel array
• CPC Class Codes: H04N 25/76 Addressed sensors, e.g. MOS or CMOS sensors
• CPC Class Codes: H04N 25/77 Pixel circuitry, e.g. memories, A/D converters, pixel amplifiers, shared circuits or shared components

Abstract

In various example embodiments, an imaging system and method are provided. In an embodiment, the system comprises a first image sensor array, a first optical system to project a first image on the first image sensor array, the first optical system having a first zoom level. A second optical system is to project a second image on a second image sensor array, the second optical system having a second zoom level. The second image sensor array and the second optical system are pointed in the same direction as the first image sensor array and the first optical system. The second zoom level is greater than the first zoom level such that the second image projected onto the second image sensor array is a zoomed in on portion of the first image projected on the first image sensor array. The first image sensor array includes at least four megapixels and the second image sensor array includes one-half or less than the number of pixels in the first image sensor array.

Claims

1. An imaging system, comprising:

a first image sensor array;

a first optical system configured to project a first image on the first image sensor array, the first optical system having a first zoom level covering a first field-of-view;

a second image sensor array;

a second optical system configured to project a second image on the second image sensor array, the second optical system having a second zoom level covering a second field-of-view, the imaging system to combine information from the first field-of-view and the second field-of-view to produce a digital image that substantially spans the first field-of-view and the second field-of-view, the digital image having information content covering a third field-of-view; and

first pixel circuitry for reading image data from the first image sensor array and second pixel circuitry for reading image data from the second image sensor array and an electronic global shutter configured to stop charge integration between the first image sensor array and the first pixel circuitry and between the second image sensor array and the second pixel circuitry at substantially the same time;

wherein the second image sensor array and the second optical system are pointed generally in the same direction as the first image sensor array and the first optical system;

wherein the second zoom level is greater than the first zoom level such that the second image projected onto the second image sensor array is a zoomed in portion of the first image projected on the first image sensor array;

wherein the first image sensor array includes at least four megapixels; and

wherein the second image sensor array includes one-half or less than the number of pixels in the first image sensor array; and

wherein the imaging system is configured to be selectable to operate in one of at least three modes;

in a first mode, the first image is utilized and is the sole source of image data for presentation to the user;

in a second mode, the second image is utilized and is the sole source of image data for presentation to the user; and

in a third mode, both the first image and the second image are to be conveyed to a graphical processing unit, the graphical processing unit to use the first image and the second image to generate a third image that combines information containing the first image and the second image, the third image having a higher-resolution region near a center.

2. The imaging system of claim 1, wherein the first image sensor array includes at least six megapixels.

3. The imaging system of claim 1, wherein the first image sensor array includes at least eight megapixels.

4. The imaging system of claim 1, wherein the second image sensor array includes four megapixels or less.

5. The imaging system of claim 1, wherein the second image sensor array includes two megapixels or less.

6. The imaging system of claim 1, wherein the second image sensor array includes one megapixel or less.

7. The imaging system of claim 1, wherein the first image sensor array includes a first array of first pixel regions and the second image sensor array includes a second array of second pixel regions, wherein each of the first pixel regions is larger than each of the second pixel regions.

8. The imaging system of claim 7, wherein each of the first pixel regions has a lateral distance across a first pixel region of less than 2.5 microns.

9. The imaging system of claim 7, wherein each of the first pixel regions has an area of less than about 2.5 microns squared.

10. The imaging system of claim 7, wherein each of the first pixel regions has a lateral distance across a first pixel region of less than 2 microns.

11. The imaging system of claim 7, wherein each of the first pixel regions has an area of less than about 2 microns squared.

12. The imaging system of claim 7, wherein each of the first pixel regions has a lateral distance across a first pixel region of less than 1.5 microns.

13. The imaging system of claim 7, wherein each of the first pixel regions has an area of less than about 1.5 microns squared.

14. The imaging system of claim 1, further comprising:

a third image sensor array; and

a third optical system configured to project a third image on the third image sensor array, the third optical system having a third zoom level, wherein the third image sensor array and the third optical system are pointed generally in the same direction as the first image sensor array and the first optical system.

15. The imaging system of claim 14, wherein the third image sensor array includes the same number of pixels as the second image sensor array.

16. The imaging system of claim 1, wherein the first image sensor array and the second image sensor array are formed on a same substrate.

17. The imaging system of claim 1, further comprising a user interface control for selecting a zoom level and circuitry for reading out images from the first image sensor array and the second image sensor array and generating an output image based on the selected zoom level.

18. The imaging system of claim 1, wherein the first image is selected for output when the first zoom level is selected.

19. The imaging system of claim 1, wherein the second image is used to enhance the first image for output when the first zoom level is selected.

20. The imaging system of claim 1, wherein the second image is selected for output when the first zoom level is selected and the first image is used to enhance the second image.

21. The imaging system of claim 1, wherein the imaging system is part of a camera device and wherein a user control may be selected to output both the first image and the second image from the camera device.

22. The imaging system of claim 1, wherein the imaging system is part of a camera device and wherein a user control may be selected to output the first image, the second image and the third image from the camera device.

23. The imaging system of claim 1, wherein the electronic global shutter is configured to stop a period of the charge integration for each pixel region in the first image sensor array and the second image sensor array within one millisecond of one another.

24. The imaging system of claim 14, further comprising third pixel circuitry for reading image data from the third image sensor array, wherein the electronic global shutter is configured to stop charge integration between the third image sensor array and a third pixel circuitry at substantially the same time as the first image sensor array and the second image sensor array.

25. An imaging system, comprising:

a primary image sensor array;

a primary optical system configured to project a primary image on the primary image sensor array, the primary optical system having a first zoom level covering a first field-of-view;

a plurality of secondary image sensor arrays;

a secondary optical system for each of the secondary image sensor arrays, wherein each secondary optical system is configured to project a secondary image on a respective one of the secondary image sensor arrays, each of the secondary optical systems having a respective zoom level different than the first zoom level and each covering a secondary field-of-view, respectively, the imaging system to combine information from the first field-of-view and at least one of the secondary fields-of-view to produce a digital image that substantially spans the first field-of-view and the at least one secondary field-of-view to produce the digital image with information content covering a third field-of-view;

wherein each of the secondary image sensor arrays and each of the secondary optical systems are pointed generally in the same direction as the primary image sensor array and the primary optical system;

wherein the primary image sensor array is larger than each of the secondary image sensor arrays; and

wherein the imaging system is configured to be selectable to operate in one of at least three modes;

in a first mode, the primary image is utilized and is the sole source of image data for presentation to the user;

in a second mode, at least one secondary image is utilized and is the sole source of image data for presentation to the user; and

in a third mode, both the primary image and the at least one secondary image are to be conveyed to a graphical processing unit, the graphical processing unit to use the primary image and the at least one secondary image to generate a third image that combines information containing the primary image and the at least one secondary image, the third image having a higher-resolution region near a center.

26. The imaging system of claim 25, further comprising a control circuit to output a primary image output based on the first image projected onto the primary image sensor array during a first mode of operation, wherein the primary image output is not generated based on any of the secondary images projected onto the secondary image sensor arrays.

27. The imaging system of claim 25, further comprising a control circuit to output a primary image output based on the first image projected onto the primary image sensor array during a first mode of operation, wherein the primary image output is enhanced based on at least one of the secondary images.

28. The imaging system of claim 25, wherein a control circuit is configured to output a zoomed image having a zoom level greater than the first zoom level during a second mode of operation, wherein the zoomed image is based on at least one of the secondary images and the primary image.

29. The imaging system of claim 25, wherein each of the secondary optical systems has a different zoom level from one another.

30. The imaging system of claim 25, wherein at least some of the zoom levels of the plurality of secondary optical systems are greater than the first zoom level.

31. The imaging system of claim 25, wherein at least some of the zoom levels of the plurality of secondary optical systems are less than the first zoom level.

32. The imaging system of claim 25, wherein the plurality of secondary optical systems include at least two respective secondary optical systems having a zoom level greater than the first zoom level and at least two respective secondary optical systems having a zoom level less than the first zoom level.

33. The imaging system of claim 25, wherein the imaging system is part of a camera device, further comprising control circuitry configured to output a plurality of images during a mode of operation, wherein the plurality of images includes at least one image corresponding to each of the image sensor arrays.

34. The imaging system of claim 25, wherein the imaging system is part of a camera device, further comprising control circuitry configured to output an image with super resolution generated from the first image and at least one of the secondary images.

35. The imaging system of claim 25, further comprising global electronic shutter circuitry configured to control an imaging period for the primary image sensor array and each of the secondary image sensor arrays to be substantially the same.

36. The imaging system of claim 25, further comprising global electronic shutter circuitry configured to control an integration period for the primary image sensor array and each of the secondary image sensor arrays to be substantially the same.

37. An imaging system, comprising:

a semiconductor substrate;

a plurality of image sensor arrays, including a primary image sensor array and a plurality of secondary image sensor arrays;

a plurality of optical systems, including at least one optical system for each image sensor array, a first of the plurality of optical systems being configured to project a primary image, covering a first field-of-view, onto the primary image sensor array, each of a remaining plurality of optical systems covering a secondary field-of-view, respectively, the imaging system to combine information from the first field-of-view and at least one of the secondary fields-of-view to produce a digital image that substantially spans the first field-of-view and the at least one secondary field-of-view to produce the digital image with information content covering a third field-of-view;

wherein each of the plurality of optical systems has a different zoom level;

each of the image sensor arrays including pixel circuitry formed on the semiconductor substrate for reading an image signal from the respective image sensor array, wherein the pixel circuitry for each of the image sensor arrays includes switching circuitry;

a control circuit operatively coupled to the switching circuitry of each of the image sensor arrays; and

wherein the imaging system is configured to be selectable to operate in one of at least three modes;

in a first mode, the primary image is utilized and is the sole source of image data for presentation to the user;

in a second mode, at least one secondary image from at least one of the secondary fields-of-view is utilized and is the sole source of image data for presentation to the user; and

in a third mode, both the primary image and the at least one secondary image are to be conveyed to a graphical processing unit, the graphical processing unit to use the primary image and the at least one secondary image to generate a third image that combines information containing the primary image and the at least one secondary image, the third image having a higher-resolution region near a center.

38. The image sensor of claim 37, wherein the control circuit is configured to switch the switching circuitry at substantially the same time to provide a global electronic shutter for each of the image sensor arrays.

39. The image sensor of claim 37, wherein the control circuit is configured to switch the switching circuitry to end an integration period for each of the image sensor arrays at substantially the same time.

40. The imaging system of claim 37, wherein the number of secondary image sensor arrays is at least four.

41. The imaging system of claim 37, wherein the optical systems for the secondary image sensor arrays include at least two respective optical systems having a zoom level greater than the zoom level of the primary image sensor array and at least two respective optical systems having a zoom level less than the primary image sensor array.

42. The imaging system of claim 37, wherein the primary image sensor array is larger than each of the secondary image sensor arrays.

43. The imaging system of claim 37, wherein the pixel circuitry for each image sensor array includes a plurality of pixel circuits formed on the semiconductor substrate corresponding to pixel regions of the respective image sensor array, each pixel circuit comprising a charge store and a switching element between the charge store and the respective pixel region.

44. The imaging system of claim 43, wherein the switching circuitry of each image sensor array is operatively coupled to each of the switching elements of the pixel circuits in the image sensor array, such that an integration period for each of the pixel circuits is configured to end at substantially the same time.

45. The imaging system of claim 37, wherein each pixel region comprises optically sensitive material over a pixel circuit for a respective pixel region.

46. The imaging system of claim 37, wherein each pixel region comprises an optically sensitive region on a first side of the semiconductor substrate, wherein a pixel circuit includes read out circuitry for the respective pixel region on the second side of the semiconductor substrate.

47. The imaging system of claim 43, wherein the charge store comprises a pinned diode.

48. The imaging system of claim 43, wherein the switching element is a transistor.

49. The imaging system of claim 43, wherein the switching element is a diode.

50. The imaging system of claim 43, wherein the switching element is a parasitic diode.

51. The imaging system of claim 43, wherein the control circuitry is configured to switch the switching element of each of the pixel circuits at substantially the same time.

52. The imaging system of claim 45, wherein each pixel region comprises a respective first electrode and a respective second electrode, wherein the optically sensitive material of the respective pixel region is positioned between the respective first electrode and the respective second electrode of the respective pixel region.

53. The imaging system of claim 43, wherein each pixel circuit is configured to transfer charge between a first electrode to the charge store when the switching element of the respective pixel region is in a first state and to block a transfer of charge from the first electrode to the charge store when the switching element of the respective pixel region is in a second state.

54. The imaging system of claim 43, wherein the control circuitry is configured to switch the switching element of each of the pixel circuits from a first state to a second state at substantially the same time for each of the pixel circuits after an integration period of time.

55. The imaging system of claim 45, wherein each pixel circuit further comprises reset circuitry configured to reset a voltage difference across the optically sensitive material while a switching element is in a second state.

56. The imaging system of claim 43, wherein each pixel circuit further comprises a read out circuit formed on one side of the semiconductor substrate below a plurality of pixel regions.

57. The imaging system of claim 45, wherein the optically sensitive material is a continuous film of nanocrystal material.

58. The imaging system of claim 43, further comprising analog to digital conversion circuitry to generate digital pixel values from a signal read out of the pixel circuits for each of the image sensor arrays and a processor configured to process the pixel values corresponding to at least two of the image sensor arrays in the first mode of operation to generate an output image.

59. The imaging system of claim 58, wherein the output image has a zoom level between the zoom level of the primary image sensor array and at least one of the secondary image sensor arrays used to generate the output image.

60. The imaging system of claim 37, further comprising a processor configured to generate an output image during a selected mode of operation based on pixel values corresponding to the primary image sensor array without modification based on images projected onto the secondary image sensor arrays.

61. The imaging system of claim 37, wherein the primary image sensor array includes a number of pixels corresponding to the full resolution of the imaging system and wherein each of the secondary image sensor arrays includes a number of pixels less than the full resolution of the imaging system.

62. The imaging system of claim 61, wherein an image corresponding to the primary image sensor array is output when a first zoom level is selected and an image generated from the primary image sensor array and at least one of the secondary image sensor arrays is output when a different zoom level is selected.