Camera system with minimal area monolithic CMOS image sensor

• US 9,462,234 B2
• Filed: 07/26/2013
• Issued: 10/04/2016
• Est. Priority Date: 07/26/2012
• Status: Active Grant

• Current Assignee: Depuy Synthes Products Incorporated (Johnson & Johnson)
• Original Assignee: Depuy Synthes Products Incorporated (Johnson & Johnson)
• Inventors: Blanquart, Laurent, Richardson, John
• Primary Examiner(s): Perungavoor, Sath V
• Assistant Examiner(s): AYNALEM, NATHNAEL B
• Application Number:
  US13/952,550
• Publication Number:
  US 20140160259A1
• Time in Patent Office: 1,166 Days
• Field of Search: 348/65, 348/308, 348/311
• US Class Current: 1/1
• CPC Class Codes: A61B 1/000095 for image enhancement
• CPC Class Codes: A61B 1/051 Details of CCD assembly
• CPC Class Codes: A61B 1/3132 for laparoscopy
• CPC Class Codes: A61B 1/317 for bones or joints, e.g. osteoscopes, arthroscopes
• CPC Class Codes: A61B 5/0077 Devices for viewing the surface of the body, e.g. camera, magnifying lens
• CPC Class Codes: H04N 23/555 for picking-up images in sites, inaccessible due to their dimensions or hazardous conditions, e.g. endoscopes or borescopes
• CPC Class Codes: H04N 23/56 provided with illuminating means
• CPC Class Codes: H04N 23/72 Combination of two or more compensation controls
• CPC Class Codes: H04N 25/75 Circuitry for providing, modifying or processing image signals from the pixel array
• CPC Class Codes: H04N 25/76 Addressed sensors, e.g. MOS or CMOS sensors
• CPC Class Codes: H04N 25/78 Readout circuits for addressed sensors, e.g. output amplifiers or A/D converters
• CPC Class Codes: H04N 7/18 Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources

Abstract

The disclosure extends to methods, systems, and computer program products for digitally imaging with area limited image sensors, such as within a lumen of an endoscope.

Claims

1. An endoscopic device for use in a closed light environment comprising:

an endoscope body comprising;

a hand holding structure;

a lumen attached by a lumen base at a first end of the body;

a tip portion of the lumen opposite of the lumen base of the body;

a lens disposed at the distal most portion of the tip portion;

a CMOS imaging sensor disposed near the tip portion of the lumen, wherein the imaging sensor comprises;

an array of pixels for sensing electromagnetic radiation, wherein the array of pixels comprises active pixels and optical black pixels, wherein each of the optical black pixels are positioned to either a first side or a second side opposite the first side of the active pixels in a vertical direction only to thereby maximize space for the active pixels within the array of pixels;

a transfer port for transmitting data generated by the pixel array;

a digitizer to convert analog pixel data into digital data;

a black clamp circuit for providing offset control for the data generated by the pixel array;

electrical connections providing electrical communication between the imaging sensor and image signal processing circuitry disposed remotely from the imaging sensor.

2. The endoscopic device of claim 1, further comprising a long registry wherein the long registry comprises control parameter entries for controlling exposure times of the pixel array and gain of the pixel array.

3. The endoscopic device of claim 1, further comprising a long registry comprising control parameter entries for controlling incremental offset adjustment of the pixel array or exposure times of the pixel array.

4. The endoscopic device of claim 1, further comprising a long registry comprising control parameter entries for controlling incremental offset adjustment of the pixel array or gain of the pixel array.

5. The endoscopic device of claim 1, further comprising a long registry consisting of control parameter entries for controlling exposure times of the pixel array or gains of the pixel array and incremental offset adjustments.

6. The endoscopic device of claim 1, further comprising a long registry comprising control parameter entries for controlling incremental offset adjustment of the pixel array and exposure times of the pixel array.

7. The endoscopic device of claim 1, further comprising a long registry comprising a plurality of control parameter entries for controlling the operation of a pixel array by adjusting any of:

analog current, analog voltage, pixel timing, vertical timing, sensor reset and sensor initialization.

8. The endoscopic device of claim 1, wherein the active pixels and the optical black pixels calibrate output from said pixel array.

9. The endoscopic device of claim 1, wherein the optical black pixels are organized in a plurality of columns within the pixel array.

10. The endoscopic device of claim 1, wherein the imaging sensor comprises a single digitizer.

11. The endoscopic device of claim 1, wherein the imaging sensor comprises a plurality of digitizers.

12. The endoscopic device of claim 1, wherein the imaging sensor comprises a single transfer port for transmitting pixel data.

13. The endoscopic device of claim 1, wherein the imaging sensor comprises a plurality of transfer ports for transmitting pixel data.

14. The endoscopic device of claim 9, wherein the columns of optical black pixels are sampled a plurality of times to reduce the number of optical black columns.

15. The endoscopic device of claim 9, wherein the columns of optical black pixels are organized on both the first side and the second side of the active pixels of the imaging sensor.

16. The endoscopic device of claim 15, wherein one of the optical black columns on the first side of the imaging sensor is sampled a plurality of times.

17. The endoscopic device of claim 15, wherein one of the optical black columns on the second side of the imaging sensor is sampled a plurality of times.

18. The endoscopic device of claim 15, wherein one of the optical black columns on the first side of the imaging sensor is sampled a plurality of times and one of the optical black columns on the second side of the imaging sensor is sampled a plurality of times.

19. The endoscopic device of claim 15, wherein a plurality of the optical black columns on the first side of the imaging sensor are sampled a plurality of times.

20. The endoscopic device of claim 15, wherein a plurality of the optical black columns on the second side of the imaging sensor are sampled a plurality of times.

21. The endoscopic device of claim 15, wherein a plurality of the optical black columns on the first side of the imaging sensor are sampled a plurality of times and a plurality of the optical black columns on the second side of the imaging sensor are sampled a plurality of times.

22. The endoscopic device of claim 1, wherein the black clamp circuit is located remotely with respect to the imaging sensor.

23. The endoscopic device of claim 22, wherein the device further comprises a command interface to control the black clamp circuit.

24. The endoscopic device of claim 1, wherein the black clamp circuitry senses a voltage generated by a digital to analog converter.

25. The endoscopic device of claim 1, wherein the black clamp circuitry senses a voltage generated by a charge pump.

26. The endoscopic device of claim 1, wherein a portion of the imaging sensor corrections are located remotely with respect to the imaging sensor.

27. The endoscopic device of claim 1, wherein all of the imaging sensor corrections are located remotely with respect to the imaging sensor.

28. The endoscopic device of claim 1, further comprising a plurality of control registers, wherein said control registers are digital latches that are loaded via shift registers.

29. The endoscopic device of claim 28, wherein said shift registers are arbitrary in length.

30. The endoscopic device of claim 28, wherein a majority of the plurality of control registers are loaded using shift registers that include many tens of bits.

31. The endoscopic device of claim 28, wherein a majority of the plurality of control registers are loaded using shift registers that include hundreds of bits.

32. The endoscopic device of claim 28, wherein a majority of the plurality of control registers are loaded using shift registers that include thousands of bits.

33. The endoscopic device of claim 28, wherein said shift registers are loaded using a serial, 2-wire protocol.

34. The endoscopic device of claim 28, wherein one of said shift registers is dedicated to frame-to-frame parameter changes.

35. The endoscopic device of claim 1, wherein the imaging sensor comprises a plurality of pixel arrays, wherein the plurality of pixel arrays are used to create a three-dimensional image.

36. The endoscopic device of claim 1, wherein the imaging sensor further comprises a first substrate comprising the pixel array and a second substrate comprising supporting circuitry for the pixel array, wherein the second substrate comprising the supporting circuitry is located remotely from the first substrate comprising the pixel array.

37. The endoscopic device of claim 36, wherein the first substrate is vertically aligned with respect to the second substrate.

38. The endoscopic device of claim 9, wherein the columns of optical black pixels are sampled a plurality of times to calculate line offsets.

39. The endoscopic device of claim 1, wherein the optical black pixels are organized into a plurality of columns or a plurality of rows adjacent to the active pixels within the pixel array.

40. The endoscopic device of claim 39, wherein a majority of the optical black columns or rows are removed from the pixel array, such that only a minority of the optical black columns or rows are functional.

41. The endoscopic device of claim 1, wherein the optical black pixels are organized into a plurality of columns and do not contain any complete optical black rows adjacent to the active pixels within the pixel array.

42. The endoscopic device of claim 1, wherein the optical black pixels are organized into a plurality of columns adjacent to the active pixels within the pixel array, wherein the plurality of columns are used to calculate a baseline black level for the black clamp circuit and black clamp calculations performed by an image signal processor.

43. The endoscopic device of claim 42, wherein the black clamp calculations are used to control the voltage offset prior to the digitizer and use a plurality of line offsets to determine the overall offset within a whole frame of data using simple exponential smoothing (SES).

44. The endoscopic device of claim 1, wherein the device further comprises two-dimensional frame data stored in memory for compensating for fixed pattern noise generated by the image sensor.

45. The endoscopic device of claim 44, wherein the two-dimensional frame data is derived from dark frame capture.

46. The endoscopic device of claim 45, wherein the dark frame capture is facilitated by not pulsing an emitter.

47. The endoscopic device of claim 45, wherein the pixel offset is computed by simple exponential smoothing for the dark frame capture.

48. A system for digital imaging in an ambient light deficient environment comprising:

a CMOS imaging sensor for sensing electromagnetic radiation, wherein said imaging sensor comprises;

an array of pixels for sensing electromagnetic radiation, wherein the array of pixels comprises active pixels and optical black pixels, wherein each of the optical black pixels are positioned to either a first side or a second side opposite the first side of the active pixels in a vertical direction only to thereby maximize space for the active pixels within the array of pixels;

a transfer port for transmitting data generated by the pixel array;

a digitizer to convert analog pixel samples to digital numbers;

a black clamp circuit for providing offset control for the data generated by the pixel array;

an endoscope for accessing the ambient light deficient environment;

a hand piece attached to said endoscope wherein said endoscope may be maneuvered by manipulation of the hand piece;

a control unit comprising a processor wherein said control unit may be in electrical communication with the imaging sensor;

and a connection cable electrically connecting the hand piece and the control unit.

49. The system of claim 48, wherein the active pixels and the optical black pixels are used for calibrating output from said pixel array.

50. The system of claim 49, wherein the optical black pixels are organized in a plurality of columns within the pixel array.

51. The system of claim 48, further comprising a black clamp control process that may be remotely located from the pixel array.

52. The system of claim 48, further comprising a programmable gate array for processing image data that may be created by the pixel array.

53. The system of claim 48, further comprising optical black pixel columns fewer in number than 50 such that they can be resampled within the operation of the system in order to provide precision.

54. The system of claim 50, wherein the optical black columns are disposed equally at the first side and the second side of the active pixels.

55. The system of claim 50, wherein the optical black columns are disposed unevenly on the first side and the second side of the active pixels.

56. The system of claim 49, further comprising a black clamp control process that averages the data received by optically black pixels and compares the averages to a predetermined target value stored within memory within the system.

57. The system of claim 48, further comprising memory for accumulating a set of measured uncorrected line offsets for an entire frame.

58. The system of claim 48, further comprising a digital to analog conversion circuit for controlling voltage to compensate for dark current.

59. The system of claim 48, further comprising a charge pump circuit for controlling a voltage to compensate for dark current.

60. The system of claim 48, further comprising a plurality of registers.

61. The system of claim 48, further comprising a command register used for top level event 1-bit commands.

62. The system of claim 48, further comprising an image sensor disposed within the endoscope at a distal portion thereof relative to the hand piece.

63. The system of claim 48, further comprising an image sensor disposed within the hand piece.

64. The system of claim 48, further comprising an emitter for emitting pulses of electromagnetic radiation transmitted through fiber optics from the emitter to a tip of the endoscope.

65. The system of claim 48, further comprising a cable comprising fiber optics for transmitting electromagnetic radiation from a light emitter to the endoscope, and electrically conductive wires for providing electronic communication from the control unit to the image sensor.

66. The system of claim 48, further comprising a controller disposed within the control unit and in electrical communication with a light emitter and the imaging sensor.

67. The system of claim 48, further comprising an image sensor disposed within the hand piece, and wherein the image sensor is in electrical communication with a light emitter.

68. The system of claim 48, further comprising an image sensor that is coupled to an emitter.

69. The system of claim 48, further comprising a remotely processed algorithm to correct for line-noise by subtracting line averages measured using a set of optical black pixels in each line.

70. The system of claim 48, further comprising a plurality of transfer ports for transmitting pixel data.

71. The system of claim 48, further comprising a plurality of digitizers.

72. The system of claim 48, further comprising remotely located digitizers relative to the pixel array.