Wide dynamic range using monochromatic sensor

• US 10,165,195 B2
• Filed: 04/10/2017
• Issued: 12/25/2018
• Est. Priority Date: 07/26/2012
• Status: Active Grant

• Current Assignee: Depuy Synthes Products Incorporated (Johnson & Johnson)
• Original Assignee: Depuy Synthes Products Incorporated (Johnson & Johnson)
• Inventors: Blanquart, Laurent, Richardson, John
• Primary Examiner(s): Aynalem, Nathnael
• Application Number:
  US15/483,934
• Publication Number:
  US 20170214841A1
• Time in Patent Office: 624 Days
• Field of Search: 348 68, 34820899, 348241, 348294, 348317, 348362
• CPC Class Codes: A61B 1/045 Control thereof
• CPC Class Codes: A61B 1/0605 for spatially modulated illumination
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 2209/042 having a single pick-up sensor
• CPC Class Codes: H04N 23/10 for generating image signals from different wavelengths
• CPC Class Codes: H04N 23/12 with one sensor only
• CPC Class Codes: H04N 23/56 provided with illuminating means
• CPC Class Codes: H04N 23/74 by influencing the scene brightness using illuminating means
• CPC Class Codes: H04N 23/741 by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors
• CPC Class Codes: H04N 23/843 Demosaicing, e.g. interpolating colour pixel values
• CPC Class Codes: H04N 25/10 for transforming different wavelengths into image signals
• CPC Class Codes: H04N 25/58 involving two or more exposures
• CPC Class Codes: H04N 25/583 with different integration times
• CPC Class Codes: H04N 25/585 with pixels having different sensitivities within the sensor, e.g. fast or slow pixels or pixels having different sizes
• CPC Class Codes: H04N 25/70 SSIS architectures; Circuits associated therewith
• CPC Class Codes: H04N 25/778 comprising amplifiers shared between a plurality of pixels, i.e. at least one part of the amplifier must be on the sensor array itself

Abstract

The disclosure extends to methods, systems, and computer program products for widening dynamic range within an image in a light deficient environment.

Claims

1. A method for imaging in a light deficient environment comprising:

emitting four electromagnetic radiation pulses of three different wavelengths into a light deficient environment during a blanking period during frame capture of an image sensor, wherein the image sensor comprises a plurality of pixels, and wherein the image sensor comprises a monochromatic image sensor;

capturing a plurality of frames in the light deficient environment using the image sensor, each one of the plurality of frames being radiated by one of the four electromagnetic radiation pulses of the three different wavelengths, wherein capturing at least one frame of the plurality frames comprises controlling a first subset of pixels during the at least one frame for a long exposure and controlling a second subset of pixels during the at least one frame for a short exposure, wherein the at least one frame for the long exposure and the at least one frame for the short exposure are two of the plurality of frames that are exposed to the same wavelength of electromagnetic radiation; and

creating a single display frame from the plurality of frames.

2. The method of claim 1, further comprising:

creating a stream of display frames by repeating the emitting electromagnetic radiation into the light deficient environment, the capturing the plurality of frames, and the creating the single display frame from the plurality of frames; and

providing the stream of display frames as a video stream.

3. The method of claim 1, wherein each of the pixels comprises a transfer gate transistor (TX).

4. The method of claim 3, wherein each TX in a subgroup of pixels is electrically connected together by a common TX line.

5. The method of claim 3, wherein the TX line provides global operation of transfer gate transistors for the subgroup of pixels.

6. The method of claim 1, wherein two or more pixels of the image sensor share a floating diffusion in a pixel share, wherein a first subgroup of pixels in the first subset is connected by a first TX line and a second subgroup of pixels in the second subset is connected by a second TX line.

7. The method of claim 6, wherein the two or more pixels are within a same row of pixels of the image sensor.

8. The method of claim 6, wherein global operation of the first TX line and the second TX line is performed during the blanking period.

9. The method of claim 6, wherein controlling the first subset of pixels during the at least one frame for the long exposure and controlling the second subset of pixels during the at least one frame for the short exposure comprises actuating the first TX line at a different time than the second TX line.

10. The method of claim 1, wherein the first subset of pixels and the second subset of pixels are arranged in a checkerboard pattern.

11. The method of claim 1, further comprising capturing at least a second frame during which the first subset of pixels is exposed for the short exposure and the second subset of pixels are exposed for the long exposure, wherein the long exposure comprises a longer length of time than the short exposure.

12. The method of claim 1, wherein emitting the electromagnetic radiation into the light deficient environment comprises controlling the emitter to pulse electromagnetic energy at a plurality of exposures corresponding to subsets of different pixels of the image sensor.

13. A system for digital imaging in a light deficient environment:

an emitter for providing illumination using electromagnetic pulses of a plurality of different wavelengths;

a monochromatic image sensor comprising a plurality of pixels that are sensitive to the electromagnetic pulses; and

a controller that causes the system to;

emit four electromagnetic radiation pulses of three different wavelengths into a light deficient environment during a blanking period during frame capture of the image sensor;

capture a plurality of frames in the light deficient environment using the image sensor, each one of the plurality of frames being radiated by one of the four electromagnetic pulses of the three different wavelengths, wherein capturing at least one frame of the repeating frames comprises controlling a first subset of pixels during the at least one frame for a long exposure and controlling a second subset of pixels during the at least one frame for a short exposure, wherein the at least one frame for the long exposure and the at least one frame for the short exposure are two of the plurality of frames that are exposed to the same wavelength of electromagnetic radiation; and

create a single display frame from the plurality of frames.

14. The system of claim 13, wherein the controller causes the system to:

create a stream of display frames by repeating the emitting electromagnetic radiation into the light deficient environment, the capturing the plurality of frames, and the creating the single display frame from the plurality of frames; and

provide the stream of display frames as a video stream.

15. The system of claim 13, wherein each of the pixels of the image sensor comprises a transfer gate transistor (TX).

16. The system of claim 15, wherein each TX in a subgroup of pixels is electrically connected together by a common TX line.

17. The system of claim 15, wherein the TX line provides global operation of transfer gate transistors for the subgroup of pixels.

18. The system of claim 13, wherein two or more pixels of the image sensor share a floating diffusion in a pixel share, wherein a first subgroup of pixels in the first subset is connected by a first TX line and a second subgroup of pixels in the second subset is connected by a second TX line.

19. The system of claim 18, wherein the two or more pixels are within a same row of pixels of the image sensor.

20. The system of claim 18, wherein global operation of the first TX line and the second TX line is performed during the blanking period.

21. The system of claim 18, wherein the controller controls the first subset of pixels during the at least one frame for the long exposure and controls the second subset of pixels during the at least one frame for the short exposure by actuating the first TX line at a different time than the second TX line.

22. The system of claim 13, wherein the first subset of pixels and the second subset of pixels are arranged in a checkerboard pattern in the image sensor.

23. The system of claim 13, wherein the controller further controls the system to capture at least a second frame during which the first subset of pixels is exposed for the short exposure and the second subset of pixels are exposed for the long exposure, wherein the long exposure comprises a longer length of time than the short exposure.

24. The system of claim 13, wherein the controller causes the system to emit the electromagnetic radiation into the light deficient environment by controlling the emitter to pulse electromagnetic energy at a plurality of exposures corresponding to subsets of different pixels of the image sensor.

25. Non-transitory computer readable storage media storing instructions that, when executed by one or more processors, cause the one or more processors to:

emit four electromagnetic radiation pulses of three different wavelengths into a light deficient environment during a blanking period during frame capture of an image sensor, wherein the image sensor comprises a plurality of pixels, and wherein the image sensor comprises a monochromatic image sensor;

capture a plurality of frames in the light deficient environment using the image sensor, each one of the plurality of frames being radiated by one of the four electromagnetic radiation pulses of the three different wavelengths, wherein capturing at least one frame of the repeating frames comprises controlling a first subset of pixels during the at least one frame for a long exposure and controlling a second subset of pixels during the at least one frame for a short exposure, wherein the at least one frame for the long exposure and the at least one frame for the short exposure are two of the plurality of frames that are exposed to the same wavelength of electromagnetic radiation; and

create a single display frame from the plurality of frames.

26. The computer readable storage media of claim 25, wherein the instructions further cause the one or more processors to:

create a stream of display frames by repeating the emitting electromagnetic radiation into the light deficient environment, the capturing the plurality of frames, and the creating the single display frame from the plurality of frames; and

provide the stream of display frames as a video stream.

27. The computer readable storage media of claim 25, wherein each of the pixels of the image sensor comprises a transfer gate transistor (TX).

28. The computer readable storage media of claim 27, wherein each TX in a subgroup of pixels is electrically connected together by a common TX line.

29. The computer readable storage media of claim 27, wherein the TX line provides global operation of transfer gate transistors for the subset of pixels.

30. The computer readable storage media of claim 25, wherein two or more pixels of the image sensor share a floating diffusion in a pixel share, wherein a first subgroup of pixels in the first subset is connected by a first TX line and a second subgroup of pixels in the second subset is connected by a second TX line.

31. The computer readable storage media of claim 30, wherein the two or more pixels are within a same row of pixels of the image sensor.

32. The computer readable storage media of claim 30, wherein the instructions cause global operation of the first TX line and the second TX line to be performed during the blanking period.

33. The computer readable storage media of claim 30, wherein the instructions cause the one or more processors to control the first subset of pixels during the at least one frame for the long exposure and control the second subset of pixels during the at least one frame for the short exposure by actuating the first TX line at a different time than the second TX line.

34. The computer readable storage media of claim 25, wherein the first subset of pixels and the second subset of pixels are arranged in a checkerboard pattern in the image sensor.

35. The computer readable storage media of claim 25, wherein the instructions further cause the one or more processors to capture at least a second frame during which the first subset of pixels is exposed for the short exposure and the second subset of pixels are exposed for the long exposure, wherein the long exposure comprises a longer length of time than the short exposure.

36. The computer readable storage media of claim 25, wherein the instructions further cause the one or more processors to emit the electromagnetic radiation into the light deficient environment by controlling the emitter to pulse electromagnetic energy at a plurality of exposures corresponding to subsets of different pixels of the image sensor.