Comprehensive fixed pattern noise cancellation

• US 10,341,593 B2
• Filed: 03/15/2014
• Issued: 07/02/2019
• Est. Priority Date: 03/15/2013
• Status: Active Grant

• Current Assignee: Depuy Synthes Products Incorporated (Johnson & Johnson)
• Original Assignee: Depuy Synthes Products Incorporated (Johnson & Johnson)
• Inventors: Blanquart, Laurent, Richardson, John
• Primary Examiner(s): Chio, Tat C
• Assistant Examiner(s): Lima, Fabio S
• Application Number:
  US14/214,789
• Publication Number:
  US 20140267654A1
• Time in Patent Office: 1,935 Days
• Field of Search: 348 65- 76
• CPC Class Codes: A61B 1/00006 of control signals
• CPC Class Codes: A61B 1/00009 of image signals during a use of endoscope
• CPC Class Codes: A61B 1/000095 for image enhancement
• CPC Class Codes: A61B 1/00057 provided with means for testing or calibration
• CPC Class Codes: G06T 2207/20224 Image subtraction
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: H04N 25/67 applied to fixed-pattern noise, e.g. non-uniformity of response
• CPC Class Codes: H04N 25/677 for reducing the column or line fixed pattern noise

Abstract

The disclosure extends to methods, systems, and computer program products for producing an image in light deficient environments having cancelled fixed pattern noise.

Claims

1. A digital imaging method for use with an endoscope in ambient light deficient environments comprising:

actuating an emitter to emit a plurality of pulses of electromagnetic radiation, wherein each pulse comprises an individual wavelength of electromagnetic radiation to cause illumination within the light deficient environment;

pulsing the emitter at a predetermined interval corresponding to a sensing interval of a pixel array;

sensing reflected electromagnetic radiation from a pulse with the pixel array to create an image frame in a plurality of cycles, the cycles including an integration time of the pixel array that is controlled using an electronic shutter, wherein a duration of the integration time is variable from cycle to cycle in the plurality of cycles between a readout of a first frame in a first cycle and a readout of a next subsequent frame in a second subsequent cycle and wherein the electromagnetic radiation pulse width and duration are variable during each cycle to adjust a level of light intensity caused by each pulse in the cycle, wherein variation of the integration time is related to the variation in the pulse width and the duration of the electromagnetic radiation pulse;

wherein the pixel array is actuated at the sensing interval that corresponds to the pulse interval of the emitter;

stopping the emitter from pulsing for a single iteration;

creating a dark frame by sensing the pixel array while the emitter is not pulsing a pulse of electromagnetic radiation;

creating one or more reference frames, based on pixel values in both the dark frame and in a plurality of previous dark frames, for use in removing fixed pattern noise;

removing fixed pattern noise from the image frame by subtracting stored reference data in the one or more reference frames from the image frame; and

creating a stream of images by combining a plurality of image frames to form a video stream.

2. The method of claim 1, wherein creating the one or more reference frames comprises creating a reference frame having average values for pixels in the dark frame and the plurality of previous dark frames.

3. The method of claim 1, wherein one or more of the dark frame and the plurality of previous dark frames are created, interspersed within image frames.

4. The method of claim 1, wherein the one or more reference frames are created upon startup of a system comprising a laser emitter and pixel array and stored within memory associated with the system.

5. The method of claim 4, wherein a plurality of reference frames may be established at different exposure times.

6. The method of claim 1, comprising periodically sampling dark frames and updating the one or more reference frames based on the periodically sampled dark frames.

7. The method of claim 6, wherein updating the one or more reference frames comprises incrementally adjusting pixel values of the one or more reference frames using exponential smoothing.

8. The method of claim 1, wherein each image frame is enhanced before being integrated into the video stream.

9. The method in claim 8, wherein the method further comprises subtracting data from a single dark reference buffer from an image frame.

10. The method in claim 8, wherein the method further comprises computing the reference data for a particular integration time by interpolating between two or more dark reference buffers established at different integration times.

11. The method of claim 10, wherein the method further comprises subtracting the reference data from an image frame.

12. The method of claim 1, wherein the emitter is a laser to emit a laser pulse of a wavelength of electromagnetic radiation to cause illumination within the light deficient environment.

13. The method of claim 1, wherein the emitter is an LED emitter to emit an LED pulse of a wavelength of electromagnetic radiation to cause illumination within the light deficient environment.

14. A system for digital imaging system for use in ambient light deficient environments comprising:

one or more processors;

one or more memory devices operably coupled to the one or more processors and storing executable and operational data, the executable and operational data effective to cause the one or more processors to;

actuate a laser emitter to emit a pulse of a wavelength of electromagnetic radiation to cause illumination within the light deficient environment;

pulse the laser emitter at a predetermined interval corresponding to a sensing interval of a pixel array;

sense reflected electromagnetic radiation from the pulse with the pixel array to create an image frame;

wherein the pixel array is actuated in a plurality of cycles, that includes the pulse interval of the laser emitter, wherein the each cycle in the plurality of cycles includes an integration time of the pixel array that is controlled using an electronic shutter, wherein a duration of the integration time is variable from cycle to cycle in the plurality of cycles between a readout of a first frame in a first cycle and a readout of a next subsequent frame in a second subsequent cycle and wherein the electromagnetic radiation pulse width and duration are variable during each cycle to adjust a level of light intensity caused by each pulse in the cycle wherein variation of the integration time is related to the variation in the pulse width and the duration of the electromagnetic radiation pulse,

stop the emitter from pulsing for a single iteration;

sense the pixel array while the emitter is not pulsing a pulse to create a dark frame;

create one or more reference frames, based on pixel values in both the dark frame and in a plurality of previous dark frames, for use in removing fixed pattern noise;

remove fixed pattern noise from an image frame by subtracting values derived from one or more reference frames; and

create a stream of images by combining a plurality of image frames to form a video stream.

15. The system of claim 14, wherein the executable and operational data cause the one or more processors to create the one or more reference frames by creating a reference frame having average values for pixels in the dark frame and the plurality of previous dark frames.

16. The system of claim 14, wherein one or more of the dark frame and the plurality of dark frames are created, interspersed within image frames.

17. The system of claim 14, wherein the one or more reference frames are created upon startup of a system comprising the laser emitter and pixel array and stored within memory associated with the system.

18. The system of claim 17, wherein a plurality of reference frames may be established at different exposure times.

19. The system of claim 14, wherein the executable and operational data cause the one or more processors to periodically sample dark frames and update the one or more reference frames based on the periodically sampled dark frames.

20. The system of claim 19, wherein the executable and operational data cause the one or more processors to update the one or more reference frames by incrementally adjusting pixel values of the one or more reference frames, using exponential smoothing.

21. The system of claim 14, wherein each image frame is enhanced before being integrated into the video stream.

22. The system of claim 21, wherein the executable and operational data effective to cause the one or more processors to subtract data from a single dark reference buffer from an image frame.

23. The system of claim 21, wherein the executable and operational data effective to cause the one or more processors to compute the reference data for a particular integration time by interpolating between two or more dark reference buffers established at different integration times.

24. The system of claim 23, wherein the executable and operational data effective to cause the one or more processors to subtract the reference data from an image frame.