Noise aware edge enhancement

• US 10,341,588 B2
• Filed: 03/14/2014
• Issued: 07/02/2019
• Est. Priority Date: 03/15/2013
• Status: Active Grant

• Current Assignee: Depuy Synthes Products Incorporated (Johnson & Johnson)
• Original Assignee: Depuy Synthes Products Incorporated (Johnson & Johnson)
• Inventors: Richardson, John, Wichern, Donald M.
• Primary Examiner(s): Messmore, Jonathan R
• Application Number:
  US14/214,320
• Publication Number:
  US 20140267653A1
• Time in Patent Office: 1,936 Days
• Field of Search: 348 65
• CPC Class Codes: G06T 2207/20192 Edge enhancement; Edge preservation
• CPC Class Codes: G06T 5/73 Deblurring; Sharpening
• CPC Class Codes: H04N 23/555 for picking-up images in sites, inaccessible due to their dimensions or hazardous conditions, e.g. endoscopes or borescopes
• CPC Class Codes: H04N 25/60 Noise processing, e.g. detecting, correcting, reducing or removing noise
• CPC Class Codes: H04N 5/142 Edging; Contouring

Abstract

The disclosure extends to methods, systems, and computer program products for enhancing edges within an image in a light deficient environment, which utilizes knowledge of the expected noise pixel by pixel, to control the strength of the edge enhancement and thereby limit the impact of the enhancement on the perception of noise.

Claims

1. A digital imaging method for use with an endoscope in ambient light deficient environments comprising:

illuminating an environment using a source of visible, infrared or ultraviolet electromagnetic radiation;

continuously focusing a scene onto a pixel array of a sensor;

sensing reflected electromagnetic radiation with said pixel array, wherein said pixel array generates image data;

creating an image frame from said image data;

detecting image textures and edges within the image frame;

enhancing textures and edges within the image frame;

retrieving from memory properties pertaining to a pixel technology comprising a known-conversion gain for a given pixel or group of pixels and an applied sensor gain of the sensor to;

determine an expectation for a magnitude of noise within the image frame created by said sensor based on the known-conversion gain for an individual pixel in the pixel array;

using said expectation for the magnitude of noise to control the enhancing of the textures and edges within the image frame; and

creating a stream of images by sequentially combining a plurality of image frames.

2. The method of claim 1, wherein the enhancing of the textures and edges within the image frame comprises:

performing a plurality of enhancements within the image frame generated by the pixel array, wherein each of the enhancements correspond to variations of noise due to variations in photo-signal.

3. The method of claim 2, further comprising calculating noise correction based on a combination of Poisson statistics of photon arrival and electronic noise arising from the pixel array and its readout electronics.

4. The method of claim 2, further comprising computing the expectation for the magnitude of noise based on the known-conversion gain of the individual pixel within the pixel array and one or more of the applied sensor gain or a voltage range of a digitizer.

5. The method of claim 1, wherein the degree of applied edge enhancement is governed by the known-conversion gain factor applied to the detected edges, which depends on the expectation for the magnitude of noise.

6. The method of claim 5, wherein said known-conversion gain factor is assessed locally for the individual pixel.

7. The method of claim 5, wherein said known-conversion gain factor is determined for a whole frame, based on the applied sensor gain.

8. The method of claim 5, wherein said known-conversion gain factor is derived from a comparison of an edge strength parameter to the expectation for the magnitude of noise located near each pixel.

9. The method of claim 8, wherein said edge strength parameter is taken to be a difference between two spatially filtered versions of the luminance component of the original image, with different filter kernels applied to each.

10. The method of claim 8, wherein said edge strength parameter is taken to be a difference between one spatially filtered and one unfiltered version of the luminance component of the original frame.

11. The method of claim 5, wherein the method of controlling the degree of edge enhancement involves applying said known-conversion gain factor to an edge strength parameter and adding the result to the luminance component of the original image.

12. The method of claim 1, further comprising creating a three dimensional image stream by combining the image frames of a plurality of pixel arrays disposed on a plurality of substrates that are stacked.

13. A digital imaging system comprising:

an endoscopic device for use in ambient light deficient environment;

a source of visible, infrared or ultraviolet electromagnetic radiation for illuminating an environment;

an image sensor comprising a pixel array, wherein a scene is continuously focused onto the pixel array;

wherein the image sensor senses reflected electromagnetic radiation with said pixel array, wherein said pixel array generates image data;

wherein an image frame is created from said image data;

wherein a processor;

detects image textures and edges within the image frame;

enhances textures and edges within the image frame;

retrieves from memory properties pertaining to a pixel technology comprising a known-conversion gain for a given pixel or group of pixels and an applied sensor gain of the sensor to determine an expectation for a magnitude of noise within the image frame based on the known-conversion gain for an individual pixel in the pixel array and using the expectation for the magnitude of noise to control the enhancing of the textures and edges within the image frame; and

wherein a stream of images is created by sequentially combining a plurality of image frames.

14. The system of claim 13, wherein the enhancing of the textures and edges within the image frame comprises:

performing a plurality of enhancements within the image frame generated by the pixel array, wherein each of the enhancements correspond to variations of noise due to variations in photo-signal.

15. The system of claim 14, wherein the system further comprises calculating noise correction based on a combination of Poisson statistics of photon arrival and electronic noise arising from the pixel array and its readout electronics.

16. The system of claim 14, wherein the system further comprises computing the expectation for the magnitude of noise based the known-conversion gain of the individual within the pixel array and one or more of the applied sensor gain or a voltage range of a digitizer.

17. The system of claim 13, wherein the degree of applied edge enhancement is governed by the known-conversion gain factor applied to the detected edges, which depends on the expectation for the magnitude of noise.

18. The system of claim 17, wherein said known-conversion gain factor is assessed locally for the individual pixel.

19. The system of claim 17, wherein said known-conversion gain factor is determined for a whole frame, based on the applied sensor gain.

20. The system of claim 17, wherein said known-conversion gain factor is derived from a comparison of an edge strength parameter to the expectation for the magnitude of noise located near each pixel.

21. The system of claim 20, wherein said edge strength parameter is taken to be a difference between two spatially filtered versions of the luminance component of the original image, with different filter kernels applied to each.

22. The system of claim 20, wherein said edge strength parameter is taken to be a difference between one spatially filtered and one unfiltered version of the luminance component of the original frame.

23. The system of claim 17, wherein the system of controlling the degree of edge enhancement involves applying said known-conversion gain factor to an edge strength parameter and adding the result to the luminance component of the original image.

24. The system of claim 13, wherein the system further comprises creating a three dimensional image stream by combining the image frames of a plurality of pixel arrays disposed on a plurality of substrates that are stacked.