Endoscopic image enhancement using contrast limited adaptive histogram equalization (CLAHE) implemented in a processor

• US 10,516,865 B2
• Filed: 05/16/2016
• Issued: 12/24/2019
• Est. Priority Date: 05/17/2015
• Status: Active Grant

• Current Assignee: EndoChoice Incorporated (Boston Scientific Corporation)
• Original Assignee: EndoChoice Incorporated (Boston Scientific Corporation)
• Inventors: Sidar, Itay, Davidson, Tal, Kronman, Achia, Mor, Lior, Levy, Idan
• Primary Examiner(s): Perez-Fuentes, Luis
• Application Number:
  US15/155,814
• Publication Number:
  US 20160335751A1
• Time in Patent Office: 1,317 Days
• CPC Class Codes: A61B 1/000095 for image enhancement
• CPC Class Codes: A61B 1/0005 combining images e.g. side-by-side, superimposed or tiled
• CPC Class Codes: G06T 2200/28 involving image processing hardware
• CPC Class Codes: G06T 5/20 using local operators
• CPC Class Codes: G06T 5/40 using histogram techniques
• CPC Class Codes: G06T 5/92 based on global image properties
• CPC Class Codes: H04N 9/646 for image enhancement, e.g. vertical detail restoration, cross-colour elimination, contour correction, chrominance trapping filters

Abstract

Systems and methods of enhancing images use a contrast limited adaptive histogram equalization (CLAHE) algorithm in a field programmable gate array (FPGA). The images may be obtained by the imaging elements of a multiple imaging elements endoscope of an endoscopy system.

Claims

1. An endoscope system comprising:

an endoscope including a first viewing element and a second viewing element;

a control unit operatively coupled to the endoscope by a cable, wherein the control unit comprises;

a base board module;

one or more processing boards that implement a contrast limited adaptive histogram equalization (CLAHE) algorithm to enhance images obtained by the first viewing element and the second viewing element, wherein the one or more processing boards perform the steps of;

receiving a first YCbCr color space video stream from the first viewing element and a second YCbCr color space video stream from the second viewing element,

converting the first YCbCr color space video stream to a first Lab color space video stream and the second YCbCr color space video stream to a second Lab color space video stream,

building and storing first histogram information based on the first Lab color space video stream,

building and storing second histogram information based on the second Lab color space video stream,

converting the first histogram information into first cumulative distribution function values including applying a first clipping threshold to the first histogram information,

converting the second histogram information into second cumulative distribution function values including applying a second clipping threshold to the second histogram information, wherein the second clipping threshold is different from the first clipping threshold,

obtaining a first plurality of Lab color space frames from the first Lab color space video stream and a second plurality of Lab color space frames from the second Lab color space video stream,

rearranging the first plurality of Lab color space frames to generate a plurality of first delayed Lab streams,

rearranging the second plurality of Lab color space frames to generate a plurality of second delayed Lab streams,

generating a first enhanced Lab stream based on the plurality of first delayed Lab streams and the first cumulative distribution function values,

generating a second enhanced Lab stream based on the plurality of second delayed Lab streams and the second cumulative distribution function values, and

sending output video streams indicative of the first enhanced Lab stream and the second enhanced Lab stream to a display.

2. The endoscope system of claim 1, wherein the first YCbCr color space video stream and the second YCbCr color space video stream are received by an adapter of the one or more processing boards, the adapter having conversion modules, and wherein the first YCbCr color space video stream and the second YCbCr color space video stream are converted to the first Lab color space video stream and the second lab color space video stream by the conversion modules.

3. The endoscope system of claim 1, wherein the first Lab color space frames and the second Lab color space frames are obtained from the first Lab color space video and the second Lab color space video stream using a frame grabber of the one or more processing boards.

4. The endoscope system of claim 3, wherein said base board module further comprises DDR3 memory in communication with said frame grabber.

5. The endoscope system of claim 1, wherein the first plurality of Lab color space frames and the second plurality of Lab color space frames are rearranged to generate the first plurality of delayed Lab streams and the second plurality of delayed Lab streams using a fetching module of the plurality of processing boards.

6. The endoscope system of claim 1, wherein the histogram information is converted into cumulative distribution function values, and the first enhanced Lab stream and the second enhanced Lab stream are, generated using an interpolation algorithm module of the plurality of processing boards.

18. The endoscope system of claim 1, wherein building and storing first histogram information based on the first Lab color space video stream includes splitting the first Lab color space video stream into a first plurality of first tiles, wherein units of the first tile heights are in video lines and units of the first tile widths are in pixels, wherein each of the first plurality of tiles overlaps with its neighboring tiles;

and,

wherein building and storing second histogram information based on the second Lab color space video stream includes splitting the second Lab color space video stream into a second plurality of tiles, wherein units of the second tile heights are in video lines and units of the second tile widths are in pixels, wherein each of the second plurality of tiles overlaps with its neighboring tiles.

7. A method of enhancing images obtained by a plurality of viewing elements in an endoscope system using a contrast limited adaptive histogram equalization (CLAHE) process, wherein said endoscope system comprises (a) a control unit having a baseboard module including one or more processing boards configured to implement said process, and (b) an endoscope operatively coupled to the control unit by a cable, the endoscope including the plurality of viewing elements, said method comprising the steps of:

receiving a plurality of YCbCr color space video streams from each of the plurality of viewing elements;

converting the plurality of YCbCr color space video streams to Lab color space video streams;

building and storing histogram information based on the plurality of Lab color space video streams including splitting the plurality of Lab color space video streams into a plurality of tiles, wherein units of the tile heights are in video lines and units of the tile widths are in pixels, wherein each of the plurality of tiles overlaps with its neighboring tiles;

converting the histogram information into cumulative distribution function values;

obtaining said Lab color space video streams as Lab color space frames;

rearranging Lab color space frame data to generate delayed Lab streams;

generating a plurality of enhanced Lab streams, using the plurality of delayed Lab streams and the cumulative distribution function values; and

displaying output video streams indicative of the plurality of enhanced Lab streams.

8. The method of claim 7, wherein converting said YCbCr color space streams to Lab color space video streams comprises the steps of:

performing YCbCr to RGB conversion using MAD instantiation;

performing a function similar to Y(x)=x{circumflex over ( 

)}2.40 using a LUT;

performing RGB to XYZ conversion using MAD instantiation;

executing a function similar to Y(x)=x{circumflex over ( 

)}0.33 using a LUT; and

performing XYZ to Lab conversion using MAD instantiation.

9. The method of claim 7, further comprising converting said enhanced Lab streams to enhanced YCbCr streams.

10. The method of claim 9, wherein converting said enhanced Lab streams to enhanced YCbCr streams comprises the steps of:

performing Lab to XYZ conversion using MAD instantiation;

executing a function similar to Y(x)=x{circumflex over ( 

)}0.33 using a LUT;

performing XYZ to RGB conversion using MAD instantiation;

performing a function similar to Y(x)=x{circumflex over ( 

)}(1/2.40) using a LUT; and

performing RGB to YCbCr conversion using MAD instantiation.

11. A non-transient computer readable medium containing program instructions for causing a computer to perform a method of enhancing images obtained by three viewing elements in an endoscope system using a contrast limited adaptive histogram equalization (CLAHE) process, wherein said endoscope system comprises (a) a control unit with a baseboard module including a processor configured to implement said process, and (b) an endoscope operatively coupled to the control unit by a cable, the endoscope including the three viewing elements, said method comprising the steps of:

receiving three YCbCr color space video streams from the three viewing elements with an adapter on said processor, each of the three viewing elements providing one of said three YCbCr color space video streams;

converting each of the three YCbCr color space video streams to corresponding three Lab color space video streams within said adapter;

building and storing histogram information based on the three Lab color space video streams within a histogram module on said processor;

converting the histogram information into cumulative distribution function values within the histogram module;

transmitting the three Lab color space video streams to a frame grabber on said processor in a form of a video stream obtained from a single viewing element with a Y input of the frame grabber being fed by a Lab color space video stream of a first viewing element of the three viewing elements, a Cb input of the frame grabber being fed by a Lab color space video stream of a second viewing element of the three viewing elements, and a Cr input of the frame grabber being fed by a Lab color space video stream of a third viewing element of the three viewing elements;

acquiring the three Lab color space video streams as corresponding Lab color space frames within said frame grabber;

acquiring the Lab color space frames from the frame grabber using a fetching module on said processor;

rearranging Lab color space frame data within said fetching module to generate three delayed Lab streams corresponding to the Lab color space frames acquired by the fetching module;

generating three enhanced Lab streams, using the three delayed Lab streams and the cumulative distribution function values, within an interpolation algorithm module on said processor; and

prompting a display of output video streams indicative of the three enhanced Lab streams on one or more displays in operative communication with the processor.

12. The computer readable medium of claim 11, wherein converting a YCbCr color space stream to a Lab color space video stream comprises the steps of:

performing YCbCr to RGB conversion using MAD instantiation;

performing a function similar to Y(x)=x{circumflex over ( 

)}2.40 using a LUT;

performing RGB to XYZ conversion using MAD instantiation;

executing a function similar to Y(x)=x{circumflex over ( 

)}0.33 using a LUT; and

performing XYZ to Lab conversion using MAD instantiation.

13. The computer readable medium of claim 11, the method further comprising converting each of the three enhanced Lab streams to corresponding three enhanced YCbCr streams.

14. The computer readable medium of claim 13, wherein converting each enhanced Lab stream to an enhanced YCbCr stream comprises the steps of:

performing Lab to XYZ conversion using MAD instantiation;

executing a function similar to Y(x)=x{circumflex over ( 

)}0.33 using a LUT;

performing XYZ to RGB conversion using MAD instantiation;

performing a function similar to Y(x)=x{circumflex over ( 

)}(1/2.40) using a LUT; and

performing RGB to YCbCr conversion using MAD instantiation.

15. The computer readable medium of claim 11 wherein the first viewing element is a left side viewing element of the endoscope, the second viewing element is a front viewing element of the endoscope, and the third viewing element is a right side viewing element of the endoscope.

16. The computer readable medium of claim 11 wherein the first of the three enhanced Lab streams is prompted for display as a first image on one or more display screens coupled with the control unit, the second of the three enhanced Lab streams is prompted for display as a second image on one or more display screens coupled with the control unit, and the third of the three enhanced Lab streams is prompted for display as a third image on one or more display screens coupled with the control unit.

17. The computer readable medium of claim 11 wherein each enhanced Lab stream is prompted for display as a contrast enhanced image on a display device coupled with the control unit, each contrast enhanced image comprising a plurality of frames having a higher amount of contrast as compared to the corresponding plurality of frames in the corresponding image obtained from a viewing element of the endoscope.

19. The non-transient computer readable medium of claim 11, wherein converting the histogram information into cumulative distribution function values within the histogram module includes applying a first clipping threshold to a first portion of the histogram information, and applying a second clipping threshold to a second portion of the histogram information, wherein the second clipping threshold is different from the first clipping threshold.