Camera system with autonomous miniature camera and light source assembly and method for image enhancement

• US 8,860,793 B2
• Filed: 10/15/2009
• Issued: 10/14/2014
• Est. Priority Date: 10/15/2008
• Status: Active Grant

• Current Assignee: Regents of the University of California (University of California)
• Original Assignee: Regents of the University of California (University of California)
• Inventors: Lo, Yu-Hwa, Mintz, Yoav, Nguyen, Truong, Tzeng, Jack
• Primary Examiner(s): Perungavoor, Sath V
• Assistant Examiner(s): LUO, KATE H
• Application Number:
  US13/124,659
• Publication Number:
  US 20110261178A1
• Time in Patent Office: 1,825 Days
• US Class Current: 348/68
• CPC Class Codes: A61B 1/05 characterised by the image sensor, e.g. camera, being in the distal end portion
• CPC Class Codes: A61B 1/0676 at distal tip of an endoscope
• CPC Class Codes: A61B 1/0684 using light emitting diodes [LED]
• CPC Class Codes: A61B 2090/309 using white LEDs
• CPC Class Codes: A61B 90/361 Image-producing devices, e.g. surgical cameras
• CPC Class Codes: G03B 15/05 Combinations of cameras with electronic flash apparatus; Electronic flash units
• CPC Class Codes: G03B 2215/0528 movable reflector, e.g. change of illumination angle or illumination direction

Abstract

The present Invention relates to a camera system suitable for use in minimally invasive surgery (MIS), among other applications. In at least one embodiment, the camera system includes an autonomous miniature camera, a light source assembly providing features such as steerable illumination and a variable radiation angle, and a control and processing unit for processing images acquired by the camera Io generate improved images having reduced blurring using a deblurring algorithm.

Claims

1. A method for deblurring a blurred first image corresponding to a first color channel of a camera that also produces a sharp second image corresponding to a second color channel of the camera, wherein the first and the second images each include a plurality of pixels with each pixel having an associated respective value, the method comprising:

decomposing the first image by filtering and downsampling to generate a first set of one or more first sub-band output coefficients, wherein each first sub-band output coefficient corresponds to a respective sub-band in a selected one of a wavelet and a contourlet domain;

decomposing the second image by filtering and downsampling to generate a second set of second sub-band output coefficients, wherein each second sub-band output coefficient corresponds to a respective sub-band in the selected domain;

selecting those second sub-band output coefficients that represent edge information, each of the selected second sub-band output coefficients corresponding to a respective selected sub-band, with the selected sub-bands together defining an edge sub-band set;

preparing a third set of sub-band output coefficients which includes the selected second sub-band output coefficients representing edge information and at least one of the first sub-band output coefficients corresponding to a sub-band other than those sub-bands in the edge sub-band set; and

reconstructing a deblurred first image by upsampling and filtering using the third set of sub-band output coefficients as input.

2. The method of claim 1, wherein each decomposing step occurs over two or more levels and the reconstructing step occurs over the same number of levels.

3. The method of claim 2, wherein the number of levels of decomposition is determined based at least in part on a frequency response of a blurring lens of the camera.

4. The method of claim 1, the first set and the second set are combined to form the third set.

5. The method of claim 1, wherein the sub-bands corresponding to first set and the second set are predetermined and non-overlapping.

6. The method of claim 1, further including prefiltering the second image prior to decomposition of the second image.

7. The method of claim 6, wherein the prefiltering uses a Wiener filter.

8. The method of claim 1, wherein the decomposition steps occur in the contourlet domain and the selected second sub-band output coefficients are those which have been determined to represent edge information.

9. The method of claim 8, wherein the edge information is determined using an ant colonization optimization scheme.

10. A method for deblurring a first image of a camera, wherein the camera generates the first image corresponding to a first color channel and a second image corresponding to a second color channel, wherein the first and the second images each includes a plurality of pixels with each pixel having an associated respective value, the method comprising:

selecting a filter bank having a decomposition portion and a reconstruction portion, the decomposition portion having at least two cascaded levels for receiving two inputs and generating intermediate sub-band output coefficients for each of a predetermined number N of sub-bands, the reconstruction portion having at least two cascaded levels for receiving the intermediate sub-band output coefficients and generating an output, wherein the decomposition portion includes multiple decomposition stages at a first level connecting to N decomposition stages at a second level, each decomposition stage including at least one of a decomposition filter and a downsampler, wherein the reconstruction portion includes N reconstruction stages at a third level connecting to N/2 reconstruction stages at a fourth level, each reconstruction stage including at least one of an upsampler and a reconstruction filter;

decomposing the first image using a first part of the decomposition portion of the filter bank to generate at least one first intermediate sub-band output coefficient corresponding to one of the N sub-bands;

decomposing the second image using a second part of the decomposition portion of the filter bank to generate at least a second, a third intermediate, and a fourth sub-band output coefficient, each corresponding to a respective one of the N sub-bands; and

reconstructing a deblurred image corresponding to the first color channel using at least the first, the second, the third, and the fourth sub-band output coefficients as input to the reconstruction side of the filter bank,

wherein each decomposing step occurs over two or more levels and the reconstructing step occurs over the same number of levels and the number of levels of decomposition is determined based at least in part on a frequency response of a blurring lens of the camera.

11. The method of claim 10, wherein the sub-bands corresponding to intermediate sub-band output coefficients of the first image are non-overlapping with the sub-bands corresponding to the intermediate sub-band output coefficients of the second image.

12. The method of claim 10, further including prefiltering the second image prior to decomposition of the second color image.

13. The method of claim 12, wherein the prefiltering uses a Wiener filter.

14. A method for deblurring a blurred first image, wherein the blurred first image corresponds to a first color channel of a camera and a second image corresponds to a second color channel of the camera, wherein the first and the second images each comprise a plurality of pixels with each pixel having an associated respective value, the method comprising:

decomposing the first image by filtering and downsampling in a decomposition portion of a filter bank to generate for a predetermined number of sub-bands in a contourlet domain a first set of first sub-band output coefficients, wherein each first sub-band output coefficient corresponds to a respective one of the sub-bands;

decomposing the second image by filtering and downsampling in the decomposition portion of the filter bank to generate for the predetermined number of sub-bands a second set of second sub-band output coefficients, wherein each second sub-band output coefficient corresponds to a respective one of the sub-bands in the contourlet domain;

determining which of the second sub-band output coefficients and corresponding respective sub-bands represent edge information;

preparing a set of third sub-band output coefficients by modifying the first set of first sub-band output coefficients to replace each of those first sub-band output coefficients in the first set which correspond to the respective determined sub-bands representing edge information with the corresponding second sub-band output coefficients from the second set; and

reconstructing a deblurred image corresponding to the first image by upsampling and filtering the set of third sub-band output coefficients in a reconstruction portion of the filter bank.

15. The method of claim 14, further including combining the deblurred image with the second image to generate a composite color image.

16. The method of claim 14, wherein the edge information is determined using an ant colonization optimization scheme.

17. The method of claim 16, further including preparing a binary edge map in the contourlet domain and using the binary edge map in the preparing step to select corresponding second sub-band output coefficients from the second set.

18. The method of claim 17, further including preparing a binary edge map in the contourlet domain, dilating the binary edge map, and using the dilated binary edge map in the preparing step to select corresponding second sub-band output coefficients from the second set.

19. A method for deblurring a blurred first image corresponding to a first color channel of a camera that also produces a sharp second image corresponding to a second color channel of the camera, wherein the first and the second images each include a plurality of pixels with each pixel having an associated respective value, the method comprising:

decomposing the first image by filtering and downsampling to generate a first set of one or more first sub-band output coefficients, wherein each first sub-band output coefficient corresponds to a respective sub-band in a selected one of a wavelet and a contourlet domain;

decomposing the second image by filtering and downsampling to generate a second set of second sub-band output coefficients, wherein each second sub-band output coefficient corresponds to a respective sub-band in the selected domain;

selecting those second sub-band output coefficients that represent edge information, each of the selected second sub-band output coefficients corresponding to a respective selected sub-band, with the selected sub-bands together defining an edge sub-band set;

preparing a third set of sub-band output coefficients which includes the selected second sub-band output coefficients representing edge information and at least one first sub-band output coefficient corresponding to a sub-band other than those sub-bands in the edge sub-band set; and

reconstructing a deblurred first image by upsampling and filtering using the third set of sub-band output coefficients as input,

wherein each decomposing step occurs over two or more levels and the reconstructing step occurs over the same number of levels, and wherein the number of levels of decomposition is determined based at least in part on a frequency response of a blurring lens of the camera.

20. A method for deblurring a blurred first image, wherein the blurred first image corresponds to a first color channel of a camera and a second image corresponds to a second color channel of the camera, wherein the first and the second images each comprise a plurality of pixels with each pixel having an associated respective value, the method comprising:

decomposing the first image by filtering and downsampling in a decomposition portion of a filter bank to generate for a predetermined number of sub-bands in a contourlet domain a first set of first sub-band output coefficients, wherein each first sub-band output coefficient corresponds to a respective one of the sub-bands;

decomposing the second image by filtering and downsampling in the decomposition portion of the filter bank to generate for the predetermined number of sub-bands a second set of second sub-band output coefficients, wherein each second sub-band output coefficient corresponds to a respective one of the sub-bands in the contourlet domain;

determining which of the second sub-band output coefficients and corresponding respective sub-bands represent edge information, wherein the edge information is determined using an ant colonization optimization scheme;

preparing a set of third sub-band output coefficients by modifying the first set of first sub-band output coefficients to replace each of those first sub-band output coefficients in the first set which correspond to the respective determined sub-bands representing edge information with the corresponding second sub-band output coefficients from the second set; and

reconstructing a deblurred image corresponding to the first image by upsampling and filtering the set of third sub-band output coefficients in a reconstruction portion of the filter bank,

wherein the method further includes;

preparing a binary edge map in the contourlet domain and using the binary edge map in the preparing step to select corresponding second sub-band output coefficients from the second set; and

preparing a binary edge map in the contourlet domain, dilating the binary edge map, and using the dilated binary edge map in the preparing step to select corresponding second sub-band output coefficients from the second set.