CAMERA SYSTEM WITH AUTONOMOUS MINIATURE CAMERA AND LIGHT SOURCE ASSEMBLY AND METHOD FOR IMAGE ENHANCEMENT

• US 20110261178A1
• Filed: 10/15/2009
• Published: 10/27/2011
• Est. Priority Date: 10/15/2008
• Status: Active Grant

• Current Assignee: Regents of the University of California (University of California)
• Original Assignee: Regents of the University of California (University of California)
• Inventors: Tzeng, Jack, Mintz, Yoav, Lo, Yu-Hwa, Nguyen, Truong
• Granted Patent: US 8,860,793 B2
• Time in Patent Office: Days
• US Class Current: 348/68
• CPC Class Codes: A61B 1/05 characterised by the image sensor, e.g. camera, being in the distal end portion
• CPC Class Codes: A61B 1/0676 at distal tip of an endoscope
• CPC Class Codes: A61B 1/0684 using light emitting diodes [LED]
• CPC Class Codes: A61B 2090/309 using white LEDs
• CPC Class Codes: A61B 90/361 Image-producing devices, e.g. surgical cameras
• CPC Class Codes: G03B 15/05 Combinations of cameras with electronic flash apparatus; Electronic flash units
• CPC Class Codes: G03B 2215/0528 movable reflector, e.g. change of illumination angle or illumination direction

Abstract

The present Invention relates to a camera system suitable for use in minimally invasive surgery (MIS), among other applications. In at least one embodiment, the camera system includes an autonomous miniature camera, a light source assembly providing features such as steerable illumination and a variable radiation angle, and a control and processing unit for processing images acquired by the camera Io generate improved images having reduced blurring using a deblurring algorithm.

Claims

1. A camera system configured to acquire image information representative of subject matter within a selected field of view, the camera system comprising,

a camera that receives reflected light and based thereon generates the image information;

and

a light source assembly operated in conjunction with the camera so as to provide illumination, the light source assembly including a first light source and a first tunable lens,

wherein at least some of the illumination output by the light source assembly is received back at the camera as the reflected light, and

wherein the tunable lens is adjustable to vary the illumination output by the light source assembly,

whereby the illumination can be varied to substantially match the selected field of view.

2. The camera system of claim 1, wherein the tunable lens is a microfluidic lens.

3. The camera system of claim 2, wherein adjusting of the microfluidic lens causes a variation in a radiation pattern of the illumination output by the light source assembly.

4. The camera system of claim 3, wherein the variation in the radiation pattern includes an increase or a decrease of a width of a light beam formed by the illumination, and Wherein the light source assembly and camera are physically housed in a shared housing.

5. The camera system of claim 2, further comprising a second tunable lens positioned between the tunable lens and the light source, wherein the first and second tunable lenses are configured to operate in combination with one another to achieve zooming operation.

6. The camera system of claim 1, wherein the light source includes at least one light emitting diode (LED).

7. The camera system of claim 6, wherein the light source is an LED array including a plurality of LEDs, wherein the light source is also adjustable in that at least one of the LEDs of the LED array can be powered on or powered off, and wherein a change of direction of the illumination occurs when at least one of the LEDs is adjusted from being powered off to being powered on or from being powered on to being powered off.

8. The camera system of claim 7, wherein the plurality of LEDs of the LED array includes a ring of LEDs and wherein, by selecting one or more of the LEDs of the ring of LEDs to be powered on or off, a beam of light output by the light source can be effectively steered, the beam of light being at least some of the illumination.

9. The camera system of claim 7, wherein the plurality of LEDs of the LED array further includes a center LED positioned within the ring of LEDs, and wherein all of the LEDs are surface mounted on a printed circuit board.

10. The camera system of claim 1, wherein the camera is a miniature camera suitable for use in minimally-invasive surgery (MIS).

11. The camera system of claim 1, wherein a first physical location of the camera is independent from a second physical location of the light source assembly.

12. The camera system of claim 1, wherein the light source assembly further includes a second light source and a second tunable lens that is adjustable independently of adjustments to the first tunable lens.

13. The camera system of claim 1, wherein the camera includes a fluidic lens system.

14. The camera system of claim 1, further including a control and processing unit communicating via at least one wireless connection with the camera and the light source assembly, wherein the control and processing unit receives the image information from the camera via the at least one wireless connection.

15. The camera system of claim 14, wherein the control and processing unit includes at least one of:

means for processing the imaging information; and

means for receiving instructions from an operator according to which operations of the camera or the light source are controlled.

16. The camera system of claim 15, wherein image information is provided by the camera to a control and processing unit, and wherein a deblurring algorithm is employed by the control and processing unit so as to generate improved image information.

17. A method of operating a camera system to acquire image information representative of subject matter within a selected field of view, the method comprising:

providing a camera and a light source assembly, wherein the light source assembly includes a light source and a tunable lens;

controlling the tunable lens;

transmitting the light beam from the light source assembly; and

receiving reflected light at the camera, wherein at least some of the light of the light beam transmitted from the light source assembly is included as part of the reflected light, and wherein the image information is based at least in part upon the reflected light,

wherein the tunable lens is controlled to vary the light beam output by the light source assembly,

whereby the light beam can be varied to substantially match the selected field of view.

18. The method of claim 17, further comprising:

controlling an on/off status of at least one LED of an LED array that is included by the light source, so as to steer the light beam.

19. The method of claim 18, further comprising:

transmitting the image information or further information based thereon wirelessly from the camera to a control and processing unit that is in wireless communications with both the light source assembly and the camera.

20. The method of claim 19, further comprising:

processing the image information or further information at the control and processing unit to obtain processed image information.

21. A method for deblurring a blurred first image corresponding to a first color channel of a camera that also produces a sharp second image corresponding to a second color channel of the camera, wherein the first and the second images each include a plurality of pixels with each pixel having an associated respective value, the method comprising:

decomposing the first image by filtering and upsampling to generate a first set of one or more first sub-band output coefficients, wherein each first sub-band output coefficient corresponds to a respective sub-band in a selected one of a wavelet and a contourlet domain;

decomposing the second image by filtering and upsampling to generate a second set of second sub-band output coefficients, wherein each second sub-band output coefficient corresponds to a respective sub-band in the selected domain;

selecting those second sub-band output coefficients that represent edge information, each of the selected second sub-band output coefficients corresponding to a respective selected sub-band, with the selected sub-bands together defining an edge sub-band set;

preparing a third set of sub-band output coefficients which includes the selected second sub-band output coefficients representing edge information and at least one first sub-band output coefficient corresponding to a sub-band other than those sub-bands in the edge sub-band set; and

reconstructing a deblurred first image by upsampling and filtering using the third set of sub-band output coefficients as input.

22. The method of claim 21, wherein each decomposing step occurs over two or more levels and the reconstructing step occurs over the same number of levels.

23. The method of claim 22, wherein the number of levels of decomposition is determined based at least in part on a frequency response of a blurring lens of the camera.

24. The method of claim 21, the first set and the second set are combined to form the third set.

25. The method of claim 21, wherein the sub-bands corresponding to first set and the second set are predetermined and non-overlapping.

26. The method of claim 21, further including prefiltering the second image prior to decomposition of the second image.

27. The method of claim 26, wherein the prefiltering uses a Wiener filter.

28. The method of claim 21, wherein the decomposition steps occur in the contourlet domain and the selected second sub-hand output coefficients are those which have been determined to represent edge information.

29. The method of claim 28, wherein the edge information is determined using an ant colonization optimization scheme.

30. A method for deblurring a first image of a camera, wherein the camera generates the first image corresponding to a first color channel and a second image corresponding to a second color channel, wherein the first and the second images each includes a plurality of pixels with each pixel having an associated respective value, the method comprising:

selecting a filter bank having a decomposition portion and a reconstruction portion, the decomposition portion having at least two cascaded levels for receiving two inputs and generating intermediate sub-band output coefficients for each of a predetermined number N of sub-bands, the reconstruction portion having at least two cascaded levels for receiving the intermediate sub-band output coefficients and generating an output, wherein the decomposition portion includes multiple decomposition stages at a first level connecting to N decomposition stages at a second level, each decomposition stage including at least one of a decomposition filter and a downsampler, wherein the reconstruction portion includes N reconstruction stages at a third level connecting to N/2 reconstruction stages at a fourth level, each reconstruction stage including at least one of an upsampler and a reconstruction filter;

decomposing the first image using a first part of the decomposition portion of the filter bank to generate at least one first intermediate sub-band output coefficient corresponding to one of the N sub-bands;

decomposing the second image using a second part of the decomposition portion of the filter bank to generate at least a second, a third intermediate, and a fourth sub-band output coefficient, each corresponding to a respective one of the N sub-bands; and

reconstructing a deblurred image corresponding to the first color channel using at least the first, the second, the third, and the fourth sub-band output coefficients as input to the reconstruction side of the filter bank.

31. The method of claim 30, wherein each decomposing step occurs over two or more levels and the reconstructing step occurs over the same number of levels.

32. The method of claim 31, wherein the number of levels of decomposition is determined based at least in part on a frequency response of a blurring lens of the camera.

33. The method of claim 30, wherein the sub-bands corresponding to intermediate sub-band output coefficients of the first image are non-overlapping with the sub-bands corresponding to the intermediate sub-band output coefficients of the second image.

34. The method of claim 30, further including prefiltering the second image prior to decomposition of the second color image.

35. The method of claim 34, wherein the prefiltering uses a Wiener filter.

36. A method for &

blurring a blurred first image, wherein the blurred first image corresponds to a first color channel of a camera and a second image corresponds to a second color channel of the camera, wherein the first and the second images each comprise a plurality of pixels with each pixel having an associated respective value, the method comprising;

decomposing the first image by filtering and downsampling in a decomposition portion of a filter bank to generate for a predetermined number of sub-bands in a contourlet domain a first set of first sub-hand output coefficients, wherein each first sub-band output coefficient corresponds to a respective one of the sub-bands;

decomposing the second image by filtering and downsampling in the decomposition portion of the filter bank to generate for the predetermined number of sub-bands a second set of second sub-band output coefficients, wherein each second sub-band output coefficient corresponds to a respective one of the sub-bands in the contourlet domain;

determining which of the second sub-hand output coefficients and corresponding respective sub-bands represent edge information;

preparing a set of third sub-band output coefficients by modifying the first set of first sub-band output coefficients to replace each of those first sub-band output coefficients in the first set which correspond to a respective determined sub-band representing edge information with the corresponding second sub-band output coefficient from the second set; and

reconstructing a deblurred image corresponding to the first image by upsampling and filtering the set of third sub-band output coefficients in a reconstruction portion of the filter bank.

37. The method of claim 36, further including combining the deblurred image with the second image to generate a composite color image.

38. The method of claim 36, wherein the edge information is determined using an ant colonization optimization scheme.

39. The method of claim 38, further including preparing a binary edge map in the contourlet domain and using the binary edge map in the preparing step to select corresponding second sub-band output coefficients from the second set.

40. The method of claim 39, further including preparing a binary edge map in the contourlet domain, dilating the binary edge map, and using the dilated binary edge map in the preparing step to select corresponding second sub-band output coefficients from the second set.

41. A light source assembly comprising:

an output port;

a light source; and

a tunable lens positioned between the first light source and the output port,

wherein the light source generates light that passes through the tunable lens and then exits the light source assembly via the output port as output light, and

wherein the tunable lens is adjustable to vary a characteristic of the output light exiting the light source assembly via the output port.

42. The light source assembly of claim 41, wherein the light source includes an array of LEDs, wherein each of the LEDs can be controlled to he turned on or turned off so as to determine a predominant direction of the output light exiting the light source assembly.