Real-time contemporaneous multimodal imaging and spectroscopy uses thereof

• US 20040225222A1
• Filed: 05/08/2003
• Published: 11/11/2004
• Est. Priority Date: 05/08/2003
• Status: Abandoned Application

• Current Assignee: British Columbia Cancer Agency, Perceptronix Medical Inc.
• Original Assignee: British Columbia Cancer Agency, Perceptronix Medical Inc.
• Inventors: Ferguson, Gary, Zeng, Haishan, Palcic, Branko, Petek, Mirjan
• Application Number:
  US10/431,939
• Publication Number:
  US 20040225222A1
• Time in Patent Office: Days
• US Class Current: 600/476
• CPC Class Codes: A61B 1/043 for fluorescence imaging
• CPC Class Codes: A61B 1/0646 with illumination filters
• CPC Class Codes: A61B 1/0676 at distal tip of an endoscope
• CPC Class Codes: A61B 1/0684 using light emitting diodes [LED]
• CPC Class Codes: A61B 5/0071 by measuring fluorescence emission
• CPC Class Codes: A61B 5/0075 by spectroscopy, i.e. measuring spectra, e.g. Raman spectroscopy, infrared absorption spectroscopy A61B5/0071 takes precedence
• CPC Class Codes: A61B 5/0084 for introduction into the body, e.g. by catheters A61B5/1459 takes precedence
• CPC Class Codes: G01J 2003/1213 Filters in general, e.g. dichroic, band
• CPC Class Codes: G01J 3/2803 using photoelectric array detector
• CPC Class Codes: G01J 3/32 Investigating bands of a spectrum in sequence by a single detector
• CPC Class Codes: G01J 3/36 Investigating two or more bands of a spectrum by separate detectors
• CPC Class Codes: G01J 3/4406 Fluorescence spectrometry

Abstract

The present invention comprises an optical apparatus, methods and uses for real-time (video-rate) multimodal imaging, for example, contemporaneous measurement of white light reflectance, native tissue autofluorescence and near infrared images with an endoscope. These principles may be applied to various optical apparati such as microscopes, endoscopes, telescopes, cameras etc. to view or analyze the interaction of light with objects such as planets, plants, rocks, animals, cells, tissue, proteins, DNA, semiconductors, etc. Multi-band spectral images may provide morphological data such as surface structure of lung tissue whereas chemical make-up, sub-structure and other object characteristics may be deduced from spectral signals related to reflectance or light radiated (emitted) from the object such as luminescence or fluorescence, indicating endogenous chemicals or exogenous substances such as dyes employed to enhance visualization, drugs, therapeutics or other agents. Accordingly, one embodiment of the present invention discusses simultaneous white light reflectance and fluorescence imaging. Another embodiment describes the addition of another reflectance imaging modality (in the near-IR spectrum). Input (illumination) spectrum, optical modulation, optical processing, object interaction, output spectrum, detector configurations, synchronization, image processing and display are discussed for various applications.

Claims

1. An optical apparatus for simultaneous measurement of images, comprising

a light source delivering interrogating broadband radiation,

a first optical modulator separating said interrogating broadband radiation into a plurality of interrogating spectral segments corresponding to a plurality of imaging modalities,

a target object to interact with said interrogating spectral segments to produce returning radiation for imaging,

a second optical modulator separating said returning radiation from the target object into a plurality of returning spectral segments corresponding to said plurality of imaging modalities, and

at least one detector to receive said returning spectral segments and to process said returning spectral segments into images.

2. The apparatus of claim 1, further comprising means to synchronize said first optical modulator and said second optical modulator.

3. The apparatus of claim 1, further comprising means to display said images.

4. The apparatus of claim 3, wherein said means to display includes means to align said images.

5. The apparatus of claim 4, wherein said means to align comprises x-y pixel shifting.

6. The apparatus of claim 1, further comprising means to synchronize said first optical modulator and said second optical modulator and means to display said images.

7. The apparatus of claim 1, wherein said light source comprises a plurality of light emitting diodes in an endoscope.

8. The apparatus of claim 7, wherein said first optical modulator separates said interrogating broadband radiation by synchronized electronic switching of said light emitting diodes.

9. The apparatus of claim 1, wherein said first optical modulator comprises a moving filter having a plurality of modulating components corresponding to said plurality of imaging modalities, said components each having a duty cycle consisting of a ratio of said broadband radiation separated by said component to said broadband radiation.

10. The apparatus of claim 9, wherein said moving filter comprises a filter wheel.

11. The apparatus of claim 10, wherein said filter wheel revolves at approximately 1800 rpm.

12. The apparatus of claim 10, wherein said plurality of modulating components comprises at least a color balance filter component and a fluorescent excitation filter component.

13. The apparatus of claim 12, wherein said duty cycle for said fluorescent excitation filter component is less than or equal to approximately 50 percent.

14. The apparatus of claim 12, wherein the duty cycle for said fluorescent excitation filter is greater than 50 percent.

15. The apparatus of claim 12, wherein said fluorescent excitation filter component corresponds to said interrogating spectral segments having a spectral range of approximately 400 to 450 nm.

16. The apparatus of claim 12, wherein said fluorescent excitation filter component corresponds to said interrogating spectral segments having a spectral range of approximately 700 to 750 nm.

17. The apparatus of claim 12, wherein said fluorescent excitation filter component corresponds to said interrogating spectral segments having spectral ranges of approximately 400 to 450 nm and 700 to 750 nm.

18. The apparatus of claim 10, wherein said filter wheel includes a light-blocking area.

19. The apparatus of claim 10, wherein said moving filter comprises at least a color balance filter and a fluorescence excitation filter.

20. The apparatus of claim 9, wherein said moving filter includes a light-blocking area.

21. The apparatus of claim 1, wherein said first optical modulator comprises an optical switching device.

22. The apparatus of claim 21, wherein said optical switching device comprises a spatial light modulator.

23. The apparatus of claim 22, wherein said spatial light modulator comprises a liquid crystal device.

24. The apparatus of claim 22, wherein said spatial light modulator comprises a digital micro-mirror device.

25. The apparatus of claim 1, further comprising at least at least a white light detector and a fluorescence detector.

26. The apparatus of claim 1, wherein said second optical modulator comprises a moving filter having a plurality of modulating components corresponding to said plurality of imaging modalities, said components having a duty cycle consisting of a ratio of said broadband radiation separated by said component to said broadband radiation.

27. The apparatus of claim 26, wherein said moving filter comprises a filter wheel.

28. The apparatus of claim 27, wherein said filter wheel revolves at approximately 1800 rpm.

29. The apparatus of claim 26, wherein said plurality of modulating components comprises at least a color balance filter component and a fluorescent reflection filter component.

30. The apparatus of claim 29, wherein said duty cycle for said fluorescent reflection filter component is less than or equal to approximately 50 percent.

31. The apparatus of claim 29, wherein the duty cycle for said fluorescent reflection filter is greater than 50 percent.

32. The apparatus of claim 29, wherein said fluorescent excitation filter component corresponds to said interrogating spectral segments having a spectral range of approximately 400 to 450 nm.

33. The apparatus of claim 29, wherein said fluorescent reflection filter component provides near 100 percent reflection in a spectral range of approximately 300 to 800 nm.

34. The apparatus of claim 29, wherein said fluorescent excitation filter component corresponds to said interrogating spectral segments having spectral ranges of approximately 400 to 450 nm and 700 to 750 nm.

35. The apparatus of claim 27, wherein said filter wheel includes a light-blocking area.

36. The apparatus of claim 26, wherein said moving filter includes a light-blocking area.

37. The apparatus of claim 1, wherein said second optical modulator comprises an optical switching device.

38. The apparatus of claim 37, wherein said optical switching device comprises a spatial light modulator.

39. The apparatus of claim 38, wherein said spatial light modulator comprises a liquid crystal device.

40. The apparatus of claim 38, wherein said spatial light modulator comprises a digital micro-mirror device.

41. The apparatus of claim 1, wherein said at least one detector comprises at least a spectrometer.

42. A method of simultaneously measuring images, comprising

producing interrogating broadband radiation,

separating said interrogating broadband radiation into a plurality of interrogating

spectral segments corresponding to a plurality of imaging modalities,

interacting said interrogating spectral segments with a target object to produce returning radiation,

separating said returning radiation into a plurality of returning spectral segments corresponding to said plurality of imaging modalities, and

processing said returning spectral segments into images.

43. The method of claim 42, further comprising synchronizing said step of separating said interrogating broadband radiation with said step of separating said returning radiation.

44. The method of claim 43, further comprising displaying said images.

45. The method of claim 42, further comprising displaying said images.

46. The method of claim 42, further comprising aligning said images.

47. The method of claim 46, further comprising x-y pixel shifting.

48. The method of claim 42, wherein said method is applied to observation of said target object through blood.

49. The method of claim 42, further comprising producing broadband radiation from light-emitting diodes.

50. An optical apparatus for simultaneous measurement of images, comprising

a light source delivering interrogating broadband radiation,

an optical modulator separating said interrogating broadband radiation into a plurality of interrogating spectral segments corresponding to a plurality of imaging modalities,

a target object to interact with said interrogating spectral segments to produce returning radiation, and

a detector to receive and process said returning radiation, comprising

means to separate said returning radiation into a plurality of returning spectral segments corresponding to said plurality of imaging modalities, and

means to process said returning spectral segments into images.

51. The apparatus of claim 50, further comprising means to synchronize said optical modulator and said detector.

52. The apparatus of claim 51, further comprising means to display said images.

53. The apparatus of claim 50, further comprising means to display said images.

54. The apparatus of claim 53, wherein said means to display includes means to align said images.

55. The apparatus of claim 54, wherein said means to align comprises x-y pixel shifting.

56. The apparatus of claim 50, wherein said light source comprises a plurality of light emitting diodes in an endoscope.

57. The apparatus of claim 56, wherein said optical modulator separates said interrogating broadband radiation by synchronized electronic switching of said light emitting diodes.

58. The apparatus of claim 50, wherein said optical modulator comprises a moving filter having a plurality of modulating components corresponding to said plurality of imaging modalities, said components having a duty cycle consisting of a ratio of said broadband radiation separated by said component to said broadband radiation.

59. The apparatus of claim 58, wherein said moving filter comprises a filter wheel.

60. The apparatus of claim 59, wherein said filter wheel revolves at approximately 1800 rpm.

61. The apparatus of claim 59, wherein said plurality of modulating components comprises at least a color balance filter component and a fluorescent excitation filter component.

62. The apparatus of claim 61, wherein said duty cycle for said fluorescent excitation filter component is less than or equal to approximately 50 percent.

63. The apparatus of claim 61, wherein the duty cycle for said fluorescent excitation filter is greater than 50 percent.

64. The apparatus of claim 61, wherein said fluorescent excitation filter component corresponds to said interrogating spectral segments having a spectral range of approximately 400 to 450 nmn.

65. The apparatus of claim 61, wherein said fluorescent excitation filter component corresponds to said interrogating spectral segments having a spectral range of approximately 700 to 750 nm.

66. The apparatus of claim 61, wherein said fluorescent excitation filter component corresponds to said interrogating spectral segments having spectral ranges of approximately 400 to 450 nm and 700 to 750 nm.

67. The apparatus of claim 59, wherein said filter wheel includes a light-blocking area.

68. The apparatus of claim 58, wherein said moving filter includes a light-blocking area.

69. The apparatus of claim 50, wherein said optical modulator comprises an optical switching device.

70. The apparatus of claim 69, wherein said optical switching device comprises a spatial light modulator.

71. The apparatus of claim 70, wherein said spatial light modulator comprises a liquid crystal device.

72. The apparatus of claim 70, wherein said spatial light modulator comprises a digital micro-mirror device.

73. The apparatus of claim 50, wherein said detector comprises at least a white light detector and a fluorescence detector.

74. The apparatus of claim 50 wherein said means to separate comprises a plurality of dichroic mirrors.

75. The apparatus of claim 74, further comprising a plurality of filters.

76. The apparatus of claim 75, wherein said plurality of filters comprise at least one band pass filter.

77. The apparatus of claim 75, wherein said plurality of filters comprise at least one long pass filter.

78. The apparatus of claim 75, wherein said plurality of filters comprise at least one band pass filter and at least one long pass filter.

79. The apparatus of claim 74, further comprising a plurality of lenses.

80. The apparatus of claim 79, wherein said plurality of lenses focus said plurality of returning, spectral segments onto a corresponding plurality of sensors.

81. The apparatus of claim 80, wherein said sensors have gain and said gain is adjustable based on said imaging modalities.

82. The apparatus of claim 80, wherein said plurality of sensors comprise CCDs.

83. The apparatus of claim 50, further comprising means to adjust intensity of said returning spectral segments.

84. The apparatus of claim 83, wherein said means to adjust is said optical modulator.

85. The apparatus of claim 58, further comprising means to adjust intensity of said returning spectral segments based on said duty cycle.

86. The apparatus of claim 50, further comprising a frame sensor to synchronize said optical modulator and said detector.

87. The apparatus of claim 50, wherein said means to process comprises

a switch to assign said returning spectral segments to one of a plurality of analog to digital converters, wherein said plurality of analog to digital recorders digitize said returning spectral segments, and

a gate array to process said digitized returning spectral segments into images.

88. The apparatus of claim 87, wherein said gate array aligns said images.

89. The apparatus of claim 88, wherein said gate array aligns said images by x-y pixel shifting.

90. The apparatus of claim 88, wherein said gate array aligns said images by measuring ratios of corresponding pixels in a plurality of said returning spectral segments.

91. A method of simultaneously measuring images, comprising

producing interrogating broadband radiation,

separating said interrogating broadband radiation into a plurality of interrogating spectral segments corresponding to a plurality of imaging modalities,

interacting said interrogating spectral segments with a target object to produce returning radiation, and

detecting said returning radiation, comprising

separating said returning radiation into a plurality of interrogating spectral segments corresponding to a plurality of imaging modalities, and

processing said returning spectral segments into images.

92. The method of claim 91, further comprising synchronizing said separating said interrogating broadband radiation step and said detecting step.

93. The method of claim 92, further comprising displaying said images.

94. The method of claim 91, further comprising displaying said images

95. The method of claim 91, further comprising aligning said images.

96. The method of claim 95, wherein said aligning step comprises x-y pixel shifting.

97. The method of claim 91, wherein said method is applied to observation of said target object through blood.

98. The method of claim 91, further comprising producing said broadband radiation by light emitting diodes.

99. The method of claim 98, further comprising producing said broadband radiation in an endoscope.

100. The apparatus of claim 99, further comprising separating said interrogating broadband radiation by synchronized electronic switching of said light emitting diodes.

101. The method of claim 100, wherein said method is applied to observation of said target object through blood.

102. An optical apparatus for simultaneous measurement of white light and fluorescence images, comprising

means for providing a desired illumination,

means for modulating said illumination at video-rate for real-time imaging,

means for producing images by interaction of said illumination with a target object,

means for separating said images at video-rate,

means for detecting said separated images,

means for processing said detected separated images,

means for controlling detection and processing of said separated images, and

means for displaying at least one of said processed images.

103. An endoscope for obtaining contemporaneous images, comprising

a probe having an inner end to be located within a body, and an outer end to extend outside said body,

a light source to produce interrogating broadband radiation,

an optical modulator separating said interrogating broadband radiation into a plurality of interrogating spectral segments corresponding to a plurality of imaging modalities,

a target object within said body to interact with said interrogating spectral segments to produce returning radiation for imaging,

a lens to focus said returning radiation on a CCD sensor,

104. The apparatus of claim 103, wherein said light source is connected to said outer end and further comprising an illumination fiber bundle conducting said broadband radiation from said light source.

105. The apparatus of claim 103, wherein said CCD sensor comprises at least one set of pixels, wherein each said pixel within each said at least one set is coated with one of a band pass filter passing blue light, a band pass filter passing green light, a band pass filter passing red light, and a band pass filter passing near infrared light.

106. The apparatus of claim 105, further comprising means for generating a blue channel image from said blue light, a green channel image from said green light, a red channel image from said red light, and a near infrared image from said near infrared light.

107. The apparatus of claim 106, further comprising means to display said channel images.

108. The apparatus of claim 107, wherein said CCD sensor comprises at least one set of pixels, wherein each said pixel within each said at least one set is coated with one of a band pass filter passing blue light, a band pass filter passing green light, and a band pass filter passing red light.

109. The apparatus of claim 108, further comprising means for generating a blue channel image from said blue light, a green channel image from said green light, and a red channel image from said red light.

110. The apparatus of claim 109, further comprising means to display said channel images.

111. The apparatus of claim 103, wherein said optical modulator comprises a moving filter having a plurality of modulating components corresponding to said plurality of imaging modalities, said components each having a duty cycle consisting of a ratio of said broadband radiation separated by said component to said broadband radiation.

112. The apparatus of claim 111, wherein said moving filter comprises a filter wheel.

113. The apparatus of claim 112, wherein said filter wheel revolves at approximately 1800 rpm.

114. The apparatus of claim 112, wherein said plurality of modulating components comprises at least a color balance filter component and a fluorescent excitation filter component.

115. The apparatus of claim 114, wherein said duty cycle for said fluorescent excitation filter component is less than or equal to approximately 50 percent.

116. The apparatus of claim 114, wherein the duty cycle for said fluorescent excitation filter is greater than 50 percent.

117. The apparatus of claim 114, wherein said fluorescent excitation filter component corresponds to said interrogating spectral segments having a spectral range of approximately 400 to 450 mm.

118. The apparatus of claim 114, wherein said fluorescent excitation filter component corresponds to said interrogating spectral segments having a spectral range of approximately 700 to 750 mm.

119. The apparatus of claim 114, wherein said fluorescent excitation filter component corresponds to said interrogating spectral segments having spectral ranges of approximately 400 to 450 nm and 700 to 750 mm.

120. The apparatus of claim 112, wherein said filter wheel includes a light-blocking area.

121. The apparatus of claim 111, wherein said moving filter comprises at least a color balance filter and a fluorescence excitation filter.

122. The apparatus of claim 111, wherein said moving filter includes a light-blocking area.

123. The apparatus of claim 103, wherein said optical modulator comprises an optical switching device.

124. The apparatus of claim 123, wherein said optical switching device comprises a spatial light modulator.

125. The apparatus of claim 124, wherein said spatial light modulator comprises a liquid crystal device.

126. The apparatus of claim 124, wherein said spatial light modulator comprises a digital micro-mirror device.

127. An endoscope for obtaining contemporaneous images, comprising

a probe having an inner end to be located within a body, and an outer end to extend outside said body,

a light source comprising a plurality of light-emitting diodes located as said inner end and producing interrogating radiation, wherein said light-emitting diodes are electronically switched to produce a plurality of interrogating spectral segments corresponding to a plurality of imaging modalities,

a target object within said body to interact with said interrogating spectral segments to produce returning radiation for imaging,

a lens to focus said returning radiation on a CCD sensor,

wherein said CCD sensor comprises at least one set of pixels and said set of pixels captures a plurality of spectral bands of said image.

128. The apparatus of claim 127, wherein each said set of pixels comprises

a pixel coated with a band pass filter passing blue light,

a pixel coated with a band pass filter passing green light,

a pixel coated with a band pass filter passing red light, and

a pixel coated with a band pass filter passing near infrared light.

129. The apparatus of claim 128, further comprising means for generating a blue channel images from said blue light, a green channel image from said green light, a red channel image from said red light, and a near infrared image from said near infrared light.

130. The apparatus of claim 129, further comprising means to display said channel images.

131. The apparatus of claim 127, wherein each said set of pixels comprises

a pixel coated with a band pass filter passing blue light,

a pixel coated with a band pass filter passing green light, and

a pixel coated with a band pass filter passing red light.

132. The apparatus of claim 131, further comprising means for generating a blue channel image from said blue light, a green channel image from said green light, and a red channel image from said red light.