ENDOSCOPE EMPLOYING STRUCTURED LIGHT PROVIDING PHYSIOLOGICAL FEATURE SIZE MEASUREMENT
1. A method for imaging a body lumen comprising:
- introducing an imaging apparatus into the body lumen;
emitting, from the imaging apparatus, structured light and non-structured light into the body lumen;
detecting, by the imaging apparatus, both structured light and non-structured light reflected from anatomical features in the body lumen;
generating, by the imaging apparatus, images from the detected light.
Disclosed herein are systems, methods, and structures providing accurate and easy to use size measurement of physiological features identified from endoscopic examination. In sharp contrast to the prior art, systems, methods, and structures according to the present disclosure employ structured light that advantageously enables size and/or distance information about lesions and/or other physiological features in a gastrointestinal (GI) tract. Advantageously, systems, methods, and structures according to the present disclosure are applicable to both capsule endoscopes and insertion endoscopes.
- 1. A method for imaging a body lumen comprising:
introducing an imaging apparatus into the body lumen; emitting, from the imaging apparatus, structured light and non-structured light into the body lumen; detecting, by the imaging apparatus, both structured light and non-structured light reflected from anatomical features in the body lumen; generating, by the imaging apparatus, images from the detected light.
- View Dependent Claims (2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18)
This application is a continuing application of U.S. patent application Ser. No. 15/927,856 filed 21 Mar. 2018, which in turn is a continuation-in-part application of U.S. patent application Ser. No. 14/884,788 filed 16 Oct. 2015 now issued on 3 Apr. 2018 as U.S. Pat. No. 9,936,151, both of which are incorporated by reference as if set forth at length herein.
This disclosure relates generally to endoscopic examination of body lumens and more specifically to endoscopes and endoscopic examination employing structured light to facilitate the accurate dimensional measurement of lesions or other features observed during such examination.
As is known, endoscopes—including capsule endoscopes—allow clinicians to find and identify lesions and other physiological features in a gastrointestinal (GI) tract. Such capsule endoscopes are capsule shaped—having a tubular body with end structures giving them their capsule shape—and may advantageously be swallowed or taken into a stomach by traversing the throat and esophagus with a voluntary muscular action, as food, drink, or other substances. From the stomach, the capsule proceeds through the intestines and subsequently exits. Subsequent diagnosis oftentimes includes an estimation of the size of the lesion/feature since any health risk posed by the lesion/feature and any subsequent treatment regime(s) often depend on its size. For example, adenomas and sessile serrated polyps in a colon are typically categorized as advanced precancerous lesions if they measure more than 1 cm in diameter.
Despite the recognized importance of physiological feature size measurement, contemporary endoscopes—particularly capsule endoscopes—lack an accurate and easy to use method of size measurement for such physiological feature(s). Accordingly, methods, systems, and structures that provide or otherwise facilitate the size measurement of such physiological features identified from endoscopic examination would represent a welcome addition to the art.
An advance in the art is made according to aspects of the present disclosure directed to methods, systems and structures providing accurate and easy to use size measurement of physiological features identified from endoscopic examination.
In sharp contrast to the prior art, systems, methods, and structures according to the present disclosure employ structured light that advantageously enables size and/or distance information about lesions and/or other physiological features in a gastrointestinal (GI) tract.
Advantageously, systems, methods and structures according to the present disclosure are applicable to both capsule endoscopes and insertion endoscopes.
Viewed from one aspect, the present disclosure is directed to endoscope systems including: a housing; at least one camera; a structured light source; and an array of microlenses that produces the structured light, the array of microlenses positioned such that light emitted from the structured light source is collimated by the microlenses into an array of beams propagating in multiple directions.
Viewed from another aspect, the present disclosure is directed to method(s) for imaging a body lumen comprising: introducing an imaging apparatus into the body lumen; emitting, from the imaging apparatus, non-structured light into the body lumen; detecting, by the imaging apparatus, non-structured light reflected from anatomical features in the body lumen; generating, by the imaging apparatus, one or more non-structured light images from the detected non-structured light; projecting structured light into the body lumen; detecting structured light reflected from the anatomical features in the body lumen; and generating one or more structured light images from the detected structured light.
A more complete understanding of the present disclosure may be realized by reference to the accompanying drawing in which:
Illustrative embodiments are described more fully by the Figures and detailed Description. Embodiments according to this disclosure may, however, be embodied in various forms and are not limited to specific or illustrative embodiments described in the Drawing and detailed Description.
The following merely illustrates the principles of the disclosure. It will thus be appreciated that those skilled in the art will be able to devise various arrangements which, although not explicitly described or shown herein, embody the principles of the disclosure and are included within its spirit and scope.
Furthermore, all examples and conditional language recited herein are principally intended expressly to be only for pedagogical purposes to aid the reader in understanding the principles of the disclosure and the concepts contributed by the inventor(s) to furthering the art and are to be construed as being without limitation to such specifically recited examples and conditions.
Moreover, all statements herein reciting principles, aspects, and embodiments of the disclosure, as well as specific examples thereof, are intended to encompass both structural and functional equivalents thereof. Additionally, it is intended that such equivalents include both currently known equivalents as well as equivalents developed in the future, i.e., any elements developed that perform the same function, regardless of structure.
Thus, for example, it will be appreciated by those skilled in the art that any block diagrams herein represent conceptual views of illustrative circuitry embodying the principles of the disclosure. Similarly, it will be appreciated that any flow charts, flow diagrams, state transition diagrams, pseudo code, and the like represent various processes which may be substantially represented in computer readable medium and so executed by a computer or processor, whether such computer or processor is explicitly shown.
The functions of the various elements shown in the Drawing, including any functional blocks labeled as “processors”, may be provided using dedicated hardware as well as hardware capable of executing software in association with appropriate software. When provided by a processor, the functions may be provided by a single dedicated processor, by a single shared processor, or by a plurality of individual processors, some of which may be shared. Moreover, explicit use of the term “processor” or “controller” should not be construed to refer exclusively to hardware capable of executing software, and may implicitly include, without limitation, digital signal processor (DSP) hardware, network processor, application specific integrated circuit (ASIC), field programmable gate array (FPGA), read-only memory (ROM) for storing software, random access memory (RAM), and non-volatile storage. Other hardware, conventional and/or custom, may also be included.
Software modules, or simply modules which are implied to be software, may be represented herein as any combination of flowchart elements or other elements indicating performance of process steps and/or textual description. Such modules may be executed by hardware that is expressly or implicitly shown.
Unless otherwise explicitly specified herein, the FIGs comprising the drawing are not drawn to scale.
By way of some additional background, we again note that—despite the importance of measuring the size(s) of physiological feature(s) that may be identified from endoscopic examination—contemporary endoscopes—including capsule endoscopes—do not adequately provide such size measuring capability. Note that for brevity, we may interchangeably use the terms “features” or “lesion” to describe such physiological features. We note further that for the purposes of this disclosure and claims, such feature or lesion is simply an object or point of interest in a field of view and no nomenclature used herein is to be considered as limiting.
As will be known and readily understood by those skilled in the art, size measurement/estimation of an endoscopic camera image is error prone as the apparent size of an object or physiological feature to be measured depends upon its magnification, which in turn depends on its distance from the camera—which is generally not known. When an object is close to a camera, i.e., small conjugate distance, as is necessarily the case for in-vivo imaging, small changes in object distance produce large changes in magnification. Moreover, for the wide-angle lenses employed in endoscopes, lens distortion produces magnification variation across the camera field of view.
Those skilled in the art will readily appreciate that a tool (i.e., scale, forceps, or other object of a known size) of some sort may be used during an endoscopic examination as a size reference by positioning it sufficiently proximate to a lesion and viewing the tool and the lesion to provide a size reference to better estimate/determine the size of the lesion. As will be further appreciated, such a procedure may be time consuming, difficult or impossible for certain lesion positions in the bowel, not sufficiently accurate, present increased procedural risk of mucosal damage, and does not automatically record a measurement into a storage medium as part of the procedure record. Moreover, such tools are not available for capsule endoscopic procedures.
Of course, an endoscope including such tool that measures distance would enable a fast, simple, and objective measurement and recording of the size of objects in the gastrointestinal (GI) tract observed during endoscopic examination and would be a welcome addition to the art.
It is noted at this point that various electronic technologies have been developed for measuring the distance of objects, including radar, ultrasound, sonar, echo-location, lidar, holography, stereo-vision, depth-from-shading, time-of-flight, optical coherence tomography, confocal microscopy, and structured light. Many of these technologies require large, complicated, expensive, and power-hungry systems, methods, and structures. Optical time-of-flight measurements—including lidar—are challenging to employ for short object distance(s) because the time of flight is quite brief and therefore millimeter scale distance resolution is difficult to achieve. Optical coherence tomography (OCT) and confocal microscopy have been used in endoscopes and procedures employing same but are insufficiently miniaturized to provide utility for non-tethered, capsule endoscope applications. Finally, many of these noted technologies require sensor hardware that operates separately from optical white-light (WL) cameras employed by gastroenterologists or others (who employ endoscopes—i.e., endoscopists) to identify lesions and other features, making the correspondence between camera-image data and 3D-sensor data challenging.
Advantageously—and as will be readily appreciated by those skilled in the art and according to aspects of the present disclosure—3D data may be acquired by triangulation between an endoscope camera and a projector of structured light (SL). As will be further appreciated, such an approach leverages the camera—thereby reducing any extra hardware required and simplifying the establishment of a correspondence between white light image data and depth measurements.
As is used herein, structured light is spatially patterned so that an object space is illuminated with a pattern of known geometry in 3D space. Structured light involves a process of projecting a known pattern (oftentimes including grid or horizontal bar pattern elements, or random, pseudorandom, or semi-periodic elements) onto a scene (e.g., any field of view of an endoscope camera). The way(s) in which pattern elements deform when striking surfaces allows the determination of the depth and surface information of objects in the scene, as used by structured light 3D scanning systems. Advantageously, structured light may employ invisible (or imperceptible) structured light without interfering with other light-based vision systems and/or processes for which the projected pattern may be confusing. Illustrative invisible structured light includes the use of infrared light or of sufficiently high frame rates alternating between different patterns, i.e., opposite patterns.
By way of some specific examples, we note that structured light may employ an array of beams—emitted from one or more centers of projection (COP)—creating a grid of spots, lines, or other patterns on an illuminated surface. For triangulation, as will be known and understood by those skilled in the art, the COPs of the structured light projections must not be co-located with the COP of a camera imaging the pattern/surface.
Structured light may be generated by an image protector (structured light projector) that projects an image from a spatial light modulator or “slide” onto a surface. With respect to an in-vivo application, such surface may generally include biological materials including mucosa and/or lesions and/or other physiological features. However, optical efficiency of such image projectors decreases with the size of the system. For a single aperture image projector, flux is proportional to the focal-length squared. Since endoscopes typically exhibit a large field of view (FOV), e.g., 160°, it is difficult for image projectors to cover such a large FOV. Similarly, alternative technologies including miniature optical scanners using—for example—micro-electro-mechanical systems (MEMS) mirrors—cannot easily cover such a large FOV. Still alternative technologies such as diffraction gratings or holograms that operate by passing coherent light through a diffractive optical element (DOE)—while they may generate spatial patterns—such patterns however, only exhibit sufficient contrast in the diffraction far-field—typically at a distance of at least 1 cm from the DOE—and uniform coverage of a FOV exceeding 60° is difficult to achieve from a single source. Still another technological approach for generating structured light employing focusing lens(es) positioned at a focal length beyond the DOE produces images in the diffraction far-field at a distance equal to twice the focal length of the lens(es) from the DOE but results in a reduced coverage area (FOV) of the image.
Given these and other difficulties, the present disclosure is directed to systems, methods, and structures for the generation of structured light in a constrained spatial volume, exhibiting sufficiently low power consumption and low cost yet well suited for the illumination of object distances in the range of millimeters and beyond. As we shall show and describe, such systems, methods, and structures according to the present disclosure are particularly attractive to endoscopic applications and more particularly in-vivo capsule endoscopes. Notwithstanding such attractiveness, systems, methods, and structures according to the present disclosure advantageously exhibit applicability to other applications as well.
As will become apparent to those skilled in the art and for the purposes of presenting an elementary analogy, systems, methods, and structures according to the present disclosure employ a variation of a basic concept of casting shadows with a shadow mask. More particularly—and according to the present disclosure—light is passed through an array of apertures including micro-lenses that collimate the light into an array of beams—the intensity of which decreases less rapidly with distance than light passing through apertures without collimating lenses. Since each beam is independently collimated, the beam angles may vary widely to cover a larger FOV. Additionally, mirrors, lens(es), or other optical elements positioned beyond the micro-lenses may be employed to redirect some—or all—of the beams and increase FOV—as we shall show and describe in greater detail later in this disclosure.
At this point we note that those skilled in the art will readily understand and appreciate that a collimator is a device that narrows one or more beam(s) of light. As used herein, collimate means to narrow or cause direction(s) of light to become more aligned in a specific direction (i.e. that light rays describing the beam become more parallel), however, it does not mean that no divergence or convergence occurs in the collimated beam. It is possible that collimation may result in beams that have a smaller spatial cross section.
As will become further apparent to those skilled in the art, much of the disclosure presented herein is illustratively presented in the context of capsule endoscopes. The disclosure is not so limited. Systems, methods, and structures according to the present disclosure contemplate capsule and insertion-tube type endoscopes, as well as other instrumentation that may benefit from size and/or distance determination of objects of interest in a scene.
Turning now to
As will be further appreciated by those skilled in the art, as the divergence angle increases, the greater the intensity of the light decreases with distance, thus requiring greater dynamic range in an image to adequately detect the presence and location of any projected patterns (spots, etc.) on both distant and near objects. As used herein and as generally understood by those skilled in the art, dynamic range describes a ratio between maximum and minimum measurable light intensities (e.g., black and white).
Reducing a and D (narrowing the mask apertures) reduces the divergence at the expense of throughput. Also, diffraction limits how much the divergence can be reduced by reducing D. Also, for projected spots to be distinguishable from neighboring spots, an aperture duty cycle of at least approximately 50% is required (i.e., the mask aperture pitch is at last 2D). Note that the shadow mask shown in the figure exhibiting square mask apertures must be substantially 50% opaque along axes in both lateral directions such that only approximately 25% of incident light striking the mask is passed. Such criteria are less for circular mask apertures.
With reference now to
While not yet specifically shown in the figures, a light source may advantageously include a “point-source” light-emitting diode (LED), which is an LED having a small aperture.
In an illustrative embodiment, a point-source LED exhibits a structure similar to that of a standard LED. However, the light emitted therefrom is emitted through a well-defined, (often circular) area, typically 25 μm˜200 μm in diameter. The light so produced will appear as a “spot” producing a narrow viewing angle. As will be appreciated, such a point-source LED may eliminate or change the requirements of the source aperture (and any source mask having the source aperture) illustratively shown. (In such case, a is equivalent to an aperture diameter of the point source.) Typically, a lateral current-confinement structure is included in an LED such that an area in which electrons and holes recombine therein is not much larger than the aperture. The aperture may an opening in a metal layer on the surface of the LED.
Of course, a source employed in systems, structures, and methods according to the present disclosure may also comprise a laser, including a vertical-cavity surface-emitting laser (VCSEL) which may have an aperture of 10 μm or less, and is known to be much more efficient that a point-source LED. Unfortunately, if such a laser is highly coherent, the generated structured light may include spurious interference patterns and speckle noise.
For a point-source LED, a would typically be in the range of 0.050 mm to 0.20 mm (e.g., 0.080 mm or 0.10 mm) and L would typically be in the range of 1 mm to 10 mm. As such, for a=0.80 mm and L=4.0 mm, 0=6°. So long as D>a, the beam divergence θ is less than the beam separation angle ϕ and the duty cycle of any spots projected on an object decreases with object distance, even if the lens duty cycle is 100% (i.e., the pitch equals D). Such a configuration is shown schematically in
At this point, note that a lens array such as that shown in the figures may be a micro lens array (MLA) formed or otherwise positioned on a transparent substrate. Such configuration is illustratively shown in the schematic diagram of
With continued reference to that figure, it is noted that the substrate may be constructed from any of a variety of known materials including glass, silica, polymeric, or other transparent material(s). Likewise, lens(es) may be constructed from any suitable material and formed by embossing, molding, or lithography with photoresist reflow and/or etching or any other known technique. The lens array may reside on glass framework, which in turn may be affixed or otherwise integrated with the overall substrate. The lenses may be integrated onto the surface of the substrate facing the source or on the opposite side or integrated into the body of the substrate. Note that if the lenses are positioned on an opposite side of the substrate with respect to a light source, the focal lengths are larger for a same substrate position and thickness resulting in a reduced beam divergence.
Note further that each individual lens in the array has an optical axis—an axis of symmetry for the lens. For each lens, a chief ray passes from the source and intersects the optical axis at the lens entrance pupil. The chief ray and the optical axis lie in a tangential plane, and the chief ray also lies in a sagittal plane perpendicular to the tangential plane.
With reference to
We note that microlenses may be configured in alternative arrangements (patterns) according to aspects of the present disclosure. For example,
With these MLA configurations in mind, we note the field-of-view half-angle covered by the structured light source is approximately
where w is the width of the MLA and f is the tangential focal length of the lens at the edge of the array. To minimize beam divergence, f, and hence the distance L from the source to the MLA, φ should be as large as available space permits.
To increase FOV, w must be increased relative to f. However, the cost of MLA scales with its area and hence w2. Also, as the angle of incidence for light through the MLA increases, lens aberrations, Fresnel losses, pupil distortion, and reduced light-emitting diode (LED) intensity (since LED intensity drops with angle) all become increasingly problematic.
Advantageously, the FOV may be increased without increasing w/f by placing an optical element after the MLA which increases the divergence of transmitted beams.
With reference now to that figure, there it shows a negative power refractive lens L1 positioned such that it follows the MLA in the optical path. In an illustrative configuration, a source is positioned on the optical axis of L1 and the lens array is perpendicular to this axis, but such arrangement advantageously need not be so exact.
As illustrated in that figure, rays are shown for two microlenses (MLs) of the MLA. Since L1 diverges the beams, the ML positive power is effectively increased such that they focus light from the source with finite conjugate to a point beyond L1. Additionally, the MLs exhibit different curvatures in the tangential and sagittal planes such that the beams in object space beyond L1 are approximately equally well collimated in both directions. Each ML collimates the light, making the rays that pass through it more parallel but somewhat convergent, and L1 further collimates the light, reducing or eliminating the convergence. Also, the ML curvatures vary with lens center position relative to the optical axis. To make beam widths substantially equal, the CA of MLs may increase with distance from the optical axis. Additionally, a different type of optical element may be used in place of or in addition to L1 to increase the structured light (SL) FOV such as a Fresnel lens, a diffractive element, one or more mirrors, or one or more prisms. Finally, note that the FOV 2ϕ covered by the SL could be over 180°, or less than 180°, including—for example—160°, 140°, 120°, 100°, 80°, and 60°.
Turning now to
For example, additional light sources may be placed around the camera in a ring or other arrangement. Additional structured light elements may likewise be positioned around the camera, and interposed between the light sources or, in any other arrangement that produces a desired illumination and structured light emissions and/or patterns.
As illustratively shown in
With continued reference to
At this point we note that a capsule endoscope such as that according to the present disclosure is swallowable (ingestible) by a human and as such will exhibit a size of approximately 4 cm or less in length and approximately 2 cm or less in diameter. Such a capsule may be constructed from any of a number biocompatible materials that survive a trip through a digestive tract without comprising components contained within. Additionally, and as will be readily appreciated by those skilled in the art—at least portions of such capsules—in addition to exhibiting suitable biocompatibility—will also exhibit suitable optical properties. Once swallowed (ingested), capsules will pass through the digestive tract via physiological processes, including peristalsis.
As those skilled in the art will readily appreciate, additional hardware/electronic/optical components and additional software executed by the controller or other comparable structure(s) are contemplated. Operationally, image data stored in memory and then transmitted by the transmitter to external storage and/or processing systems. In certain illustrative embodiments, the memory may include longer-term, or archival storage in which image data is stored until the capsule is later retrieved after being excreted or otherwise removed from a patient. In yet other illustrative embodiments, the transmitter transmits data wirelessly through the patient body to an ex vivo receiver, for example, by radio, ultrasound, human-body electrical conduction, or optically. Advantageously, a general apparatus like that illustrated in
With simultaneous reference now to
Note that with reference to these three figures, systems, methods, and structures according to the present disclosure are shown in three different configurations—while sharing many aspects of this disclosure.
The white light source may be activated during an image sensor integration period of a same frame such that both SL and white light illuminate a captured image. Advantageously, the SL source may exhibit an optical spectrum that is different than that exhibited by the white light source such that it has a distinguishable color in images that include white light illumination. For example, the spectrum may be narrower such as that of a red, green, amber, or blue LED. The spectrum could—for example—fall outside the white light spectrum such as in the infrared (IR) region of the electromagnetic spectrum. Of course, an image sensor may include pixels with color filters that have a higher transmittance for the SL than for the white light. For example, pixels that are transmissive to IR and which absorb or otherwise block white light may be included on a sensor to advantageously detect IR structured light.
Note that while
Operationally, and as noted previously, light from the LED passes through a microlens array (MLA). In one illustrative embodiment, lenses comprising the MLA are centered on rings concentric with the LED, as shown illustratively in
Referring again to
As will be appreciated, mirror M1 directs (reflects) light beams out—through the tubular wall of the capsule housing. Relative to the axis of the light source—perpendicular to the MLA—M1 increases the angular field of view of the structured light beyond 180°. For example, the FOV may be 200°, 220°, or 240°. The mirror M1 reflection effectively “creates” a virtual source on the optical axis of the source that is shifted closer to the camera than the source. In
To extract depth information from an image of the structured light captured by the camera system, the camera center of projection (COP) and the virtual source must be physically separated. As depicted in
Turning now to
Note that the lens CAs are defined by a patterned black opaque layer of material such as black chrome. The clear(er) apertures are shown as being elliptical—although they could be other shapes including circular or rectangular—among others. The long axis of the oblong lenses lies in approximately tangential planes. The projection of the aperture onto a plane perpendicular to the chief ray is foreshortened. The oblong aperture compensates for the foreshortening to produce a more symmetrical beam. Larger apertures pass more light than smaller apertures so that the relative intensity of the beams is controlled by setting the aperture sizes.
At this point we note that the optical systems depicted illustratively in
Returning our discussion of
As illustratively configured, the radial position of M2 inside the capsule is less than that of M1 and the mirror apertures do not overlap such that both M1 and M2 may exist in the same system. After reflection from M3, the beam passes out through the tubular wall of the capsule and illuminates mucosa within the field of view of the camera. The combination of M2 and M3 reflections results in a beam angle exiting the housing similarly to the angle upon exiting the MLA.
As may be observed, however, the beam is displaced and appears to emanate from a virtual source (center of projection) further from the camera than the source on the longitudinal axis. Light emitted directly from the MLA would have been blocked by the camera and therefore prevented from exiting the capsule. By moving the virtual source further from the camera than the source, the beam passes the camera without being blocked. As will be readily appreciated by those skilled in the art, this same approach may be employed to route beams around other obstacles. Note that since during normal operation mucosa will contact the capsule housing—it is desirable to position mirrors to direct the light beams to cover as much of the housing within the FOV as possible.
For example, one spot is at point A if the object is contacting the endoscope and at point B if the object is at the edge of the system'"'"'s useful range. Each spot moves on its own epipolar line (or curve, if the camera image is distorted). To extract depth information about points in the scene, the system identifies spots in the image and determines a correspondence between the spots and the epipolar lines, which are based on a camera model. The correspondence may be confounded if epipolar lines cross and a spot is detected near an intersection. Fortunately, standard, known techniques exist for resolving this and other ambiguities to establish the correspondence for all or most of the spots.
In particular, the position of each spot on its epipolar line is determined and this position establishes the depth of the object at the spot location in the image. The greater the number of spots, the better the resolution of the depth map that is determined. Since the size and brightness of the spots also decrease with object distance—these quantities may also be used to determine the distance of the object(s) onto which the spots are projected. Rather than identifying a correspondence between individual spots and epipolar lines, the shape of a surface with structured light projected thereon may be estimated by other known methods such as determining the SL pattern deformation by determining correlations between portions, comprising multiple spots, of the projected and imaged pattern with portions of the known undeformed pattern to determine a map of the pattern deformation from projection on the surface.
While not yet specifically shown in the figures, it is nevertheless noted that an endoscope system according to the present disclosure will generally include one or more computers (or equivalent systems/structures/functionality) to receive image data from the endoscope system, process the data, display image data to a human—or “display” to an expert system—receive inputs from the human/expert system via an interface (e.g., GUI), present analysis results such as estimated object size, create or update a database of procedure data, and generate reports of the medical examination results.
The images—which include SL—are analyzed to extract information about the distance of objects visualized in the images. This analysis may advantageously be performed in a batch mode for many or all SL images prior to presentation to a human reader or, to reduce processing time, it may be performed on select images that are flagged or otherwise identified by the reader or machine (e.g. expert system/algorithm(s)) that operate on a set of images to determine images of interest. For example, either a reader or machine (algorithm) might identify a possible lesion in a particular frame, and then the depth information for that frame and/or neighboring frames is extracted from the structured light image data.
We note that endoscope images are typically presented to a reader as a series of still images such as a video. The reader views the video looking for pathologies or other objects of interest. Frames containing such objects (frames of interest) may be selected and placed into a list or database of selected frames for the particular medical procedure.
As will be appreciated, some frames may include objects or regions within the overall image for which the reader desires a size measurement. Such measurement may then be operationally indicated by the reader by any of a number of well-known computational tools including GUIs. For example, the reader may select points on a periphery of a region, draw a curve around periphery or draw a line across the region.
The system will then estimate the distance across the indicated region, for example between two designated points. If the image includes structured light, it may be used to estimate the distance in object space of any objects/features of interest in the image. From the structured light, a 3D model of a scene or portion of a scene—represented by the image—may be constructed. Such model may be coarse if the density of SL points is significantly less than the pixel density of the image.
While direct depth information may be available for those pixels that lie near the centroids of the SL spots, it will not be for pixels that lie between spots. As may be understood and readily appreciated by those skilled in the art, additional information in the image such as detected edges or depth-from-shading may be used to better estimate the depth information across the image. Advantageously, the depth may also be estimated in regions between the SL spots by interpolation from the calculated depth at the SL centroids. Once the 3D coordinates for two or more points demarcating an object are estimated, the cartesian distance between them in object space is determined.
A size measurement is typically displayed to a reader and recorded in a database. The functions of identifying and demarcating a region of interest may be performed by a machine-executed algorithm instead of a human reader, or the reader may work in conjunction with such machine-executed algorithm(s) to identify and demarcate such regions.
As will be readily appreciated by those skilled in the art, if structured light (SL) and white light (WL) illumination exist in the same frame, a system must identify the structured light within a regular WL image background. Note that scattered SL may also produce background light. Note further that the structured light spots are known to line on epipolar lines. The location of these lines is determined from a camera model that may be based—at least partially—on camera and projector calibration. More particularly, the system looks for image features that best match the structured light in an expected shape, size, intensity, and color. Color—in particular—provides a useful way of distinguishing SL from WL when such SL color is sufficiently different from the WL illumination color.
We note that when reviewing a video or other set of images captured from an endoscopic system according the present disclosure, visible structured light in the video (or images) may be a distraction to the reviewer. Accordingly, various methods may be utilized to remove it from an image once such SL spots are identified.
More particularly, an estimate of an SL pixel signal may be subtracted from the image. Such estimate may be based on a model of the SL including its color. Accordingly, if a particular pixel is saturated in a color plane due to the SL or if the SL signal to be subtracted is large, then the white light image signal in that color plane may be estimated based on the signal in other color planes.
For example, if the SL is predominately red, the red color plane may be reconstructed from the green and blue color plane data for pixels within the SL spot based on a statistical correlation between red, green, and blue color planes in the region of an image around that spot. Methods such as “in painting” may also be used to fill in missing image and create a continuous image appearance. To eliminate chroma error—resulting from imperfect SL subtraction from the image—it may be advantageously displayed as a gray-scale image. If the structured light is in the IR and the structured light is detected by IR pixels, then an RGB image with minimal impairment by structured light is available.
Note that methods employed to subtract the SL from images are likely to leave some residual impact on the image quality. Therefore, it is desirable for the SL to be captured in separate frames from the white light frames. As will be understood, the time difference(s) (separation) between the white light and SL frame(s) should be short enough such that any change in scene is sufficiently small that depth information determined in the SL frame(s) may be applied to the scene(s) in the WL frame(s) with minimal error.
To reduce the impact of any scene, change(s), the reviewer/reader may demarcate an object in two or more WL frames. Then, the position and size—in pixels—of the object in a SL frame temporally positioned between two WL frames (i.e., interstitial) may be estimated by interpolation between the WL frames. If the object of interest appears in multiple frames, then the reviewer/reader (or machine system/algorithm) may select one or more frames in which to demarcate the object and a proximal SL frame in which to estimate the object size—based on an estimate of a rate of object movement—selecting frames with minimal—or acceptable—movement. We note that the amount of movement of objects in a video or series of images may be estimated by known methods such as the calculation of motion vectors. The motion metric on which frames may be selected may be based more on the motion of the particular object region to be measured in the video than the overall motion of the entire scene.
Advantageously, a reviewer/reader or image recognition system algorithm (including any employing machine learning methodologies) may identify an object in one frame that is of interest. Then, the reviewer—or system—may search for the same object in neighboring frames using pattern recognition methods and/or algorithms. Then, from the set of frames including the object, one or more frames may be selected for object-of-interest demarcation.
The frames may be selected based on multiple criteria such as a rate of object movement, the fraction of the object within an image boundary, and the quality of an image including factors such as exposure, motion blur, obscuration by fecal—or other—matter, and the presence of bubbles or turbidity. The algorithm may select particular frames and the reviewer/reader may confirm their suitability by making an entry using the GUI or other mechanism. In illustrative embodiments, selected frames may have check boxes that are selected to keep or deselect frames. The demarcation of the object in these frames may advantageously be performed manually using—for example—the GUI—or other mechanism—by the reviewer/reader or automatically by a system with confirmation or fine-tuning by the reviewer/reader. The size measurement based on the demarcation and analysis of SL in the same or proximal frames may be presented to a reviewer/reader on a screen. The measurement presentation may include—for example—error bars, confidence intervals, or other indicators of the accuracy of the measurement.
As will be readily appreciated by those skilled in the art, a video—or series of images—captured by a capsule endoscope moving autonomously through a GI tract has many image frames exhibiting redundant information since at times the capsule is not moving, moves retrograde, or dithers. The endoscope system may not display some frames that are determined to be redundant, i.e., showing the same features that are displayed in other frames. Also, multiple frames that capture overlapping images of the scene may be stitched into a composite image. As will be understood, this reduction in frame number reduces the time needed to review the video.
When an object of interest—such as a lesion—is identified in one of the displayed frames the system may display a version of the video with all frames displayed—including those previously not displayed or those combined with other frames into stitched frames. The process of finding optimal frames in which to demarcate the object and measure its size, as described previously, can be applied to this larger set of frames. The best frame(s) for demarcating the object—based on criteria described above, or others—may be one of the frames that was not originally displayed.
Note that a region of interest for size measurement may not be fully visualized in a frame, especially if the region is large. However, two or more frames containing portions of the region may be stitched together so that all or most of the region is captured in the stitched frame. The region may be demarcated in the stitched frame and cartesian distance between demarcation points may be estimated based on the structured light data in the frames stitched and/or interstitial frames.
As will be appreciated by those skilled in the art, capsule endoscopes present some particularly unique imaging conditions. Accordingly, the magnification of objects imaged by an endoscope camera (whether a capsule or insertable) is larger if the object is immersed in a fluid rather than in air or other gas. Thus, the correct estimation of object depth using structured light depends on a knowledge of the immersing medium.
During a colonoscopy, the colon is insufflated with gas. For capsule endoscopy, the colon and other organs are preferably filled with clear (colorless) water. However, gas bubbles, including large pockets of gas, do exist in the lumen during capsule endoscopy. In a video or set of images, these bubbles may be recognized due to the appearance of bright specular reflections of the illuminating light from the wet mucosal surface and a change in mucosal color, relative to water immersed mucosa. Moreover, a meniscus is visible where the bubble boundary crosses the capsule housing.
When a reviewer/reader or a machine algorithm has identified an object for size measurement, the reviewer may be queried to determine whether the object is immersed in a liquid or a gas. Since the object may be partially in a liquid and partially in a gas, the reviewer/reader may indicate a gas/liquid ratio for the immersion or may use a cursor tool (or other GUI or other mechanism) to mark areas that are in gas or in liquid. Of course, a computer implemented method/algorithm may perform these same functions.
The geometric model of the SL is modified based on the medium selected. Alternatively, a measurement based on a fixed single-medium model may be scaled ad hoc based on the selected medium. For example, if the SL model assumes water immersion, but a fraction P of the diameter of a measured object is in gas (e.g., P=0.40), the size estimate may be adjusted by PM where M is the relative magnification in gas versus liquid. Finally, M may be a function of field position and estimated object distance and may be based on an a-priori camera model and calibration.
At this point we note that endoscope calibration may advantageously be performed during manufacturing. More particularly, an endoscope may be presented with targets at known positions and orientations relative to the endoscope camera. Some targets may include a pattern such as a checkerboard. The location of the features in the pattern in the recorded image can help determine a model of the camera including focal length, COP, pose, and distortion. Other calibration images are formed by illuminating SL from the endoscope onto one or more targets. These calibration images help determine a model of the SL projection including COPs, pose, epipolar lines, and color.
Note that for a capsule endoscope, it is convenient to store calibration data as well as any images and/or parameters derived from images, in a capsule endoscope memory. This data can then be downloaded with any in vivo data to a workstation for processing the in vivo data and extracting depth information from in vivo images using camera and SL models derived from—at least partially—the calibration data. Alternatively, the calibration data for a capsule can be associated with a capsule identifier, such as a serial number, and be stored in a database. Upon recovering the in vivo data and the identifier from the capsule, the calibration data associated with the identifier can be retrieved from the database and use for processing the in vivo data.
Image sensors used in endoscopes oftentimes include a mosaic of color filters on the pixels. For example, a sensor may have red (R), green (G), and blue (B) pixels with responsivity spectra as illustratively shown in
Operationally, when SL illuminates mucosa, some light is scattered from the surface of the mucosa and some light penetrates the mucosa tissues and experiences a combination of absorption and bulk scattering. Some of the bulk scattered light emerges from the mucosa some distance from the point of incidence. The visible SL spot is thus spatially broader than the light incident on the mucosa due to the diffusion of light in the tissues. This broadening or blooming could make it difficult to distinguish one spot from another.
We note that the image sensor has a limited dynamic range and there is a maximum luma, luma-sat, corresponding to the maximum irradiance that can be recorded, for a particular sensor gain. Luma-sat is determined by the sensor analog to digital converter (ADC). For the case of a 10-bit ADC, the maximum luma is 1023 digital counts. With continued reference to
For the situation illustrated in
A sensor with pixels responsive to different color spectra, as opposed to a monochrome gray-scale sensor, increases the effective dynamic range of the SL detection if the response to the SL light is different but non-zero for at least two of the color channels. The channels can be combined into a single channel of increased dynamic range or analyzed separately. The example given is an RGB sensor, but other color channels could be used such yellow, clear (white), magenta, cyan, violet, or IR.
In this illustrative system the FOV of the imaging system is 360° about the capsule and from approximately 45° to 135° relative to the longitudinal axis. Mirrors within the lens module fold the optical axes of the lenses. In a particular illustrative embodiment, images are formed on a common image sensor, which may have pixels in four separate regions. The capsule includes white light LEDs or other sources for illuminating the lumen wall.
As illustratively shown in
Illustratively, the MLA includes microlenses arrayed in substantially concentric rings such as that illustratively shown in
Shown further in
As may be further observed in
As noted throughout this disclosure, endoscope configurations in which additional optical elements follow an MLA in an optical path may afford distinct advantages to those endoscopes.
Computer system 1700 includes processor 1710, memory 1720, storage device 1730, and input/output structure 1740. One or more input/output devices may include a display 1745. One or more busses 1750 typically interconnect the components, 1710, 1720, 1730, and 1740. Processor 1710 may be a single or multi core.
Processor 1710 executes instructions in which embodiments of the present disclosure may comprise steps described in one or more of the Figures. Such instructions may be stored in memory 1720 or storage device 1730. Data and/or information may be received and output using one or more input/output devices.
Memory 1720 may store data and may be a computer-readable medium, such as volatile or non-volatile memory. Storage device 1730 may provide storage for system 1700 including for example, the previously described methods. In various aspects, storage device 1730 may be a flash memory device, a disk drive, an optical disk device, or a tape device employing magnetic, optical, or other recording technologies.
Input/output structures 1740 may provide input/output operations for system 1700. Input/output devices utilizing these structures may include, for example, keyboards, displays 1745, pointing devices, and microphones—among others. As shown and may be readily appreciated by those skilled in the art, computer system 1700 for use with the present disclosure may be implemented in a desktop computer package 1760, a laptop computer 1770, a hand-held computer, for example a tablet computer, personal digital assistant or Smartphone 1780, or one or more server computers which may advantageously comprise a “cloud” computer 1790.
At this point, while we have presented this disclosure using some specific examples, those skilled in the art will recognize that our teachings are not so limited. More specifically, our methods can be further extended in that the structural events can embed more temporal information and consider more sophisticated structures including considering more finegrained temporal information, e.g., the transition time distribution, to enrich mined structural events. Also, we have focussed on transition relations among log patterns. There are other useful relations among logs, such as running in parallel that may be employed. Those relations can be further modeled in the workflow graph using undirected edges. We also believe that the methods according to the present disclosure can achieve more utility in an interactive setting, where system admins can interactively explore the system behaviors with different focusses (parameter settings) on coverage, quality or connectivity.
Accordingly, this disclosure should be only limited by the scope of the claims attached hereto.