Video imaging system with image processing optimized for small-diameter endoscopes

• US 5,475,420 A
• Filed: 10/08/1993
• Issued: 12/12/1995
• Est. Priority Date: 06/09/1993
• Status: Expired due to Term

• Current Assignee: Covidien AG (Medtronic Plc)
• Original Assignee: Origin Medsystems Inc. (Boston Scientific Corporation)
• Inventors: Buchin, Michael P.
• Primary Examiner(s): Chin, Tommy P.
• Assistant Examiner(s): RAO, ANAND SHASHIKANT
• Application Number:
  US08/134,580
• Time in Patent Office: 795 Days
• Field of Search: 348/72, 348/71, 348/69, 348/67, 348/65, 128/6, 128/5, 128/7
• US Class Current: 348/72
• CPC Class Codes: A61B 1/0005 combining images e.g. side-by-side, superimposed or tiled
• CPC Class Codes: A61B 1/00059 provided with identification means for the endoscope
• CPC Class Codes: A61B 1/00126 optical, e.g. for light supply cables
• CPC Class Codes: A61B 1/042 characterised by a proximal camera, e.g. a CCD camera
• CPC Class Codes: H04N 23/555 for picking-up images in sites, inaccessible due to their dimensions or hazardous conditions, e.g. endoscopes or borescopes
• CPC Class Codes: H04N 23/72 Combination of two or more compensation controls

Abstract

An output video signal is derived from an input video signal generated by an image sensor in response to an image formed on only part of the image sensor. The output video signal is derived in response to only the portion of the input video signal generated by the part of the image sensor on which the image is formed. The input video signal additionally includes a portion generated by the part of the image sensor on which the image is not formed. An image signal generating apparatus is provided in which the image sensor is mounted, and which includes a detachably-attached image-forming device. When attached to the image signal generating apparatus, the image-forming device forms the image on the part of the image sensor. A frame of the input video signal is received from the image sensor. The frame of the input video signal has a structure and includes an image portion generated by the part of the image sensor on which the image is formed, and an external portion generated by the part of the image sensor on which the image is not formed. The image portion of the frame of the input video signal is identified, and the identified image portion is extracted from the frame of the input video signal to provide the image portion of a frame of the output signal which has a frame structure substantially similar to the frame structure of the input video signal.

Claims

1. A method of deriving an output video signal from an input video signal generated by an image sensor having an image formed on only a part thereof, the output video signal being derived in response to only a portion of the input video signal generated by the part of the image sensor whereon the image is formed, the input video signal additionally including a portion generated by the part of the image sensor whereon the image is not formed, the method comprising steps of:

providing an image signal generating apparatus wherein the image sensor is mounted, the image signal generating apparatus including a detachable image-forming device selectably attached thereto, the detachable image-forming device including a lens and a proximal fibre-optic bundle, and, when attached to the image signal generating apparatus, forming the image on the part of the image sensor, the image having a size and a position;

receiving a frame of the input video signal from the image sensor, the frame of the input video signal having a structure and including an image portion generated by the part of the image sensor on which the image is formed, and an external portion generated by the part of the image sensor on which the image is not formed;

identifying the image portion of the frame of the input video signal; and

extracting from the frame of the input video signal the image portion of the input video signal identified in the identifying step to provide an image portion of a frame the output signal, the frame of the output video signal having a structure substantially similar to the structure of the frame of the input video signal.

2. The method of claim 1, wherein:

in the step of providing an image signal generating apparatus, the detachable image-forming device is of a first type, and the image signal generating apparatus is capable of having a detachable image-forming device of a second type detachably attached thereto in lieu of the detachable image-forming device of the first type, each of the detachable image-forming devices, when attached to the image signal generating apparatus, forming an image on the image sensor, the image having a size and a position depending on the type of the detachable image-forming device, at least one of the detachable image-forming devices forming the image on part of the image sensor;

the method additionally comprises a step of determining the type of the detachable image-forming device forming the image; and

the step of identifying the image portion of the frame of the input video signal is performed in response to the type of the detachable image-forming device determined in the determining step.

3. The method of claim 1, wherein:

in the step of providing an image signal generating apparatus, the detachable image-forming device includes storage means for storing image information for the detachable image-forming device, the image information being information whence is identified the image portion of the frame of the input video signal generated by the part of the image sensor whereon the image is formed by the detachable image-forming device;

the method additionally comprises a step of retrieving, from the storage means in the detachable image-forming device, the image information for the detachable image-forming device; and

in the step of identifying the image portion of the frame of the input video signal, the image portion of the frame of the input video signal is identified using the image information retrieved from the detachable image-forming device in the retrieving step.

4. The method of claim 1, wherein the method additionally comprises a step of storing, in the detachable image-forming device, image information for the detachable image-forming device, the image information being information whence is identified the image portion of the frame of the input video signal generated by the part of the image sensor whereon the image is formed by the detachable image-forming device;

and

the step of identifying the image portion of the frame or the input video signal includes a step of retrieving the image information from the detachable image-forming device.

5. The method of claim 1, wherein:

in the step of receiving the frame of the input video signal, the frame of the input video signal includes lines; and

the step of identifying the image portion of the frame of the input video signal includes a step of;

analyzing the input video signal to determine ones of the lines of the frame of the input video signal that include an image part generated by the part of the image sensor whereon the image is formed, and an external part generated by the part of the image sensor whereon the image is not formed, and, for each of such lines, finding a position of a boundary between the image part and the external part.

6. The method of claim 1, wherein:

in the step of receiving the frame of the input video signal, the frame of the input video signal includes lines, each of the lines including an image part generated by the part of the image sensor whereon the image is formed, and an external part generated by the part of the image sensor whereon the image is not formed;

in the step of providing an image signal generating apparatus, the image formed on the part of the image sensor has a known shape, an unknown position on the image sensor, and an unknown size; and

the step of identifying the image portion of the frame of the input video signal includes steps of;

analyzing the input video signal to determine parameters for calculating the size and the position of the image on the image sensor, and

calculating, from the parameters determined in the analyzing step, and from the known shape of the image, the lines including an image portion and an external portion, and, for each of such lines, a position of a boundary between the image part and the external part.

7. The method of claim 2, wherein:

the method additionally comprises a step of storing image information for each type of the detachable image-forming device, the image information being information whence is identified the image portion of the input video signal generated by the part of the image sensor whereon We image is formed by the type of the detachable image-forming device; and

the step of identifying the image portion of the frame of the input video signal includes steps of;

retrieving the image information for the type of image-forming device in response to the type of the detachable image-forming device determined in the determining step, and

identifying the image portion of the frame of the input video signal using the image information retrieved in the retrieving step.

8. The method of claim 2, wherein:

in the step of providing an image signal generating apparatus, only the detachable image-forming device of the first type forms the image on the part of the image sensor;

the step of determining the type of the detachable image-forming device forming the image on the image sensor is responsive only to the detachable image-forming device of the first type; and

the step of identifying the image portion, of the input video signal is performed only when the determining step determines that the detachable image-forming device of the first type is forming the image on the part of the image sensor.

9. The method of claim 3, wherein:

in the step of receiving the frame of the input video signal, the frame of the input video signal includes lines that include an image part generated by the part of the image sensor whereon the image is formed, and an external part generated by the part of the image sensor whereon the image is not formed; and

the image information stored in the storage means includes, for each of the lines, a position of a boundary between the image part and the external part of the line.

10. The method of claim 4, wherein:

in the step of receiving the frame of the input video signal, the frame of the input video signal includes lines that include an image part generated by the part of the image sensor whereon the image is formed, and an external part generated by the part of the image sensor whereon the image is not formed; and

in the step of storing image information, the image information includes, for each of the lines, a position of a boundary between the image part and the external part of the line.

11. The method of claim 5, wherein, the step of analyzing the input video signal to determine ones of the lines of the frame of the input video signal that include an image part and an external part includes a step of examining each of the lines to detect a boundary between the image part and the external part.

12. The method of claim 6, wherein:

the shape of the image formed on He part of the sensor is substantially circular, and has a radius and a center; and

the step of analyzing the input video signal to determine parameters includes a step of determining the radius of the image, and a one of the lines and a position on the one of the lines corresponding to the center of the image as the parameters for calculating the size and the position of the image on the image sensor.

13. The method of claim 6, wherein:

the shape of the image formed on the part of the sensor is substantially elliptical, and has a center and two axes, each of the axes having a length; and

the step of analyzing the input video signal to determine parameters includes a step of determining the lengths of the axes, and a one of the lines and a position on the one of the lines corresponding to the center of the image as the parameters for calculating the size and the position of the image on the image sensor.

14. The method of claim 6, wherein the boundary whose position is calculated in the calculating step is a first boundary, and the calculating step additionally calculates, for each of the lines including an image part and an external part, a position of a second boundary between the image part and the external part.

15. The method of claim 7 wherein:

in the step of receiving the frame of the input video signal, the frame of the input video signal includes lines that include an image part generated by the part of the image sensor whereon the image is formed, and an external part generated by the part of the image sensor whereon the image is not formed; and

in the step of storing image information the image information includes, for each of the lines, a position of a boundary between the image part and the external part of the line.

16. The method of claim 15, wherein, in the step of storing image information, the image information includes the position of a first boundary between the image part and the external part of each one of the lines, and additionally includes a position of a second boundary between the image part and the external part of the one of the lines.

17. The method of claim 8, wherein:

the step of determining the type of the detachable image-forming device comprises steps of;

providing a switch operable only by the detachable image-forming device of the first type, the switch having a state, and

changing the state of the switch to a changed state when the detachable image-forming device of the first type is attached to the image signal generating apparatus; and

the step of identifying the image portion of the frame of the input video signal comprises steps of;

retrieving, in response to the changed state of the switch, stored image information for the detachable image-forming device of the first type, the image information being information whence is identified the image portion of the frame of the input video signal generated by the part of the image sensor whereon the image formed by the detachable image-forming device of the first type, and

identifying the image portion of the frame of the input video signal using the stored image information retrieved in the retrieving step.

18. The method of claim 9, wherein the image information stored in the storage means includes the position of a first boundary between the image part and the external part of each one of the lines, and additionally includes a position of a second boundary between the image part and the external part of the one of the lines.

19. A method for generating an output video signal from an input video signal, position-dependent luminance errors in the input video signal being corrected in the output video signal, the input video signal being generated by an image sensor having an image formed on only a part thereof, the output video signal being derived in response to only a portion of the input video signal generated by the part of the image sensor whereon the image is formed, the input video signal additionally including a portion generated by the part of the image sensor whereon the image is not formed, the image including an image element having a position in the image, the method comprising steps of:

receiving a frame of the input video signal from the image sensor, the frame of the input video signal having a structure and including an image portion generated by the part of the image sensor on which the image is formed, and an external portion generated by the part of the image sensor on which the image is not formed, the image portion of the input video signal including a signal element corresponding to the image element;

identifying the image portion of the frame of the input video signal;

extracting from the frame of the input video signal the image portion identified in the identifying step;

multiplying the signal element of the image portion by a position-dependent function to provide a modified image portion, the position-dependent function depending on the position in the image of the image element; and

incorporating the modified image portion in a frame the output video signal as an image portion thereof, the frame of the output video signal having a structure substantially similar to the structure of the frame of the input video signal.

20. The method of claim 19, wherein:

the image formed on the part of the image sensor is substantially circular, and has a center, and the image element is located in the image at a distance from the center of the image; and

in the multiplying step, the position-dependent function depends on the distance of the image element from the center of the image.

21. The method of claim 19, wherein:

the method additionally comprises steps of;

providing an image signal generating apparatus wherein the image sensor is mounted, the image signal generating apparatus including an image-forming device detachably attached thereto, the image-forming device being of a first type, the image signal generating apparatus being capable of having an image-forming device of a second type detachably attached thereto in lieu of the image-forming device of the first type, each of the image-forming devices when attached to the image signal generating apparatus, forming an image on the image sensor, the image having a size and a position depending on the type of the image-forming device, at least one of the image-forming devices forming the image on the part of the image sensor;

determining the type of the image-forming device forming the image on the part of the image sensor to generate image-forming device type information, and

retrieving a stored position-dependent function for the type of the image-forming device in response to the image-forming device type information; and

in the multiplying step, the signal element is multiplied by the position-dependent function retrieved in the retrieving step for the type of the image forming device.

22. The method of claim 19, wherein:

the method additionally comprises steps of;

providing an image signal generating apparatus wherein the image sensor is mounted, the image signal generating apparatus including an image-forming device detachably attached thereto, the image-forming device being of a first type, the image signal generating apparatus being capable of having an image-forming device of a second type detachably attached thereto in lieu of the image-forming device of the first type, each of the image-forming devices, when attached to the image signal generating apparatus, forming an image on the image sensor, the image having a size and a position depending on the type of the image-forming device, at least one of the image-forming devices forming the image on the part of the image sensor,

determining when the image-forming device forming the image on the image sensor is of a type that forms the image on the part of the image sensor, and

retrieving a stored position-dependent function for the image-forming device of the type that forms the image on the part of the image sensor; and

in the multiplying step, the signal element is multiplied by the position-dependent function retrieved in the retrieving step for the image forming device of the type that forms the image on the part of the image sensor.

23. The method of claim 19, wherein:

the method additionally comprises steps of;

providing an image signal generating apparatus wherein the image sensor is mounted, the image signal generating apparatus including an image-forming device detachably attached thereto, the image-forming device being of a first type, the image signal generating apparatus being capable of having an image-forming device of a second type detachably attached thereto in lieu of the image-forming device of the first type, each of the image-forming devices, when attached to the image signal generating apparatus, forming an image on the image sensor, the image having a size and a position depending on the type of the image-forming device, at least one image-forming device forming the image on the part of the image sensor, and including means for storing a position-dependent function for the image-forming device, and

retrieving from the image-forming device forming the image on the image sensor the stored position-dependent function for the image-forming device; and

in the multiplying step, the signal element is multiplied by the position-dependent function retrieved in the retrieving step for the image-forming device.

24. The method of claim 19, additionally comprising steps of:

providing an image signal generating apparatus wherein the image sensor is mounted, the image signal generating apparatus including an image-forming device attached thereto, the image-forming device forming the image on the part of the image sensor;

providing a uniformly-illuminated target;

pointing the image-forming device at the uniformly-illuminated target; and

executing a set-up procedure to determine a position-dependent function to provide a uniform sensitivity within the image.

25. The method of claim 24, wherein:

the step of executing a set-up routine comprises steps of;

determining a reference luminance for a reference part of the image portion of the input video signal, the reference part of the image portion corresponding to a central region in the image,

determining an element luminance for the signal element; and

storing a ratio of the reference luminance and the element luminance as the position-dependent function for the signal element; and

in the multiplying step, the signal element is multiplied by the stored position-dependent function for the signal element.

26. A method for processing an input video signal generated by an image sensor having an image formed on only a part thereof to generate a control signal for controlling an automatic luminance system responsive to a level of the control signal averaged over a time corresponding to frames of the input video signal, the control signal being generated only in response to a portion of the input video signal generated by the part of the image sensor whereon the image is formed, the input video signal additionally including a portion generated by the part of the image sensor whereon the image is not formed, the method comprising steps of:

receiving a frame of the input video signal from the image sensor, the frame of the input video signal having a structure and including an image portion generated by the part of the image sensor on which the image is formed, and an external portion generated by the part of the image sensor on which the image is not formed, the image portion having an average luminance;

identifying the image portion of the frame of the input video signal; and

deriving from the frame of the input video signal a frame of the control signal, the frame of the control signal having an average level corresponding to the average luminance of the image portion of the frame of the input video signal identified in the identifying step.

27. The method of claim 26, wherein the step of deriving a frame of the control signal includes steps of:

extracting from the frame of the input video signal only the image portion identified in the identifying step; and

repeatedly inserting the image portion of the frame of the input video signal extracted in the extracting step into the frame of the control signal until the frame of the control signal is filled.

28. The method of claim 26, wherein the step of deriving a frame of the control signal includes steps of:

determining an average luminance of the image portion of the frame of the input video signal identified in the identifying step; and

generating the frame of the control signal with a substantially constant level, the substantially constant level corresponding to the average luminance of the image portion of the frame of the input video signal determined in the determining step.

29. The method of claim 26, wherein the image sensor has a first area, and the image formed on the part of the image sensor has a second area, and wherein:

the step of deriving a frame of the control signal includes steps of;

determining the second area in response to the identifying step,

calculating a ratio between the first area and the second area,

averaging luminance of the frame of the input video signal to generate an average luminance for the frame, and

multiplying the average luminance for the frame of the input video signal by the ratio to provide the frame of the control signal.

30. The method of claim 26, additionally providing a reduced sensitivity of the control signal to reflections from small, highly-reflective objects in the image formed on the part of the image sensor, wherein the step of deriving a frame of the control signal includes steps of:

dividing the image portion of the frame of the input video signal identified in the identifying step into parts, each of the parts corresponding to a part of the image formed on the part of the image sensor;

determining an average luminance for each of the parts of the image portion of the frame of the input video signal;

deriving a weighted average luminance from the average luminances of the parts of the image portion, an average luminance substantially greater than other average luminances being included at a reduced weight; and

generating the frame of the control signal with a level corresponding to the weighted average luminance.

31. The method of claim 26, additionally providing a reduced sensitivity of the control signal to reflections from small, highly-reflective objects in the image formed on the part of the image sensor, wherein the step of deriving the frame of the control signal includes steps of:

dividing the image portion of the frame of the input video signal identifying in the identifying step into parts, each of the parts corresponding to a part of the image formed on the part of the image sensor;

determining an average luminance for each of the parts of the image portion of the frame of the input video signal;

deriving a weighted average luminance from the average luminances of the parts of the image portion of the input video signal, similar average luminances being included in the weighted average at a greater weight than substantially dissimilar average luminances; and

generating the frame of the control signal with a level corresponding to the weighted average luminance.

32. The method of claim 26, wherein the control signal is formatted as a video signal for controlling an automatic luminance system responsive to the average luminance of complete frames of a video signal.

33. The method of claim 29, wherein, in the dividing step, the image portion of the frame of the input video signal identified in the identifying step is divided into parts corresponding to sectors of the image.

34. The method of claim 29, wherein, in the dividing step, the image portion of the frame of the input video signal identified in the identifying step is divided into parts corresponding to radial zones of the image.

35. The method of claim 31, wherein, in the dividing step, the image portion of the frame of the input video signal is divided into parts corresponding to sectors of the image.

36. The method of claim 31, wherein, in the dividing step, the image portion of the frame of the input video signal is divided into parts corresponding to radial zones of the image.

37. A method for deriving an output video signal from an input video signal generated by an image sensor having an image formed on only a part thereof, the output video signal being derived in response to only a portion of the input video signal generated by the part of the image sensor whereon the image is formed, the input video signal additionally including a portion generated by the part of the image sensor whereon the image is not formed, the output video signal having a lower noise level than the portion of the input video signal generated by the part of the image sensor whereon the image is formed, the method comprising steps of:

providing a memory, the memory including pages, each of the pages including storage locations;

receiving a frame of the input video signal from the image sensor, the frame of the input video signal having a structure, being divided into picture elements, and including an image portion generated by the part of the image sensor on which the image is formed, and an external portion generated by the part of the image sensor on which the image is not formed;

identifying the image portion of each frame of the input video signal;

storing the picture elements of the image portion, identified in the identifying step, of successive frames of the input video signal in corresponding ones of the memory locations in successive ones of the pages of the memory;

averaging the picture elements of the image portion of the successive frames of the input video signal stored in the corresponding ones of the memory locations in the successive ones of the pages of the memory to provide average values; and

generating an image portion of a frame of the output video signal from the average values provided by the averaging step, the frame of the output video signal having a structure substantially similar to the structure of the frame of the input video signal.

38. The method of claim 37, wherein:

the method additionally comprises a step of determining a movement index between the corresponding picture elements of the successive frames of the image portion of the input video signal stored in the successive ones of the pages of the memory to determine an index for each frame; and

the step of averaging the corresponding picture elements of the image portion of the successive frames of the input video signal stored in the corresponding ones of the memory locations is performed only on the frames stored in the ones of the pages of the memory whose index is below a threshold value.

39. The method of claim 38, wherein:

in the determining step, the index is determined from selected parts of the image portion of the input video signal identified in the identifying step; and

in the averaging step, the successive ones of the pages of the memory on which averaging is performed on the whole image portion is determined by the index determined for the selected parts of the input video signal in the determining step.

40. The method of claim 38, wherein:

in the determining step, an index for each of the selected parts of the image portion of the input video signal identified in the identifying step is determined; and

in the averaging step, the successive one of the pages of the memory on which averaging is performed in each one of the selected parts is determined by the respective index determined for each one of the selected parts of the input video signal in the determining step.

41. A method for deriving an output video signal from an input video signal generated by an image sensor having an image formed on only a part thereof, the output video signal being derived in response to only a portion of the input video signal generated by the part of the image sensor whereon the image is formed, the input video signal additionally including a portion generated by the part of the image sensor whereon the image is not formed, the output video signal having a lower noise level than the portion of the input video signal generated by the part of the image sensor whereon the image is formed, and the method comprising steps of:

receiving a frame of the input video signal from the image sensor, the frame of the input video signal having a structure and including an image portion generated by the part of the image sensor on which the image is formed, and an external portion generated by the part of the image sensor on which the image is not formed, the image portion of the frame of the input video signal being divided into picture elements, each of the picture elements having a sample value;

identifying the image portion of the frame of the input video signal;

providing a memory, the memory including storage locations, each of the storage locations corresponding to one of the picture elements of the image portion of the input video signal;

reading an accumulated sample value for each one of the picture elements of the image portion of the input video signal from a respective one of the storage locations in the memory;

multiplying the accumulated sample value read from the memory in the reading step by a degradation factor to provide a multiplied accumulated sample value;

averaging the multiplied accumulated sample value provided by the multiplying step by the sample value of the one of the picture elements of the frame of the image portion of the input video signal identified in the identifying step to provide an average value, the one of the picture elements corresponding to the multiplied accumulated sample value;

providing the average value provided in the averaging step as a sample value of a picture element of an image portion of the output video signal, the picture element of the image portion of the output video signal corresponding to the one of the picture elements of the input video signal, the frame of the output video signal having a structure substantially similar to the structure of the frame of the input video signal; and

storing the average value provided in the averaging step in the one of the storage locations in the memory as the accumulated sample value for the one of the picture elements of the input video signal.

42. The method of claim 41, wherein the step of multiplying the accumulated sample value read from the memory in the reading step by a degradation factor includes steps of:

determining a movement index between the accumulated sample value read from the memory in the reading step and the sample value of the one of the picture elements of the image portion of the input video signal, the one of the picture elements corresponding to the accumulated sample value; and

using the movement index to determine the degradation factor.

43. An apparatus for deriving an output video signal from an input video signal generated by an image sensor having an image formed on only a part thereof, the output video signal being derived in response to only a portion of the input video signal generated by the part of the image sensor whereon the image is formed, the input video signal additionally including a portion generated by the part of the image sensor whereon the image is not formed, the apparatus comprising:

an image signal generator wherein the image sensor is mounted, the image signal generator including an image-forming device detachably attached thereto, the image-forming device, when attached to the image signal generator, forming the image on part of the image sensor;

a frame store memory means for storing one of a frame of the input video signal and a frame of the output video signal, the frame of the input video signal having a structure and including an image portion generated by the part of the image sensor on which the image is formed, and an external portion generated by the part of the image sensor on which the image is not formed; and

a digital signal processor means, operating together with the frame store memory means, for;

identifying the image portion of the frame of the input video signal in response to identification information; and

extracting only the image portion identified by the identification information from the frame of the input video signal to provide an image portion of the frame of the output video signal, the frame of the output video signal having a structure substantially similar to the structure of the frame of the input video signal.

44. The apparatus of claim 43, wherein:

the digital signal processor means is additionally for receiving the frame of the input video signal; and

the digital signal processor means provides the image portion of the frame of the output by;

storing the image portion extracted from the frame of the input video signal in response to the identification information in the frame store memory means,

reading the image portion out of the frame store memory means, and

providing the image portion read out from the frame store memory means as a part of the frame of the output video signal.

45. The apparatus of claim 43, wherein:

the digital signal processor means is additionally for;

receiving the frame of the input video signal, and

storing the frame of the input video signal in the frame store memory means; and

the digital signal processor means provides the image portion of the frame of the output by;

reading the image portion out of the frame store memory means in response to the identification information, and

providing the image portion read out of the frame store memory means as a part of the frame of the output video signal.