Method and system for object detection for instrument control

• US 5,649,021 A
• Filed: 06/07/1995
• Issued: 07/15/1997
• Est. Priority Date: 06/07/1995
• Status: Expired due to Fees

• Current Assignee: Sarnoff Corporation (SRI International, Inc.)
• Original Assignee: David Sarnoff Research Center, Inc.
• Inventors: Pletcher, Timothy A., Aceti, John G., Matey, James R.
• Primary Examiner(s): Razavi, Michael T.
• Assistant Examiner(s): CHANG, JON CARLTON
• Application Number:
  US08/472,832
• Time in Patent Office: 769 Days
• Field of Search: 382/128, 382/164, 382/163, 382/165, 382/282, 382/291, 382/175, 382/317, 382/152, 382/100, 600/118, 600/134, 348/143, 348/156, 348/152, 348/65, 606/38, 606/39, 606/46
• US Class Current: 382/128
• CPC Class Codes: A61B 18/1482 having a long rigid shaft for accessing the inner body transcutaneously in minimal invasive surgery, e.g. laparoscopy
• CPC Class Codes: A61B 2018/1422 Hook
• CPC Class Codes: A61B 90/92 coded with colour
• CPC Class Codes: G06V 10/40 Extraction of image or video features

Abstract

An apparatus and method for controlling an instrument is provided. The device may be a cutting tool for use in surgery, or other type of device applied to transform a specific object located in a defined area. The apparatus comprises a marker having a predetermined color. The marker is applied to the specific device or tool to form a marked device. An imager generates video signals representing a color image of the defined area. A color detector processes the imager signals to detect a range of colors in the image represented by the imager signals. The range of colors includes the predetermined color. The color detector provides an output signal representing instances of signals in the detected color range. A detection mechanism processes the output signal of the color detector to provide an indication of whether the detected instances represented by the output signal conform to the marked device. A control mechanism disables the marked device if the detected instances do not conform to the marked device, in response to the indication provided by the detection mechanism.

Claims

1. Apparatus for controlling a device, said apparatus comprising:

a marker having a predetermined color and being applied to the device to form a marked device;

an imager which generates imager signals representing a color image of a region proximate to the imager;

a detector which processes the imager signals to detect a range of colors, which includes the predetermined color, in the image represented by the signals and for providing an output signal representing instances of imager signals in the detected color range;

detection means for processing the output signal of the color detector to provide an indication of whether the output signal of the color detector conforms to the marked device, thereby to indicate whether the marked device is included in the color image represented by the imager signals; and

control means responsive to the detection means for disabling the marked device whenever the detection means indicates that the marked device is not in the color image, and for allowing the marked device to be enabled whenever the detection means indicates that the marked device is in the color image.

2. The apparatus of claim 1, wherein the detection means comprises:

means for counting a number of pixels within the detected color range in a single frame of said imager signals, based on said output signal, and

means for indicating that the detected instances conform to the marked device if the number of pixels exceeds a predetermined threshold value.

3. The apparatus of claim 1, wherein the detection means comprises:

means for estimating a contour of the marked device,

means for determining a number of pixels that are within the detected color range and within said contour, in a single frame of said imager signals, based on said output signal, and

means for indicating that the detected instances conform to the marked device if the number of instances exceeds a predetermined threshold value.

4. The apparatus of claim 3, wherein said determining means includes spatial filter means for extracting spatial features from the output signal to determine whether said detected instances conform to said marked device.

5. The apparatus of claim 1, wherein the device is a surgical tool.

6. The apparatus of claim 1, further comprising a laparoscope having a field of view and coupled to the imager for providing the color image to the imager, wherein:

the device is an electrosurgical tool, and

the control means includes means for disabling the electrosurgical tool if the electrosurgical tool is outside the field of view of the laparoscope.

7. The apparatus of claim 6, wherein the predetermined color is blue.

8. Apparatus for controlling a device that is applied to transform a specific object, said apparatus comprising:

a marker having a predetermined color and being applied to the device to form a marked device;

an imager which generates imager signals representing a color image of a region proximate to the imager;

means for selecting a subset of said imager signals to form a reduced area image of the region;

a color detector which processes the subset of the imager signals to detect a range of colors, which includes the predetermined color, in the reduced area image represented by the subset of the imager signals and for providing an output signal representing instances of imager signals in the detected color range;

detection means for processing the output signal of the color detector to provide an indication of whether the output signal of the color detector conforms to the marked device, thereby to indicate whether the marked device is included in the reduced area image represented by the imager signals; and

control means responsive to the detection means for disabling the marked device whenever the detection means indicates that the marked device is not in the reduced area image, and for allowing the marked device to be enabled whenever the detection means indicates that the marked device is in the reduced area image.

9. Apparatus for controlling a tool that is used to cut tissue during surgery, said apparatus comprising:

a marker having a predetermined color and being applied to the tool to form a marked tool;

a laparoscope having a field of view which includes a region in which the tissue to be cut during surgery is located and receives light therefrom;

an imager coupled to receive light from the laparoscope and generate imager signals representing a color image of the field of view of the laparoscope;

a color detector which processes the imager signals to detect a range of colors, which includes the predetermined color, in the image represented by the imager signals and for providing an output signal representing instances of imager signals in the detected color range;

detection means for processing the output signal of the color detector to provide an indication of whether the output signal of the color detector conforms to the marked tool, thereby to indicate whether the marked tool is in the field of view of the laparoscope; and

control means responsive to the detection means for disabling the marked tool whenever the detection means indicates that the marked tool is outside the field of view of the laparoscope, and for allowing the marked tool to be enabled whenever the detection means indicates that the marked tool is within the field of view of the laparoscope.

10. The apparatus of claim 9, wherein the predetermined color is blue.

11. A method for controlling a device that is applied to transform a specific object, said method comprising the steps of:

(a) applying a marker having a predetermined color to the device to form a marked device;

(b) generating imager signals which represent a color image of a region proximate to the imager;

(c) detecting a range of colors in the color image represented by the imager signals, said range of colors including the predetermined color;

(d) forming an output signal representing instances of imager signals in the detected color range;

(e) processing the output signal to provide an indication of whether the output signal of the color detector conforms to the marked device, thereby identifying whether the marked device is included in the color image represented by the imager signals;

(f) disabling the marked device whenever the indication identifies the marked device as not being included in the color image represented by the imager signals; and

(g) allowing the marked device to be enabled whenever the indication identifies the marked device as being included in the color image represented by the imager signals.

12. A method according to claim 11, wherein step (c) includes the steps of:

analyzing said output signal to determine a number of pixels within the detected color range in a single frame of said imager signals, and

indicating that the detected instances conform to the marked device if the number of pixels exceeds a predetermined threshold value.

13. The method of claim 11, wherein step (c) includes the steps of:

estimating a contour of the marked device,

analyzing said output signal to determine a number of pixels that are within the detected color range and within said contour, in a single frame of said imager signals, and

indicating that the detected instances conform to the marked device if the number of instances exceeds a predetermined threshold value.

14. The method of claim 11, wherein the imager is coupled to an instrument which has a field of view and provides the color image to the imager, the device is a tool, and step (f) includes:

disabling the tool if the tool is outside the field of view of the instrument.

15. The method of claim 14, further comprising executing the following steps before step (b):

(1) sampling a plurality of training imager signals during at least one surgical procedure;

(2) extracting color histogram data from the sampled training imager signals; and

(3) identifying the range of colors based on the color histogram data.

16. The method of claim 14, further comprising executing the following steps before step (b):

(1) sampling training imager signals during first and second surgical procedures;

(2) extracting a first set of color histogram data from the training imager signals sampled during the first surgical procedure, and a second set of color histogram data from the training imager signals sampled during the second surgical procedure;

(3) combining the first and second sets of color histogram data to form a composite color space probability density function; and

(4) identifying the range of colors based on the composite color space probability density function.

17. The method of claim 11, wherein the predetermined color is blue.