Speech interface for an automated endoscopic system

• US 6,965,812 B2
• Filed: 03/11/2002
• Issued: 11/15/2005
• Est. Priority Date: 09/22/1994
• Status: Expired due to Fees

• Current Assignee: Intuitive Surgical Operations, Inc. (Intuitive Surgical Holdings, LLC)
• Original Assignee: Computer Motion Inc. (Intuitive Surgical Holdings, LLC)
• Inventors: Wang, Yulun, Uecker, Darrin
• Primary Examiner(s): Davis, George B.
• Application Number:
  US10/095,488
• Publication Number:
  US 20020183894A1
• Time in Patent Office: 1,345 Days
• Field of Search: 700/258, 700/245, 318/567
• US Class Current: 700/258
• CPC Class Codes: A61B 1/0004 for electronic operation
• CPC Class Codes: A61B 1/00042 for mechanical operation
• CPC Class Codes: A61B 1/00149 using articulated arms
• CPC Class Codes: A61B 1/0016 using motor drive units
• CPC Class Codes: A61B 2017/00203 with speech control or speech recognition
• CPC Class Codes: A61B 34/10 Computer-aided planning, simulation or modelling of surgical operations
• CPC Class Codes: A61B 34/70 Manipulators specially adapted for use in surgery
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G16H 40/63 for local operation

Abstract

A robotic system which controls the movement of a surgical instrument in response to voice commands from the user. The robotic system has a computer controlled arm that holds the surgical instrument. The user provides voice commands to the computer through a microphone. The computer contains a phrase recognizer that matches the user'"'"' speech with words stored in the computer. Matched words are then processed to determine whether the user has spoken a robot command. If the user has spoken a recognized robot command the computer will move the robotic arm in accordance with the command.

Claims

1. A surgical robotic system for use in minimally invasive surgery, the minimally invasive surgery performed at an internal surgical site of a patient body, the surgical robotic system comprising:

a surgical instrument having a proximal end and a distal end, the instrument extendible distally through an incision to the internal surgical site;

a robot supporting the proximal end of the instrument, the robot moving the instrument in response to motor signals;

a voice recognition system for inputting a plurality of spoken instructions, each spoken instruction including a spoken qualifier and a spoken direction of movement, the voice recognition system changing from a first state to a second state in response to the spoken qualifier and identifying the spoken direction of movement from among a plurality of alternative directions in the second state; and

a computer coupling the voice recognition system to the robot, the computer generating the motor signals in response to the spoken instructions to move the distal end of the surgical instrument at the internal surgical site in the spoken directions by pivoting the instrument about the incision.

2. The surgical robotic system of claim 1, wherein the surgical instrument comprises an endoscope, and further comprising a monitor coupled to the endoscope for displaying an image of the internal surgical site.

3. The surgical robotic system of claim 2, wherein the distal end of the surgical instrument comprises a camera tip defining a camera coordinate frame, and wherein the computer calculates transformations between the camera coordinate frame and coordinate frames of the robot.

4. The surgical robotic system of claim 3, wherein the computer generates the motor signals using the transformations so that the camera tip moves in the internal surgical site to effect an instructed change in the image shown in the display.

5. The surgical robotic system of claim 3, wherein the camera tip defines a tip angle, and wherein the computer generates the motor signals using the transformations so that the camera tip moves in the internal surgical site to effect a change in zoom of the image shown in the display in response to a change in zoom instruction.

6. The surgical robotic system of claim 1, wherein the first surgical instrument is any of the following:

a surgical tool;

a forceps;

a device positioned within the patient'"'"'s body for transmitting an image outside of the patient'"'"'s body;

a laparoscope; and

a medical telescope.

7. A minimally invasive surgical robotic method comprising:

inserting a distal end of a surgical instrument through an incision to an internal surgical site;

manipulating a proximal end of the instrument with a robot in response to motor signals;

inputting spoken instructions, including a qualifier and a desired direction of movement, into a voice recognition system;

changing a state of the voice recognition system from a first state to a second state in response to the qualifiers;

selecting the desired direction of movement from a plurality of alternative directions of movement associated with the second state;

generating the motor signals in response to the spoken instructions with a computer; and

moving the distal end of the surgical instrument in the desired directions at the internal surgical site by pivoting the instrument about the incision with the robot in response to the motor signals.

8. The surgical robotic method of claim 7, further comprising displaying an image of the internal surgical site on a monitor.

9. The surgical robotic method of claim 8, wherein the surgical instrument comprises an endoscope, and further comprising inputting a change of image instruction and moving the endoscope with the robot so as to effect the instructed image change.

10. The surgical robotic method of claim 9, wherein the instructed image change comprises a change in zoom.

11. The surgical robotic method of claim 10, wherein the endoscope has a camera tip disposed at an angle relative to a shall of the camera.

12. The surgical robotic method of claim 7, wherein the distal end of the surgical instrument defines a distal coordinate reference frame, and further comprising calculating transformations between the distal coordinate frame and a coordinate frame of the robot with the computer and using the transformation to generate the motor signals.

13. A system for positioning a surgical instrument relative to a patient'"'"'s body, comprising:

a robotic manipulator having at least one controlled degree of freedom;

a controller controlling the robotic manipulator;

an instrument holder attaching a first surgical instrument to the robotic manipulator so that the first surgical instrument can pivot about an aperture into the patient body; and

a voice recognition system coupled to the controller, the voice recognition system permitting a surgeon to verbally command motions of the robotic manipulator, the instrument initiating pivoting about the aperture in response to the commands only when the commands are preceded by a verbal qualifier, the instrument motions halting when the command comprises a stop command without a verbal qualifier.

14. A system for positioning a surgical instrument relative to a patient'"'"'s body, as in claim 13, wherein the voice recognition system further permits the surgeon to select commands or operating modes from menus.

15. A system for positioning a surgical instrument relative to a patient'"'"'s body, as in claim 13, further comprising a speech synthesis system to provide the surgeon with voice messages containing information about the operation of the system.

16. A system for positioning a surgical instrument relative to a patient'"'"'s body, as in claim 15, wherein the first surgical instrument is a surgical camera and the speech synthesis system provides a message to the surgeon stating information about the movement of the surgical camera.

17. A system for positioning a surgical instrument relative to a patient'"'"'s body, as in claim 15, wherein the first surgical instrument is a surgical camera and the speech synthesis system provides a message to the surgeon stating information about the movement of the surgical camera effect a zoom operation of the surgical camera.

18. A surgical system for use in minimally invasive surgery performed at an internal surgical site of a patient body, the system comprising:

an endoscope having a proximal portion and a distal end, said endoscope extendable distally through an aperture in a body wall to the internal surgical site;

a display operatively coupled to said endoscope for allowing a surgeon to view an image of the internal surgical site, the image changing in response to signals;

a voice recognition system for inputting a plurality of spoken instructions, the spoken instructions each including a qualifier and a command selected from a plurality of commands so as to effect a desired change in the image, the voice recognition system changing from a first state to a second state in response to the qualifier and only accepting the plurality of command in the second state; and

a computer operatively coupling the voice recognition system to the endoscope, the computer generating the signals in response to the spoken instructions to effect the desired change to the image of the internal surgical site.

19. The surgical system of claim 18, wherein the computer generates said signals so as to effect an instructed change in zoom of the image shown in the display in response to a change in zoom instruction.

20. The surgical system of claim 18, wherein the voice recognition system comprises a microphone for detecting spoken instructions and a speaker for providing audible messages to a surgeon regarding operation of the system.

21. The system of claim 20, wherein said audible messages comprise audible feedback indicating successful receipt of said spoken instructions.

22. The system of claim 20, wherein said audible messages comprise synthesized voice messages.

23. A minimally invasive surgical method comprising:

inserting a distal end of an endoscope through an aperture in a body wall of a patient to an internal surgical site;

displaying an image of the internal surgical site for perception by a surgeon controlling said endoscope in response to signals;

inputting a plurality of spoken instructions into a voice recognition system, each instruction comprising a qualifier which changes a state of the voice recognition system and a subsequent command;

generating, with a computer, said motor signals in response to said spoken instructions; and

changing the image in response to the signals.

24. The method of claim 23, further comprising inputting a change of image instruction and moving the endoscope with a robotic arm so as to effect the instructed image change.

25. The method of claim 23, wherein the instructed image change comprises a change in zoom.

26. The method of claim 23, further comprising providing audible messages containing information about the robotic surgery.

27. The method of claim 26, wherein said audible messages comprise audible feedback indicating successful receipt of said spoken instruction.

28. The method of claim 26, wherein said audible messages comprise synthesized voice messages.

29. The method of claim 26, wherein said audible messages comprise audible feedback indicating undesired movement of the distal end of the surgical instrument.

30. A system for positioning an endoscopic surgical instrument relative to a surgical site within a patient'"'"'s body, comprising:

a robotic manipulator having at least one controlled degree of freedom;

a controller that is adapted to be capable of controlling the robotic manipulator;

an instrument holder for coupling said surgical instrument to said robotic manipulator; and

a voice recognition command input system operatively coupled to the controller, the voice recognition system having a first state and a second state, the voice recognition system changing from the first state to the second state in response to a verbal qualifier, the voice recognition system permitting a surgeon to specify desired motion of the surgical instrument to the controller only when in the second state, such that the robotic manipulator moves said instrument to a position relative to the patient'"'"'s body, as specified by the surgeon using the command input system, by pivoting the instrument about an aperture into the patient'"'"'s body.

31. A surgical system for use in an operating room to control a surgical instrument, the surgical instrument having a control input accepting control signals, the system comprising:

a voice recognition system having a first state and a second state, the voice recognition system in the first state accepting a verbal qualifier and changing from the first state to the second state in response to the verbal qualifier, the voice recognition system in the second state accepting a verbal command and comparing the verbal command to a plurality of command alternatives associated with the second state so as to identify a matching command; and

a processor operatively coupling the voice recognition system to the device, the processor generating the control signals at least in part in response to the identification of the matching command.

32. The system of claim 31, wherein the voice recognition system also has a third state, the voice recognition system changing from the second state to the third state in response to the verbal command being identified as a matching command from among the plurality of command alternatives, the voice recognition system in the third state accepting another plurality of command alternatives associated with the third state, the other command alternatives of the third state being different than the command alternatives of the second state.

33. The system of claim 32, wherein the voice recognition system has a fourth state, the voice recognition system changing from the second state to the fourth state in response to another verbal command selected from among the plurality of command alternatives of the second state, the voice recognition system in the fourth state accepting a plurality of command alternatives associated with the fourth state.