Speech interface for an automated endoscope system

• US 7,395,249 B2
• Filed: 09/15/2004
• Issued: 07/01/2008
• Est. Priority Date: 09/22/1994
• Status: Expired due to Term

• Current Assignee: Intuitive Surgical Operations, Inc. (Intuitive Surgical Holdings, LLC)
• Original Assignee: Intuitive Surgical Incorporated (Intuitive Surgical Holdings, LLC)
• Inventors: Wang, Yulun, Uecker, Darrin
• Primary Examiner(s): Holmes; Michael B.
• Application Number:
  US10/942,374
• Publication Number:
  US 20050033580A1
• Time in Patent Office: 1,385 Days
• Field of Search: 700/258, 700/245, 700/264, 606/130, 606/425, 128/897, 704/246, 600/425, 706/14
• US Class Current: 706/14
• CPC Class Codes: A61B 1/0004 for electronic operation
• CPC Class Codes: A61B 1/00042 for mechanical operation
• CPC Class Codes: A61B 1/00149 using articulated arms
• CPC Class Codes: A61B 1/0016 using motor drive units
• CPC Class Codes: A61B 2017/00203 with speech control or speech recognition
• CPC Class Codes: A61B 34/10 Computer-aided planning, simulation or modelling of surgical operations
• CPC Class Codes: A61B 34/70 Manipulators specially adapted for use in surgery
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G16H 40/63 for local operation

Abstract

A robotic system which controls the movement of a surgical instrument in response to voice commands from the user. The robotic system has a computer controlled arm that holds the surgical instrument. The user provides voice commands to the computer through a microphone. The computer contains a phrase recognizer that matches the user'"'"' speech with words stored in the computer. Matched words are then processed to determine whether the user has spoken a robot command. If the user has spoken a recognized robot command the computer will move the robotic arm in accordance with the command.

Claims

1. A voice recognition system for use with a surgical instrument, the system comprising:

a processor coupled to a microphone and to a memory, the processor having a first state and a second state;

wherein the microphone receives a plurality of spoken surgical instructions, each of the spoken surgical instructions including a spoken qualifier and a spoken command, the plurality of spoken surgical instructions including a first surgical instruction having a first spoken qualifier and a first spoken command;

wherein the memory stores a first plurality of allowable commands associated with the second state of the processor; and

wherein in the first state, the processor is configured to;

receive the first surgical instruction and determine if the first spoken qualifier included in the first surgical instruction satisfies a first condition;

if the first spoken qualifier satisfies the first condition, then advance to the second state; and

if the first spoken qualifier does not satisfy the first condition, then remain in the first state; and

wherein in the second state, the processor is configured to;

determine whether the first spoken command included in the first surgical instruction is among the first plurality of allowable commands associated with the second state; and

if the first spoken command is among the first plurality of allowable commands, then transmit a first command signal to the surgical instrument in response to the first spoken command being among the first plurality of allowable commands.

2. The voice recognition system of claim 1, wherein the surgical instrument includes an endoscope configured to be coupled to a monitor, the processor configured to be coupled to the endoscope so as to alter an image from the endoscope that is shown on the monitor.

3. The voice recognition system of claim 2, the surgical instrument including a robotic arm supporting the endoscope, a distal end of the endoscope including a tip defining a viewing coordinate frame, wherein the processor is configured to calculate transformations between the viewing coordinate frame and a coordinate frame of the robotic arm, and the first command signal includes motor signals derived from the transformations so that the tip moves in an internal surgical site to effect an instructed change in the image shown on the monitor.

4. The voice recognition system of claim 1, wherein:

the processor is configured to be coupled to the surgical instrument.

5. The voice recognition system of claim 1, wherein the plurality of spoken surgical instructions includes a second instruction having a second spoken qualifier and a second spoken command, the memory operable to store a second plurality of allowable commands, the processor operable to change to a third state in response to the second spoken qualifier satisfying a second condition, the processor in the third state configured to determine whether the second spoken command is among the second plurality of allowable commands associated with the third state, the processor operable to generate a second command signal in response to the second spoken command being among the second plurality of allowable commands.

6. The voice recognition system of claim 5, wherein the processor has a fourth state, the processor operable to change to the fourth state in response to the second spoken command, the processor in the fourth state operable to accept a third plurality of allowable commands stored in the memory and associated with the fourth state.

7. The voice recognition system of claim 1, further comprising a speaker coupled to the processor for generating audible messages to a surgeon regarding operation of the system.

8. The voice recognition system of claim 7, wherein the audible messages include audible feedback indicating successful receipt of each spoken surgical instruction.

9. The voice recognition system of claim 8, wherein the audible messages include synthesized voice messages.

10. The voice recognition system of claim 1, wherein the microphone accepts a spoken stop command, and the processor is configured to transmit a stop command signal to the surgical instrument in response to the spoken stop command without an associated spoken qualifier.

11. The voice recognition system of claim 10, wherein the stop command signal from the processor is configured to inhibit potential injury to the patient that might otherwise be inflicted by the surgical instrument.

12. The voice recognition system of claim 1, wherein:

determining whether the first spoken command is among the first plurality of allowable commands includes comparing the first spoken command to at least one of the allowable commands in the first plurality of allowable commands.

20. The voice recognition system of claim 1, wherein the first plurality of allowable commands comprises a save command.

21. The voice recognition system of claim 1, wherein the first plurality of allowable commands comprises a return command.

22. The voice recognition system of claim 1, wherein the first plurality of allowable commands comprises a track instrument command.

23. The voice recognition system of claim 1, wherein the first plurality of allowable commands comprises a track head command.

26. The voice recognition system of claim 1, wherein:

the plurality of spoken surgical instructions includes a second instruction including the first spoken qualifier and a second spoken command including a first part and a second part;

the memory is further operable to store a second plurality of allowable commands associated with a third state of the processor;

the processor is further operable in the second state to;

determine whether the first part of the second spoken command is among the first plurality of allowable commands associated with the second state;

if the first part is among the first plurality of allowable commands, then advance to the third state and determine whether the second part of the second spoken command is among the second plurality of allowable commands associated with the third state; and

if the second part is among the second plurality of allowable commands, then transmit a second command signal to the surgical instrument in response to the second spoken command.

13. A method comprising:

receiving a spoken surgical instruction, the spoken surgical instruction comprising a verbal qualifier and a verbal control command, wherein the verbal qualifier precedes the verbal control command;

determining whether the verbal qualifier matches an expected qualifier associated with a medical device;

if the verbal qualifier matches the expected qualifier, then determining whether the verbal control command is among one or more predefined commands from a library of multiple predefined commands; and

providing an output command signal that corresponds to the verbal control command to the medical device only if the verbal qualifier matches the expected qualifier and the verbal control command is among the one or more predefined commands.

14. The method of claim 13, further comprising:

providing audio or visual feedback after receiving the verbal control command.

15. The method of claim 13, wherein:

the expected qualifier includes a name of the medical device.

16. The method of claim 13, wherein:

the medical device comprises a robotic arm.

17. The method of claim 13, wherein:

determining whether the verbal qualifier matches the expected qualifier includes comparing the verbal qualifier to the expected qualifier; and

determining whether the verbal control command is among the one or more predefined commands includes comparing the verbal control command to at least one of the one or more predefined commands.

18. A voice recognition system for use with a surgical instrument, the system comprising:

a microphone for inputting a plurality of spoken surgical instructions, each of the spoken surgical instructions including a spoken qualifier and a spoken command, the plurality of spoken surgical instructions including a first instruction comprising a first spoken qualifier and a first spoken command, wherein the first spoken command comprises a first portion and a second portion;

a memory for storing a first plurality of allowable commands and a second plurality of allowable compounds; and

a processor coupled to the microphone and the memory, the processor having a first state, a second state, and a third state, wherein the first plurality of allowable commands are associated with the second state, wherein the second plurality of allowable commands are associated with the third state, and wherein the processor is configured to;

in the first state, in response to the first spoken qualifier matching an expected qualifier, advance to the second state;

in the second state, determine if the first portion of the first spoken command is among the first plurality of allowable commands associated with the second state and to advance to the third state in response to the first portion being among the first plurality of allowable commands;

in the third state, determine if the second portion of the first spoken command is among the second plurality of allowable commands associated with the third state; and

in response to the second portion of the first spoken command being among the second plurality of allowable commands provide a first command signal to the surgical instrument corresponding to the first spoken command.

19. A method for controlling a surgical instrument, the method comprising:

in a first state, receiving a plurality of spoken surgical instructions, each of the spoken surgical instructions including a spoken qualifier and a spoken command, the plurality of spoken surgical instructions including a first surgical instruction having a first spoken qualifier and a first spoken command;

determining if the first spoken qualifier included in the first surgical instruction satisfies a first condition;

if the first spoken qualifier satisfies the first condition, then advancing to a second state;

in the second state, determining whether the first spoken command included in the first surgical instruction is among a plurality of allowable commands associated with the second state; and

if the first spoken command is among the first plurality of allowable commands, then transmitting a first command signal to the surgical instrument in response to the first spoken command being among the plurality of allowable commands.

24. A method comprising:

receiving a first spoken surgical instruction, the first spoken surgical instruction comprising a verbal qualifier and a first verbal control command, the verbal qualifier preceding the first verbal control command;

determining whether the verbal qualifier matches an expected qualifier;

if the verbal qualifier matches the expected qualifier, then determining whether the first verbal control command is among one or more predefined commands from a first library of multiple predefined commands;

providing a first output command signal that corresponds to the first verbal control command to a medical device only if the verbal qualifier matches the expected qualifier and the first verbal control command is among the predefined commands in the first library;

receiving a second spoken surgical instruction, the second spoken surgical instruction comprising the verbal qualifier and a second verbal control command, the verbal qualifier preceding the second verbal control command;

determining whether the verbal qualifier matches the expected qualifier;

if the verbal qualifier matches the expected qualifier, then determining whether the second verbal control command is among one or more predefined commands from a second library of multiple predefined commands; and

providing a second output command signal that corresponds to the second verbal control command to the medical device only if the verbal qualifier matches the expected qualifier and the second verbal control command is among the predefined commands in the second library.

25. A method comprising:

receiving a spoken surgical instruction, the spoken surgical instruction comprising a verbal qualifier and a verbal control command comprising a first part and a second part, the verbal qualifier preceding the verbal control command;

determining whether the verbal qualifier matches an expected qualifier;

if the verbal qualifier matches the expected qualifier, then determining whether the first part of the verbal control command is among one or more predefined commands from a first library of multiple predefined commands;

if the first part of the verbal control command is among the predefined commands in the first library, then determining whether the second part of the verbal control command is among one or more predefined commands in a second library of multiple predefined commands; and

providing an output command signal to a medical device, wherein the output command signal corresponds to the first and second parts of the verbal control command only if;

the verbal qualifier matches the expected qualifier, the first part of the verbal control command is among the predefined commands in the first library, and the second part of the verbal control command is among the predefined commands in the second library.