System and method for hazard mitigation in voice-driven control applications

• US 9,514,746 B2
• Filed: 09/26/2006
• Issued: 12/06/2016
• Est. Priority Date: 09/26/2006
• Status: Active Grant

• Current Assignee: Storz Endoskop Produktions GmbH (Karl Storz Se & Co. Kg)
• Original Assignee: Storz Endoskop Produktions GmbH (Karl Storz Se & Co. Kg)
• Inventors: Wang, Gang, Contolini, Matteo, Zheng, Chengyi, Stiller, Heinz-Werner
• Primary Examiner(s): ROBERTS, SHAUN A
• Application Number:
  US11/527,142
• Publication Number:
  US 20080077408A1
• Time in Patent Office: 3,724 Days
• Field of Search: 704/275, 704/274, 704/231
• US Class Current: 1/1
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: G10L 2015/228 of application context
• CPC Class Codes: G10L 25/51 for comparison or discrimination

Abstract

A speech recognition and control system including a receiver for receiving an audio input, an event detector for analyzing the audio input and identifying at least one event of the audio input, a recognizer for interpreting at least a portion of the audio input, a database including a plurality of rules, and a controller for generating a control command based on the at least one event and at least one rule.

Claims

1. A speech recognition and control system for controlling a medical device in an operating room, comprising:

a medical device;

a microphone receiving an audio input including one or more speech commands;

an event detector analyzing the audio input and identifying at least one event of the audio input;

a database including a plurality of rules, a first one of the plurality of rules immediately stopping a system or device activity as soon as it detects that the one or more speech commands is begun within a predetermined period of time after the activity was commenced, and a second one of the plurality of rules alerting if a disconnection with the microphone is detected; and

a controller adapted to determine a system status including actions currently being performed by devices;

the controller further adapted to determine whether or not hazard mitigation is necessary based on a comparison of the at least one event, at least one rule and the system status;

if having determined that hazard mitigation is necessary, the controller sending a control command to the medical device instructing it to perform an action.

2. The system of claim 1, wherein the at least one event includes an audio input duration exceeding a predetermined duration.

3. The system of claim 1 wherein the at least one event includes an audio input duration less than a predetermined duration.

4. The system of claim 1, wherein the at least one event includes one of a microphone malfunction.

5. The system of claim 1, wherein the at least one event includes a functioning microphone.

6. The system of claim 1, wherein the at least one event is an audio input error.

7. The system of claim 1, wherein the at least one event includes at least one of a microphone connection and a microphone disconnection.

8. The system of claim 1, wherein said controller generates at least one speech command based on an interpretation of at least a portion of the audio input.

9. The system of claim 8, wherein the controller sends the at least one speech command to a device operable by the at least one speech command.

10. The system of claim 1, wherein the control command countermands the activity based on the first one of the plurality of rules.

11. The system of claim 1, wherein said medical device is a surgical table or an endoscope.

12. The system of claim 1, wherein the device has at least one user interface.

13. The system of claim 1, wherein the predetermined period of time occurs immediately after the activity was commenced.

14. The system of claim 1, wherein a third one of the plurality of rules alerts if implementing the one or more speech commands would result in an unsafe condition.

15. A speech recognition and control system for controlling a medical device in an operating room, comprising:

a medical device;

a microphone receiving an audio input including one or more speech commands;

a processor in communication with said microphone;

software executing on said processor analyzing the audio input and identifying at least one event of the audio input;

a database in communication with said processor, said database including a plurality of rules, the plurality of rules comprising one or more of a rule that provides for any system or device activity to implement an action, a rule to stop an action, or a rule to issue a warning, and said plurality of rules being predefined rules that are dependent on a system status, a first one of the plurality of rules immediately stopping a system or device activity if the one or more speech commands is begun within a predetermined period of time after the activity was commenced, and a second one of the plurality of rules alerting if a disconnection with the microphone is detected;

software executing on said processor determining the system status including actions currently being performed by devices; and

software executing on said processor determining whether to implement an action, stop a previously implemented action, or issue a warning for hazard mitigation based on a comparison of the at least one event, at least one rule and the system status;

if having determined that hazard mitigation is necessary, the processor sending a control command to the medical device instructing it to perform an action;

wherein said plurality of rules mitigate conflicting commands with the system status.

16. The system of claim 15, wherein the action is a hazard mitigating action.

17. A method of mitigating hazards in a voice-driven control system for controlling a medical device in an operating room, comprising the steps of:

providing a medical device;

providing a database including a plurality of rules, a first one of the plurality of rules immediately stopping a system or device activity if one or more speech commands is begun within a predetermined period of time after the activity was commenced, and a second one of the plurality of rules alerting if a disconnection with a microphone is detected;

receiving an audio input including the one or more speech commands;

identifying at least one event of the audio input;

determining a system status including actions currently being performed by devices;

determining whether hazard mitigation is necessary based on a comparison of the at least one event, the system status, and at least one rule; and

if having determined that hazard mitigation is necessary,

sending a control command to the medical device in communication with the voice-driven control system instructing it to perform an action.

18. The method of claim 17, wherein the control command countermands a speech command to the medical device.

19. The method of claim 17, wherein the at least one event includes one of an audio input duration above a predetermined duration and the audio input duration below the predetermined duration.

20. The method of claim 17, wherein the at least one event includes a microphone malfunction.

21. The method of claim 17, wherein the event is an audio input error.

22. The method of claim 17, wherein the at least one event includes a disconnection of a device for receiving the audio input and converting the audio input into digital data.

23. The method of claim 22, further comprising the step of generating a warning in response to the event.

24. The method according to claim 17, further comprising directing the voice-driven control system to stop receiving speech commands based on the plurality of rules.

25. A speech recognition and control system for controlling a medical device in an operating room, comprising:

a medical device;

one or more devices receiving an audio input including one or more speech commands and converting the audio input into digital data;

an event detector identifying at least one event of the device or the audio input;

a database including a plurality of rules, the plurality of rules comprising one or more of a rule to implement an action, a rule to stop an action, or a rule to issue a warning, and said plurality of rules being dynamically generated during operation of the system, a first one of the plurality of rules immediately stopping a system or device activity if the one or more speech commands is begun within a predetermined period of time after the activity was commenced, and a second one of the plurality of rules alerting if a disconnection with a microphone is detected; and

a controller adapted to determine a system status including actions currently being performed by devices;

the controller further adapted to determine whether or not hazard mitigation is necessary based on a comparison of the at least one event, at least one rule and the system status;

if having determined that hazard mitigation is necessary, the controller sending a control command to the medical device instructing it to perform;

wherein said plurality of rules mitigate conflicting commands with the system status.

26. The system of claim 25, wherein the at least one event includes a disconnection of said one or more devices.

27. The system of claim 26, wherein said controller generates a warning based on the at least one event.

28. The system of claim 25, wherein the at least one event includes a malfunction of said one or more devices.