Surgical system with voice control

• US 10,028,794 B2
• Filed: 12/19/2016
• Issued: 07/24/2018
• Est. Priority Date: 12/19/2016
• Status: Active Grant

• Current Assignee: Cilag Gmbh International (Johnson & Johnson)
• Original Assignee: Ethicon, LLC (Johnson & Johnson)
• Inventors: Cagle, David J., Smith, Eric, Aldridge, Jeffrey L., Mootoo, Mary E., Asher, Ryan
• Primary Examiner(s): Chawan, Vijay B
• Application Number:
  US15/383,564
• Publication Number:
  US 20180168755A1
• Time in Patent Office: 582 Days
• Field of Search: 704270, 704275, 704231, 704246, 606130, 3403847, 340 354, 348 77, 700275
• CPC Class Codes: A61B 2017/00203 with speech control or speech recognition
• CPC Class Codes: A61B 2017/00216 with eye tracking or head position tracking control
• CPC Class Codes: A61B 2017/00221 with wireless transmission of data, e.g. by infrared radiation or radiowaves
• CPC Class Codes: A61B 2034/2063 Acoustic tracking systems, e.g. using ultrasound
• CPC Class Codes: A61B 2034/301 for introducing or steering flexible instruments inserted into the body, e.g. catheters or endoscopes
• CPC Class Codes: A61B 2034/302 specifically adapted for manipulations within body cavities, e.g. within abdominal or thoracic cavities
• CPC Class Codes: A61B 2090/3612 with images taken automatically
• CPC Class Codes: A61B 2090/502 Headgear, e.g. helmet, spectacles
• CPC Class Codes: A61B 34/25 User interfaces for surgical systems
• CPC Class Codes: A61B 34/30 Surgical robots
• CPC Class Codes: A61B 34/35 for telesurgery
• CPC Class Codes: A61B 34/74 Manipulators with manual electric input means
• CPC Class Codes: G10L 17/00 Speaker identification or verification techniques
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command

Abstract

A surgical system includes a plurality of voice sensors located in a surgical environment and configured to detect sound and generate a first plurality of signals. The surgical system also includes a position indicator, in proximity to a designated user, configured to indicate a first position of the designated user and generate a second signal representative of the first position. The surgical system further includes a processor configured to receive the first plurality of signals and the second signal and determine, based on the first plurality of signals, a second position. The processor is also configured to compare the detected sound with registered voice command of the designated user stored in a memory to verify the designated user'"'"'s credentials, and send a command signal to a surgical instrument to carry out an operation related to the voice command based on at least one of the verification of the designated user'"'"'s credentials, the first position and the second position.

Claims

1. A surgical system, comprising:

a plurality of voice sensors located in a surgical environment and configured to detect sound and generate a first plurality of signals;

a position indicator, located in the surgical environment in proximity to a designated user, configured to indicate a first position of the designated user in the surgical environment and generate a second signal representative of the first position; and

a processor configured to;

receive the first plurality of signals and the second signal,

determine, based on the first plurality of signals, a second position within the surgical environment,

compare the detected sound with a registered voice command of the designated user stored in a memory to verify the designated user'"'"'s credentials, and

send a command signal to a surgical instrument to carry out an operation related to the voice command based on at least one of the verification of the designated user'"'"'s credentials, the first position, and the second position.

2. The system of claim 1, wherein the detected sound is a voice command generated by the designated user.

3. The system of claim 2, wherein the voice command is detected by at least one voice sensor in a headset worn by the designated user.

4. The system of claim 3, wherein the headset includes a camera configured to detect a motion of lips of the designated user during the generation of the voice command and to send a signal related to the detected motion of lips to the processor.

5. The system of claim 4, wherein the processor is configured to determine whether the detected motion of lips of the designated user matches the generated voice commands and to send the command signal when the detected motion of lips of the designated user matches the generated voice commands.

6. The system of claim 2, wherein the voice command is detected by at least one voice sensor disposed on the surgical instrument.

7. The system of claim 1, wherein the second position is determined based on one or more delays in detection of the sound by one or more of the plurality of voice sensors.

8. The system of claim 1, wherein the processor is configured to send the command signal when the difference between the first position and the second position is less than a predetermined value.

9. The system of claim 1, wherein the processor is configured to compare the detected sound with the registered voice command using a voice recognition algorithm.

10. The system of claim 1, wherein the surgical instrument includes a switch and the surgical instrument carries out the operation related to the voice command when the switch is actuated.

11. The system of claim 1, wherein the processor is configured to send the command signal based on at least two of the verification of the designated user'"'"'s credentials, the first position, and the second position.

12. A robotic surgical system, comprising:

a robotic arm;

a tool assembly removably coupled to the robotic arm and comprising a shaft extending distally from a housing and an end effector coupled to a distal end of the shaft, the end effector being configured to treat tissue;

a plurality of voice sensors located in a surgical environment housing the robotic arm and configured to detect sound and generate a first plurality of signals;

a position indicator, located in the surgical environment in proximity to a designated user, configured to indicate a first position of the designated user in the surgical environment and generate a second signal representative of the first position; and

a processor configured to;

receive the first plurality of signals and the second signal,

determine, based on the first plurality of signals, a second position within the surgical environment,

compare the detected sound with a registered voice command of the designated user stored in a memory to verify the designated user'"'"'s credentials, and

send a command signal to the tool assembly to carry out an operation related to the voice command based on at least one of the verification of the designated user'"'"'s credentials, the first position, and the second position.

13. The robotic surgical system of claim 12, wherein the second position is determined based on one or more delays in detection of the sound by one or more of the plurality of voice sensors.

14. A method, comprising:

detecting, by a plurality of voice sensors, sound produced in a surgical environment;

indicating, by a position sensor, a first position associated with a designated user within the surgical environment;

determining, based on the detected sound, a second position within the surgical environment;

comparing the detected sound with registered voice commands stored in a memory to verify the designated user'"'"'s credentials; and

sending a command signal to a surgical instrument to carry out an operation related to the voice command based on at least one of the verification of the user'"'"'s credential, the first position, and the second position.

15. The method of claim 14, further comprising registering the designated user'"'"'s voice.

16. The method of claim 15, wherein the registered voice is stored in memory.

17. The method of claim 14, wherein the second position is determined based on one or more delays in detection of the sound by one or more of the plurality of voice sensors.

18. The method of claim 14, wherein the processor sends the command signal when the difference between the first position and the second position is less than a predetermined value.

19. The method of claim 14, wherein the processor sends the command signal based on at least two of the verification of the designated user'"'"'s credentials, the first position, and the second position.