Vocally activated surgical control system

• US 10,866,783 B2
• Filed: 04/07/2017
• Issued: 12/15/2020
• Est. Priority Date: 08/21/2011
• Status: Active Grant

• Current Assignee: Asensus Surgical Europe SARL (Asensus Surgical, Inc.)
• Original Assignee: Transenterix Europe, Sarl (Asensus Surgical, Inc.)
• Inventors: Atarot, Gal, Frimer, Motti, Nir, Tal, Alpert, Lior
• Primary Examiner(s): Tran, Khoi H
• Assistant Examiner(s): Peche, Jorge O
• Application Number:
  US15/481,496
• Publication Number:
  US 20170212723A1
• Time in Patent Office: 1,348 Days
• CPC Class Codes: A41D 13/11 Protective face masks, e.g. for surgical use, or for use in foul atmospheres eye-masks A61F9/04 ; hoods affording protection against heat or harmful chemical agents A62B17/04
• CPC Class Codes: A61B 2017/00017 Electrical control of surgical instruments
• CPC Class Codes: A61B 2017/00119 alarm; indicating an abnormal situation
• CPC Class Codes: A61B 2017/00132 Setting operation time of a device A61B2017/00123 takes precedence
• CPC Class Codes: A61B 2017/00203 with speech control or speech recognition
• CPC Class Codes: A61B 2017/00221 with wireless transmission of data, e.g. by infrared radiation or radiowaves
• CPC Class Codes: A61B 2017/00225 Systems for controlling multiple different instruments, e.g. microsurgical systems
• CPC Class Codes: A61B 2034/2048 using an accelerometer or inertia sensor
• CPC Class Codes: A61B 2034/2065 Tracking using image or pattern recognition
• CPC Class Codes: A61B 34/30 Surgical robots
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/28 Constructional details of speech recognition systems

Abstract

The following invention is a vocally activated control system for controlling an apparatus in a surgical setting, the system comprises:

• • a. a voice sensor configured to detect vocal commands generated by surgeons during surgery;
  • b. a signal transmitter connected to the voice sensor, the transmitter is configured to convert a vocal command into a transmittable signal and transmit it;
  • c. a processor connected to a signal transmitter configured to receive a transmittable vocal signal, the processor is configured to convert a vocal signal to a predetermined set of operative instructions associated with the apparatus, the predetermined set of operative instructions comprising at least one instruction; and
  • d. control means connected to the processor and apparatus; the control means is configured to receive a predetermined set of operative instructions and to cause the apparatus to operate accordingly;

Said voice sensor and said transmitter are integrated within a wearable element.

Claims

1. A vocally activated control system for controlling at least one apparatus in a surgical setting, said vocally activated control system comprises:

a. a voice sensor configured to detect at least one vocal command generated by at least one surgeon in said surgical setting;

b. a signal transmitter operatively connected to said voice sensor, said transmitter is configured to convert said at least one vocal command into at least one transmittable vocal signal and transmit said at least one transmittable vocal signal;

c. a processor operatively connected to said signal transmitter configured to receive said at least one transmittable vocal signal, said processor is configured to convert said at least one transmittable vocal signal to at least one predetermined set of operative instructions associated with said at least one apparatus, said at least one predetermined set of operative instructions comprising at least one instruction; and

d. at least one control means operatively connected to said processor and said at least one apparatus;

said at least one control means is configured to receive said predetermined set of operative instructions and to cause said at least one apparatus to operate accordingly;

wherein said voice sensor and said transmitter are integrated within a surgical wearable element selected from a group consisting of a mask, a helmet, a cap, a shoe cover, a glove, a hospital robe, surgical garb, a gown, a wristlet or an armlet

wherein said at least one apparatus is a robotically controlled surgical tool, wherein said conversion of said transmittable vocal signal to said at least one predetermined set of operative instructions is performed to generate instructions according to a predetermined set of rules, said at least one predetermined set of rules comprising at least one rule selected from a group consisting of;

most used tool rule, right tool rule, left tool rule, field of view rule, no fly zone rule, route rule, environmental rule, operator input rule, proximity rule;

collision prevention rule, history-based rule, tool-dependent allowed and restricted movement rule, preferred volume zone rule, preferred tool rule, movement detection rule, tagged tool rule, go-to rule, change of speed rule and any combination thereof;

said allowed movement being movements of said surgical tool permitted by said controller and said restricted movement being movements of said surgical tool denied by said controller according to said predetermined set of rules.

2. The vocally activated control system according to claim 1, wherein at least one response to at least one said at least one vocal command is a smart response, said smart response being configured to take into account at least one feature of the an environment surrounding said robotically controlled surgical tool, said feature selected from a group consisting of a hazard or obstacle, an interference with a second surgical tool, an interference between two parts of an apparatus, interference between two apparatus, possibility of damage to a portion of a patient, and any combination thereof.

3. The vocally activated control system according to claim 1, wherein said at least one vocal command comprises at least one complex sentence;

said voice sensor further comprising a context recognition unit configured to recognize said at least one vocal command from said complex sentence.

4. The vocally activated control system according to claim 1, wherein said vocal command can be unqualified or can comprise at least one qualifier, said at least one qualifier configured to modify at least one component of a response, and wherein at least one of the following:

said qualifier is selected from a group consisting of an amount, a surgical tool, an apparatus, and any combination thereof; and

said amount is either a fixed-term value or a fractional value.

5. The vocally activated control system according to claim 4, wherein said unqualified command is configured to comprise a predetermined qualifier or to continue a process until stopped.

6. The vocally activated control system according to claim 5, wherein at least one of the following:

a. said predetermined qualifier is an amount, said amount being either a fixed-term value or a fractional value;

b. said vocal command is configured to reversibly select a member of a group consisting of an apparatus, a surgical tool, at least a portion of a patient, and any combination thereof;

a duration of said selection is selected from a group consisting of;

for a predetermined time, or until reception of a change command;

c. said vocal command is configured to change a member of a group consisting of;

a predetermined amount;

a value, a type of value and any combination thereof.

7. The vocally activated control system according to claim 1, wherein at least one of the following:

a. at least one said apparatus comprises a maneuvering subsystem configured to spatially reposition said at least one surgical tool according to said predetermined set of rules;

b. said route rule comprises a communicable database storing predefined route in which said at least one surgical tool is configured to move within said surgical environment;

said predefined route comprises n 3D spatial positions of said at least one surgical tool;

n is an integer greater than or equal to 2;

said allowed movement is a movement in which said at least one surgical tool is located substantially in at least one of said n 3D spatial positions of said predefined route, and said restricted movement is a movement in which said location of said at least one surgical tool is substantially different from said n 3D spatial positions of said predefined route;

c. said environmental rule comprises a comprises a communicable database;

said communicable database configured to receive at least one real-time image of said surgical environment and is configured to perform real-time image processing of the same and to determine the 3D spatial position of at least one hazard or obstacle in said surgical environment;

said environmental rule is configured to determine said allowed movement an said restricted movement according to said hazards or obstacles in said surgical environment, such that said restricted movement is a movement in which said at least one surgical tool is located substantially in at least one of said 3D spatial positions, and said allowed movement is a movement in which the location of said at least one surgical tool is substantially different from said 3D spatial positions;

said at least one hazard or obstacle being selected from a group consisting of tissue, a surgical tool, an organ, an endoscope and any combination thereof;

d. said proximity rule is configured to define at least one of a predetermined distance between at least two surgical tools and a predetermined angle between at least two surgical tools;

for said predetermined distance, said allowed movement is a movement which is within the range or out of the range of said predetermined distance and said restricted movement is a movement which is out of the range or within the range of said predetermined distance;

for said predetermined angle;

said allowed movement is a movement which is within the range or out of the range of said predetermined angle and said restricted movement is a movement which is out of the range or within the range of said predetermined angle;

e. said collision prevention rule is configured to define a predetermined distance between said at least one surgical tool and an anatomical element within said surgical environment;

said allowed movement is a movement which is in a range that is larger than said predetermined distance, and said restricted movement is a movement which is in a range that is smaller than said predetermined distance;

said anatomical element being selected from a group consisting of tissue, organ, another surgical tool and any combination thereof;

f. said no fly zone rule comprises a communicable database comprising n 3D spatial positions;

n is an integer greater than or equal to 2;

said n 3D spatial positions define a predetermined volume within said surgical environment;

said no fly zone rule is configured to determine said restricted movement if said movement is within said no fly zone and allowed movement if said movement is outside said no fly zone, such that said restricted movement is a movement in which said at least one of said surgical tool is located substantially in at least one of said n 3D spatial positions, and said allowed movement is a movement in which the location of said at least one surgical tool is substantially different from said n 3D spatial positions;

g. said preferred volume zone rule comprises a communicable database comprising n 3D spatial positions;

n is an integer greater than or equal to 2;

said n 3D spatial positions provides said preferred volume zone;

said preferred volume zone rule is configured to determine said allowed movement of said surgical tool within said n 3D spatial positions and restricted movement of said surgical tool outside said n 3D spatial positions, such that said allowed movement is a movement in which at least a portion of said surgical tool is located substantially in at least one of said n 3D spatial positions, and said restricted movement is a movement in which the location of said surgical tool is substantially different from said n 3D spatial positions;

h. said history-based rule comprises a communicable database storing each 3D spatial position of each of said surgical tool, such that each movement of each surgical tool is stored;

said history-based rule is configured to determine said allowed movement and said restricted movement according to historical movements of said at least one surgical tool, such that said allowed movement is a movement in which said at least one surgical tool is located substantially in at least one of said 3D spatial positions, and said restricted movement is a movement in which the location of said at least one surgical tool is substantially different from said n 3D spatial positions; and

i. said maneuvering subsystem is configured to spatially reposition said at least one surgical tool during a surgery according to said predetermined set of rules, such that if said movement of said at least one surgical tool is a restricted movement, said maneuvering subsystem prevents said movement.

8. The vocally activated control system according to claim 1, wherein said system is configured to provide an alert of said restricted movement of said at least one surgical tool.

9. The vocally activated control system according to claim 1, wherein said operator input rule comprises a communicable database;

said communicable database is configured to receive an input from the operator of said system regarding said allowed movement and said restricted movement of said at least one surgical tool;

at least one of the following;

a. said input comprises n 3D spatial positions;

n is an integer greater than or equal to 2;

wherein at least one of which is defined as allowed location and at least one of which is defined as restricted location, such that said allowed movement is a movement in which said at least one surgical tool is located substantially in at least one of said n 3D spatial positions, and said restricted movement is a movement in which the location of said at least one surgical tool is substantially different from said n 3D spatial positions;

b. said input comprises at least one rule according to which said allowed movement and said restricted movement of said at least one surgical tool are determinable, such that the spatial position of said at least one surgical tool is controlled by said controller according to said allowed movement and said restricted movement;

said predetermined set of rules comprising at least one rule selected from a group consisting of;

most used tool, right tool rule, left tool rule, field of view rule, no fly zone rule, route rule, environmental rule, operator input rule, proximity rule;

collision prevention rule, preferred volume zone rule, preferred tool rule, movement detection rule, history-based rule, tool-dependent allowed and restricted movement rule, and any combination thereof; and

c. said operator input rule converts an allowed movement to a restricted movement and a restricted movement to an allowed movement.

10. The vocally activated control system according to claim 1, wherein said allowed movement is permitted by said controller and said restricted movement is denied by said controller.

11. The vocally activated control system according to claim 1, wherein at least one of the following:

(a) said system additionally comprises an endoscope;

said endoscope is configured to provide real-time image of said surgical environment;

(b) at least one of said surgical tools is an endoscope configured to provide real-time image of said surgical environment.

12. The vocally activated control system according to claim 11, wherein at least one of the following:

a. said most used tool rule comprises a communicable database counting the amount of movement of each of said surgical tools;

said most used tool rule is configured to constantly position said endoscope to track the movement of the most moved surgical tool;

said alert being selected from a group consisting of audio signaling, voice signaling, light signaling, flashing signaling and any combination thereof;

b. said right tool rule is configured to determine said allowed movement of said endoscope according to the movement of the surgical tool positioned to right of said endoscope;

further wherein said left tool rule is configured to determine said allowed movement of said endoscope according to the movement of the surgical tool positioned to left of said endoscope;

c. said tagged tool rule comprises means configured to tag at least one surgical tool within said surgical environment and to determine said allowed movement of said endoscope to constantly track the movement of said tagged surgical tool;

d. said field of view rule comprises a communicable database comprising n 3D spatial positions;

n is an integer greater than or equal to 2;

the combination of all of said n 3D spatial positions provides a predetermined field of view;

said field of view rule is configured to determine said allowed movement of said endoscope within said n 3D spatial positions so as to maintain a constant field of view, such that said allowed movement is a movement in which said endoscope is located substantially in at least one of said n 3D spatial positions, and said restricted movement is a movement in which the location of said endoscope is substantially different from said n 3D spatial positions;

e. said preferred tool rule comprises a communicable database, said database stores a preferred tool;

said preferred tool rule is configured to determine said allowed movement of said endoscope to constantly track the movement of said preferred tool;

f. said tool-dependent allowed and restricted movement rule comprises a communicable database;

said communicable database is configured to store predetermined characteristics of at least one of said at least one surgical tool;

said tool-dependent allowed and restricted movement rule is configured to determine said allowed movement and said restricted movement according to said predetermined characteristics;

such that said allowed movement is a movement of said endoscope which tracks said at least one of said at least one surgical tool having said predetermined characteristics;

said predetermined characteristics of said surgical tool being selected from a group consisting of;

physical dimensions, structure, weight, sharpness, and any combination thereof; and

g. said movement detection rule comprises a communicable database comprising the real-time 3D spatial positions of each of said surgical tool;

said movement detection rule is configured to detect movement of said at least one surgical tool when a change in said 3D spatial positions is received, such that said allowed movement is a movement in which said endoscope is re-directed to focus on the moving surgical tool.

13. The vocally activated control system according to claim 1, additionally comprising at least one location estimator, wherein said at least one location estimator comprises at least one endoscope configured to acquire real-time images of said surgical environment within said human body;

and at least one surgical instrument spatial location software configured to receive said real-time images of said surgical environment and to estimate a 3D spatial position of said at least one surgical tool.

14. The vocally activated control system of claim 13, wherein at least one of the following:

a. said at least one location estimator comprises (a) at least one element selected from a group consisting of optical imaging means, radio frequency transmitting and receiving means, at least one mark on said at least one surgical tool and any combination thereof; and

, (b) at least one surgical instrument spatial location software configured to estimate said 3D spatial position of said at least one surgical tool by means of said element;

b. said at least one location estimator is an interface subsystem between a surgeon and the at least one surgical tool, the interface subsystem comprises;

i. at least one array comprising N regular or pattern light sources, where N is a positive integer;

ii. at least one array comprising M cameras, each of the M cameras, where M is a positive integer;

iii. optional optical markers and means for attaching the optical marker to the at least one surgical tool; and

;

iv. a computerized algorithm operable via the controller, the computerized algorithm configured to record images received by each camera of each of the M cameras and to calculate therefrom the position of each of the tools, and further configured to provide automatically the results of the calculation to the human operator of the interface.