Device and method for assisting laparoscopic surgery—rule based approach

• US 10,092,164 B2
• Filed: 08/21/2012
• Issued: 10/09/2018
• Est. Priority Date: 08/21/2011
• Status: Active Grant

• Current Assignee: Asensus Surgical Europe SARL (Asensus Surgical, Inc.)
• Original Assignee: M.S.T. Medical Surgery Technologies Ltd (Asensus Surgical, Inc.)
• Inventors: Sholev, Mordehai, Pfeffer, Udi, Frimer, Motti
• Primary Examiner(s): Luong, Peter
• Application Number:
  US14/239,897
• Publication Number:
  US 20140163359A1
• Time in Patent Office: 2,240 Days
• CPC Class Codes: A61B 1/00006 of control signals
• CPC Class Codes: A61B 1/00009 of image signals during a use of endoscope
• CPC Class Codes: A61B 1/00011 characterised by signal transmission
• CPC Class Codes: A61B 1/0002 provided with data storages
• CPC Class Codes: A61B 1/00045 Display arrangement
• CPC Class Codes: A61B 1/00149 using articulated arms
• CPC Class Codes: A61B 1/0016 using motor drive units
• CPC Class Codes: A61B 1/04 combined with photographic or television appliances
• CPC Class Codes: A61B 1/0661 Endoscope light sources
• CPC Class Codes: A61B 1/3132 for laparoscopy
• CPC Class Codes: A61B 17/00234 for minimally invasive surgery A61B17/0218, A61B17/0469, A61B17/12013, A61B17/1285, A61B17/29, A61B17/320016 take precedence
• CPC Class Codes: A61B 2034/107 Visualisation of planned trajectories or target regions
• CPC Class Codes: A61B 2034/2051 Electromagnetic tracking systems
• CPC Class Codes: A61B 2034/2065 Tracking using image or pattern recognition
• CPC Class Codes: A61B 2034/301 for introducing or steering flexible instruments inserted into the body, e.g. catheters or endoscopes
• CPC Class Codes: A61B 2090/0811 for the position of a particular part of an instrument with respect to the rest of the instrument, e.g. position of the anvil of a stapling instrument
• CPC Class Codes: A61B 34/20 Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
• CPC Class Codes: A61B 34/25 User interfaces for surgical systems
• CPC Class Codes: A61B 34/30 Surgical robots
• CPC Class Codes: A61B 5/062 using magnetic field
• CPC Class Codes: A61B 5/064 : using markers A61B5/062 takes precedence
• CPC Class Codes: A61B 6/032 : Transmission computed tomography [CT]
• CPC Class Codes: A61B 8/0841 : for locating instruments
• CPC Class Codes: A61B 8/12 : in body cavities or body tracts, e.g. by using catheters catheters per se A61M25/00
• CPC Class Codes: A61B 90/10 : for stereotaxic surgery, e.g. frame-based stereotaxis
• CPC Class Codes: A61B 90/39 : Markers, e.g. radio-opaque or breast lesions markers
• CPC Class Codes: G16H 20/40 : relating to mechanical, radiation or invasive therapies, e.g. surgery, laser therapy, dialysis or acupuncture
• CPC Class Codes: G16H 30/40 : for processing medical images, e.g. editing

Abstract

The present invention provides a surgical tracking system for assisting an operator to perform a laparoscopic surgery of a human body, said surgical tracking system comprising: a. at least one endoscope adapted to acquire real-time images of a surgical environment within said human body; b. a maneuvering subsystem adapted to control the spatial position of said endoscope during said laparoscopic surgery; and, c. a tracking subsystem in communication with said maneuvering subsystem, adapted to control the maneuvering system so as to direct and modify the spatial position of said endoscope to a region of interest.

Claims

1. A surgical tracking system for assisting an operator to perform a laparoscopic surgery of a human body, said surgical tracking system comprising:

at least one endoscope configured to acquire real-time images of a surgical environment within said human body;

an automated maneuvering subsystem configured to control the spatial position of said endoscope during said laparoscopic surgery; and

,

an automated tracking subsystem in communication with said automated maneuvering subsystem, configured to control the automated maneuvering subsystem so as to direct and modify the spatial position of said endoscope to a region of interest;

wherein said automated tracking subsystem comprises a data processor;

said data processor is configured to perform real-time image processing of said surgical environment and to instruct said automated maneuvering subsystem to modify the spatial position of said endoscope according to input received from a maneuvering function f(t);

said maneuvering function f(t) is configured to (a) receive input from at least two instructing functions g_i(t), where i is 1, . . . , n and n≥

2;

where t is time;

i and n are integers, said instructing functions g_i(t) being functions configured to assess said surgical environment of said at least one endoscope and said laparoscopic surgery, and to output data configured to guide said tracking subsystem; and

, to (b) output instructions to said automated maneuvering subsystem based on said input from said at least two instructing functions g_i(t), so as to spatially position said endoscope to said region of interest;

wherein each of said instructing functions g_i(t) is weighted by a weighting function α

_i(t);

further wherein said weighting functions α

_i(t) are time-varying functions, wherein the value of which at said time t is determined by said operator, said instructing functions g_i(t) and any combination thereof, said weighting functions α

_i(t) being configured to arbitrate between said at least two instructing functions g_i(t) to move said automated maneuvering system.

2. The surgical tracking system of claim 1, wherein each of said instructing functions g_i(t) is selected from a group consisting of:

most used tool function, a right tool function, left tool function, field of view function, preferred volume zone function, preferred tool function, no fly zone function, a tool detection function, a movement detection function, an organ detection function, a collision detection function, an operator input function, a prediction function, a past statistical analysis function, proximity function, a tagged tool function, and any combination thereof.

3. The surgical tracking system of claim 2, wherein said tool detection function is configured to detect surgical tools in said surgical environment and to output instruction to said automated tracking subsystem to instruct said automated maneuvering subsystem to direct said endoscope on said detected surgical tools.

4. The surgical tracking system of claim 2, wherein said movement detection function comprises a communicable database comprising the real-time 3D spatial positions of each of the surgical tool in said surgical environment;

and to detect movement of said at least one surgical tool when a change in said 3D spatial positions is received, and to output instructions to said automated tracking subsystem to instruct said automated maneuvering subsystem to direct said endoscope on said moved surgical tool.

5. The surgical tracking system of claim 2, wherein said organ detection function is configured to detect organs in said surgical environment and to output instructions to said automated tracking subsystem to instruct said automated maneuvering subsystem to direct said endoscope on said detected organs.

6. The surgical tracking system of claim 2, wherein said right tool function is configured to detect surgical tool positioned to right of said endoscope and to output instructions to said automated tracking subsystem to instruct said automated maneuvering subsystem to constantly direct said endoscope on said right tool and to track said right tool.

7. The surgical tracking system of claim 2, wherein said left tool function is configured to detect surgical tool positioned to left of said endoscope and to output instructions to said automated tracking subsystem to instruct said automated maneuvering subsystem to constantly direct said endoscope on said left tool and to track said left tool.

8. The surgical tracking system of claim 2, wherein said operator input function comprises a communicable database;

said communicable database is configured to receive an input from said operator of said system;

said input comprising n 3D spatial positions;

n is an integer greater than or equal to 2; and

to output instructions to said automated tracking subsystem to instruct said automated maneuvering subsystem to direct said endoscope to said at least one 3D spatial position received.

9. The surgical tracking system of claim 2, wherein said proximity function is configured to define a predetermined distance between at least two surgical tools;

and to output instructions to said automated tracking subsystem to instruct said automated maneuvering subsystem to direct said endoscope to said two surgical tools if the distance between said two surgical tools is less than said predetermined distance.

10. The surgical tracking system of claim 2, wherein said proximity function is configured to define a predetermined angle between at least three surgical tools;

and to output instructions to said automated tracking subsystem to instruct said automated maneuvering subsystem to direct said endoscope to said three surgical tools if the angle between said two surgical tools is less than or greater than said predetermined angle.

11. The surgical tracking system of claim 2, wherein said preferred volume zone function comprises communicable database comprising n 3D spatial positions;

n is an integer greater than or equal to 2;

said n 3D spatial positions provides said preferred volume zone;

said preferred volume zone function is configured to output instructions to said automated tracking subsystem to instruct said automated maneuvering subsystem to direct said endoscope to said preferred volume zone.

12. The surgical tracking system of claim 2, wherein said preferred tool function comprises a communicable database, said database stores a preferred tool;

said preferred tool function is configured to output instructions to said automated tracking subsystem to instruct said automated maneuvering subsystem to constantly direct said endoscope to said preferred tool, such that said endoscope constantly tracks said preferred tool.

13. The surgical tracking system of claim 2, wherein said collision prevention function is configured to define a predetermined distance between said at least one surgical tool and an anatomical element within said surgical environment;

and to output instructions to said automated tracking subsystem to instruct said automated maneuvering subsystem to direct said endoscope to said surgical tool and said anatomical element within said surgical environment if the distance between said at least one surgical tool and an anatomical element is less than said predetermined distance;

wherein said anatomical element is selected from a group consisting of tissue, organ, another surgical tool and any combination thereof.

14. The surgical tracking system of claim 2, wherein said field of view function comprises a communicable database comprising n 3D spatial positions;

n is an integer greater than or equal to 2;

the combination of all of said n 3D spatial positions provides a predetermined field of view;

said field of view function is configured to output instructions to said automated tracking subsystem to instruct said automated maneuvering subsystem to direct said endoscope to at least one 3D spatial position substantially within said n 3D spatial positions so as to maintain a constant field of view.

15. The surgical tracking system of claim 2, wherein said no fly zone function comprises a communicable database comprising n 3D spatial positions;

n is an integer greater than or equal to 2;

said n 3D spatial positions define a predetermined volume within said surgical environment;

said no fly zone function is configured to output instructions to said automated tracking subsystem to instruct said automated maneuvering subsystem to direct said endoscope to at least one 3D spatial position substantially different from all said n 3D spatial positions.

16. The surgical tracking system of claim 2, wherein said most used tool function comprises a communicable database counting the amount of movement of each surgical tool located within said surgical environment;

said most used tool function is configured to output instructions to said automated tracking subsystem to instruct said automated maneuvering subsystem to direct said endoscope to constantly position said endoscope to track the movement of the most moved surgical tool.

17. The surgical tracking system of claim 2, wherein said prediction function comprises a communicable database storing each 3D spatial position of each of surgical tool within said surgical environment, such that each movement of each surgical tool is stored;

said prediction function is configured to (a) to predict the future 3D spatial position of each of said surgical tools; and

, (b) to output instructions to said automated tracking subsystem to instruct said automated maneuvering subsystem to direct said endoscope to said future 3D spatial position.

18. The surgical tracking system of claim 2, wherein said past statistical analysis function comprises a communicable database storing each 3D spatial position of each of surgical tool within said surgical environment, such that each movement of each surgical tool is stored;

said past statistical analysis function is configured to (a) statistical analyze said 3D spatial positions of each of said surgical tools; and

, (b) to predict the future 3D spatial position of each of said surgical tools; and

, (c) to output instructions to said automated tracking subsystem to instruct said automated maneuvering subsystem to direct said endoscope to said future 3D spatial position.

19. The surgical tracking system of claim 2, wherein said tagged tool function is configured to tag at least one surgical tool within said surgical environment and to output instructions to said automated tracking subsystem to instruct said automated maneuvering subsystem to constantly direct said endoscope to said tagged surgical tool;

a constant tag being maintained on said tagged surgical tool.

20. The surgical tracking system of claim 2, wherein controller'"'"'s database comprises n 3D spatial positions;

n is an integer greater than or equal to 2;

the combination of all of said n 3D spatial positions provides a predetermined field of view;

said field of view function is configured to relocate said endoscope if movement of at least one of said surgical tools has been detected, such that said field of view is maintained.

21. The surgical tracking system of claim 1, wherein at least one of the following is being held true (a) said image processing is obtained by at least one algorithm selected from a group consisting of:

image stabilization algorithm, image improvement algorithm, image compilation algorithm, image enhancement algorithm, image detection algorithm, image classification algorithm, image correlation with the cardiac cycle or the respiratory cycle of said human body, smoke or vapor, steam reduction from said endoscope and any combination thereof;

(b) said endoscope comprises an image acquisition device selected from a group consisting of;

a camera, a video camera, an electromagnetic sensor, a computer tomography imaging device, a fluoroscopic imaging device, an ultrasound imaging device, and any combination thereof;

(c) said surgical tracking system further comprising a display configured to provide input or output to said operator regarding the operation of said system;

said display is used for visualizing said region of interest by said operator and further said display is configured to output said acquired real-time images of a surgical environment with augmented reality elements;

(d) said image processing algorithm is configured to analyze 2D or 3D representation rendered from said real-time images of the surgical environment;

(e) said data processor is further configured to operate a pattern recognition algorithm for assisting the operation of said instructing functions g_i(t);

(f) said surgical tracking system additionally comprising at least one location estimator for locating the position of at least one surgical tool in said surgical environment; and

any combination thereof.

22. The surgical tracking system of claim 21, wherein said at least one location estimator is an interface subsystem between said operator and the at least one surgical tool, the interface subsystem comprises:

at least one array comprising N regular or pattern light sources, where N is a positive integer;

at least one array comprising M cameras, each of the M cameras, where M is a positive integer;

optional optical markers attachable to the at least one surgical tool; and

,

a computerized algorithm operable via the controller, the computerized algorithm configured to record images received by each camera of each of the M cameras and to calculate therefrom the position of each of the tools, and further configured to provide automatically the results of the calculation to an interface operator.

23. A method for assisting an operator to perform a laparoscopic surgery of a human body, method comprising steps of:

providing a surgical tracking system, comprising;

(i) at least one endoscope configured to acquire real-time images of a surgical environment within said human body;

(ii) an automated maneuvering subsystem in communication with said endoscope; and

, (iii) an automated tracking subsystem in communication with said automated maneuvering subsystem, said automated tracking subsystem comprises a data processor;

performing real-time image processing of said surgical environment; and

controlling said automated maneuvering subsystem via said automated tracking subsystem, thereby directing and modifying the spatial position of said endoscope to a region of interest according to input received from a maneuvering function f(t);

wherein said maneuvering function f(t) is configured to (a) receive input from at least two instructing functions g_i(t), where i is 1, . . . , n and n≥

2;

where t is time;

i and n are integers, said instructing functions g_i(t) being functions configured to assess said surgical environment of said at least one endoscope and said laparoscopic surgery, and to output data configured to guide said tracking subsystem; and

, to (b) output instructions to said automated maneuvering subsystem based on said input from said at least two instructing functions g_i(t), so as to spatially position said endoscope to said region of interest;

wherein each of said instructing functions g_i(t) is weighted by a weighting function α

_i(t);

further wherein said weighting functions α

_i(t) are time-varying functions, wherein the value of which at said time t is determined by said operators, said instructing functions g_i(t) and any combination thereof, said weighting functions α

_i(t) being configured to arbitrate between said at least two instructing functions g_i(t) to move said automated maneuvering system.

24. The method of claim 23, wherein each of said instructing functions g_i(t) is selected from a group consisting of:

most used tool function, right tool function, left tool function, field of view function, preferred volume zone function, preferred tool function, no fly zone function, tool detection function, movement detection function, organ detection function, collision detection function, operator input function, prediction function, past statistical analysis function, proximity function, tagged tool function, and any combination thereof.

25. The method of claim 24, wherein said tool detection function is configured to detect surgical tools in said surgical environment and to output instructions to said automated tracking subsystem to instruct said automated maneuvering subsystem to direct said endoscope on said detected surgical tools.

26. The method of claim 24, wherein said movement detection function comprises a communicable database comprising real-time 3D spatial positions of each said surgical tool in said surgical environment;

and to detect movement of said at least one surgical tool when a change in said 3D spatial positions is received, and to output instructions to said automated tracking subsystem to instruct said automated maneuvering subsystem to direct said endoscope on said moved surgical tool.

27. The method of claim 24, wherein said organ detection function is configured to detect organs in said surgical environment and to output instructions to said automated tracking subsystem to instruct said automated maneuvering subsystem to direct said endoscope on said detected organs.

28. The method of claim 24, wherein said right tool function is configured to detect surgical tool positioned to right of said endoscope and to output instructions to said automated tracking subsystem to instruct said automated maneuvering subsystem to constantly direct said endoscope on said right tool and to track said right tool.

29. The method of claim 24, wherein said left tool function is configured to detect surgical tool positioned to left of said endoscope and to output instructions to said automated tracking subsystem to instruct said automated maneuvering subsystem to constantly direct said endoscope on said left tool and to track said left tool.

30. The method of claim 24, wherein said operator input function comprises a communicable database;

said communicable database is configured to receive an input from said operator of said system;

said input comprising n 3D spatial positions;

n is an integer greater than or equal to 2; and

to output instructions to said automated tracking subsystem to instruct said automated maneuvering subsystem to direct said endoscope to said at least one 3D spatial position received.

31. The method of claim 24, wherein said proximity function is configured to define a predetermined distance between at least two surgical tools;

and to output instructions to said automated tracking subsystem to instruct said automated maneuvering subsystem to direct said endoscope to said two surgical tools if the distance between said two surgical tools is less than said predetermined distance.

32. The method of claim 24, wherein said proximity function is configured to define a predetermined angle between at least three surgical tools;

and to output instructions to said automated tracking subsystem to instruct said automated maneuvering subsystem to direct said endoscope to said three surgical tools if the angle between said two surgical tools is less than or greater than said predetermined angle.

33. The method of claim 24, wherein said preferred volume zone function comprises communicable database comprising n 3D spatial positions;

n is an integer greater than or equal to 2;

said n 3D spatial positions provides said preferred volume zone;

said preferred volume zone function is configured to output instructions to said automated tracking subsystem to instruct said automated maneuvering subsystem to direct said endoscope to said preferred volume zone.

34. The method of claim 24, wherein said preferred tool function comprises a communicable database, said database stores a preferred tool;

said preferred tool function is configured to output instructions to said automated tracking subsystem to instruct said automated maneuvering subsystem to constantly direct said endoscope to said preferred tool, such that said endoscope constantly tracks said preferred tool.

35. The method of claim 24, wherein said collision prevention function is configured to define a predetermined distance between said at least one surgical tool and an anatomical element within said surgical environment;

and to output instructions to said automated tracking subsystem to instruct said automated maneuvering subsystem to direct said endoscope to said surgical tool and said anatomical element within said surgical environment if the distance between said at least one surgical tool and an anatomical element is less than said predetermined distance;

wherein said anatomical element is selected from a group consisting of tissue, organ, another surgical tool and any combination thereof.

36. The method of claim 24, wherein said field of view function comprises a communicable database comprising n 3D spatial positions;

n is an integer greater than or equal to 2;

the combination of all of said n 3D spatial positions provides a predetermined field of view;

said field of view function is configured to output instructions to said automated tracking subsystem to instruct said automated maneuvering subsystem to direct said endoscope to at least one 3D spatial position substantially within said n 3D spatial positions so as to maintain a constant field of view.

37. The method of claim 24, wherein said no fly zone function comprises a communicable database comprising n 3D spatial positions;

n is an integer greater than or equal to 2;

said n 3D spatial positions define a predetermined volume within said surgical environment;

said no fly zone function is configured to output instructions to said automated tracking subsystem to instruct said automated maneuvering subsystem to direct said endoscope to at least one 3D spatial position substantially different from all said n 3D spatial positions.

38. The method of claim 24, wherein said most used tool function comprises a communicable database counting the amount of movement of each surgical tool located within said surgical environment;

said most used tool function is configured to output instructions to said automated tracking subsystem to instruct said automated maneuvering subsystem to direct said endoscope to constantly position said endoscope to track the movement of the most moved surgical tool.

39. The method of claim 24, wherein said prediction function comprises a communicable database storing each 3D spatial position of each of surgical tool within said surgical environment, such that each movement of each surgical tool is stored;

said prediction function is configured to (a) to predict the future 3D spatial position of each of said surgical tools; and

, (b) to output instructions to said automated tracking subsystem to instruct said automated maneuvering subsystem to direct said endoscope to said future 3D spatial position.

40. The method of claim 24, wherein past statistical analysis function comprises a communicable database storing each 3D spatial position of each of surgical tool within said surgical environment, such that each movement of each surgical tool is stored;

said past statistical analysis function is configured to (a) statistical analyze said 3D spatial positions of each of said surgical tools; and

, (b) to predict the future 3D spatial position of each of said surgical tools; and

, (c) to output instructions to said automated tracking subsystem to instruct said automated maneuvering subsystem to direct said endoscope to said future 3D spatial position.

41. The method of claim 24, wherein said tagged tool function is configured to tag at least one surgical tool within said surgical environment and to output instructions to said automated tracking subsystem to instruct said automated maneuvering subsystem to constantly direct said endoscope to said tagged surgical tool;

a constant tag being maintained on said tagged surgical tool.

42. The method of claim 24, wherein at least one of the following is being held true (a) said image processing is obtained by at least one algorithm selected from a group consisting of:

image stabilization algorithm, image improvement algorithm, image compilation algorithm, image enhancement algorithm, image detection algorithm, image classification algorithm, image correlation with the cardiac cycle or the respiratory cycle of said human body, smoke or vapor, steam reduction from said endoscope and any combination thereof;

(b) said endoscope comprises an image acquisition device selected from a group consisting of;

a camera, a video camera, an electromagnetic sensor, a computer tomography imaging device, a fluoroscopic imaging device, an ultrasound imaging device, and any combination thereof;

(c) said method further comprising step of providing a display configured to provide input or output to said operator regarding the operation of said system;

said display is used for visualizing said region of interest by said operator;

(d) said image processing algorithm is configured to analyze 2D or 3D representation rendered from said real-time images of the surgical environment;

(e) said data processor is further configured to operate a pattern recognition algorithm for assisting the operation of said instructing functions g_i(t); and

any combination thereof.

43. The method of claim 23, additionally comprising at least one step selected from a group consisting of (a) preliminary tagging at least one of said surgical tools;

(b) toggling said surgical tools;

(c) re-tagging said at least one of said surgical tools until a desired tool is selected; and

any combination thereof.

44. The method of claim 23, additionally comprising step of locating the 3D position of at least one surgical tool in said surgical environment, wherein said step of locating the 3D position of at least one surgical tool is provided by at least one location estimator;

said at least one location estimator is an interface subsystem between a surgeon and the at least one surgical tool, the interface subsystem comprises;

at least one array comprising N regular or pattern light sources, where N is a positive integer;

at least one array comprising M cameras, each of the M cameras, where M is a positive integer;

optional optical markers attachable to the at least one surgical tool; and

,

a computerized algorithm operable via the controller, the computerized algorithm configured to record images received by each camera of each of the M cameras and to calculate therefrom the position of each of the tools, and further configured to provide automatically the results of the calculation to an interface operator.