MEDICAL ROBOTIC SYSTEM PROVIDING COMPUTER GENERATED AUXILIARY VIEWS OF A CAMERA INSTRUMENT FOR CONTROLLING THE POSITIONING AND ORIENTING OF ITS TIP

• US 20090326556A1
• Filed: 12/17/2008
• Published: 12/31/2009
• Est. Priority Date: 06/27/2008
• Status: Active Grant

• Current Assignee: Intuitive Surgical Operations, Inc. (Intuitive Surgical Holdings, LLC)
• Original Assignee: Intuitive Surgical Incorporated (Intuitive Surgical Holdings, LLC)
• Inventors: Gomez, Daniel, Larkin, David Q., Mohr, Paul W., Lilagan, Paul, Diolaiti, Nicola, Mustufa, Tabish
• Granted Patent: US 8,864,652 B2
• Time in Patent Office: Days
• US Class Current: 606/130
• CPC Class Codes: A61B 1/00006 of control signals
• CPC Class Codes: A61B 1/00009 of image signals during a use of endoscope
• CPC Class Codes: A61B 1/00042 for mechanical operation
• CPC Class Codes: A61B 1/00045 Display arrangement
• CPC Class Codes: A61B 1/0005 combining images e.g. side-by-side, superimposed or tiled
• CPC Class Codes: A61B 1/00087 Tools A61B17/00234 takes precedence
• CPC Class Codes: A61B 1/00149 using articulated arms
• CPC Class Codes: A61B 1/00154 using guiding arrangements for insertion
• CPC Class Codes: A61B 1/05 characterised by the image sensor, e.g. camera, being in the distal end portion
• CPC Class Codes: A61B 1/313 for introducing through surgical openings, e.g. laparoscopes
• CPC Class Codes: A61B 2017/00119 alarm; indicating an abnormal situation
• CPC Class Codes: A61B 2034/102 Modelling of surgical devices, implants or prosthesis
• CPC Class Codes: A61B 2034/2074 Interface software
• CPC Class Codes: A61B 2034/301 for introducing or steering flexible instruments inserted into the body, e.g. catheters or endoscopes
• CPC Class Codes: A61B 2034/305 Details of wrist mechanisms at distal ends of robotic arms
• CPC Class Codes: A61B 2090/364 Correlation of different images or relation of image positions in respect to the body
• CPC Class Codes: A61B 2090/365 augmented reality, i.e. correlating a live optical image with another image
• CPC Class Codes: A61B 34/20 Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
• CPC Class Codes: A61B 34/30 Surgical robots
• CPC Class Codes: A61B 34/37 Master-slave robots A61B34/35 takes precedence
• CPC Class Codes: A61B 34/74 : Manipulators with manual electric input means
• CPC Class Codes: A61B 34/77 : Manipulators with motion or force scaling
• CPC Class Codes: A61B 5/065 : Determining position of the probe employing exclusively positioning means located on or in the probe, e.g. using position sensors arranged on the probe
• CPC Class Codes: A61B 90/03 : Automatic limiting or abutting means, e.g. for safety
• CPC Class Codes: A61B 90/36 : Image-producing devices or illumination devices not otherwise provided for
• CPC Class Codes: A61B 90/361 : Image-producing devices, e.g. surgical cameras
• CPC Class Codes: G06T 7/70 : Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G09B 23/285 : for injections, endoscopy, bronchoscopy, sigmoidscopy, insertion of contraceptive devices or enemas

Abstract

A medical robotic system includes an entry guide with surgical tools and a camera extending out of its distal end. To supplement the view provided by an image captured by the camera, an auxiliary view including articulatable arms of the surgical tools and/or camera is generated from sensed or otherwise determined information about their positions and orientations and displayed on a display screen from the perspective of a specified viewing point. Intuitive control is provided to an operator with respect to the auxiliary view while the operator controls the positioning and orienting of the camera.

Claims

1. A method for positioning and orienting a camera tip, the method comprising:

determining positions of mechanical elements used for positioning and orienting the camera tip;

determining a position and orientation of the camera tip using the determined positions of the mechanical elements;

generating a view of a computer model of the camera corresponding to a perspective of a virtual camera;

displaying the view on a display screen; and

controlling the positioning and orienting of the camera tip by moving the mechanical elements in response to manipulation of an input device so that the positioning and orienting of the camera tip intuitively appears to an operator who is manipulating the input device while viewing the display screen to correspond to the displayed view of the computer model of the camera.

2. The method according claim 1, wherein the determination of the positions of the mechanical elements comprises:

determining the positions of the mechanical elements using information provided by sensors that sense the positions.

3. The method according to claim 2, wherein the mechanical elements include joints of a manipulator used to position and orient the camera tip.

4. The method according to claim 2, wherein the mechanical elements include links of a manipulator used to position and orient the camera tip.

5. The method according claim 1, wherein the determination of the positions of the mechanical elements comprises:

determining the positions of the mechanical elements using information provided by encoders that sense actuation of actuators used to position the mechanical elements.

6. The method according to claim 1, wherein the mechanical elements are included in a manipulator used for positioning and orienting the camera tip and the determination of the positions of the mechanical elements comprises:

receiving information from a sensor disposed on the camera tip; and

calculating the positions of the mechanical elements by applying the information received from the sensor to inverse kinematics of the manipulator.

7. The method according to claim 1, wherein the mechanical elements are included in a manipulator used for positioning and orienting the camera tip and the determination of the position and orientation of the camera tip using the determined positions of the mechanical elements comprises:

calculating the position and orientation of the camera tip by applying the determined positions of the mechanical elements to forward kinematics of the manipulator.

8. The method according to claim 1, wherein the generation of the view of the computer model of the camera corresponding to the perspective of the virtual camera comprises:

generating a three-dimensional computer model of the camera; and

translating the computer model to the perspective of the virtual camera.

9. The method according to claim 8, wherein the camera is an articulatable camera instrument extending out an entry guide, the virtual camera has identical characteristics as the camera, and the perspective of the virtual camera is at a position and orientation from which the camera instrument extending out of the entry guide is within a field of view of the virtual camera.

10. The method according to claim 9, wherein at least one articulatable instrument extends out of the entry guide along with the articulatable camera instrument and the perspective of the virtual camera is at a position and orientation from which the at least one articulatable instrument and the camera instrument extending out of the entry guide are at least partially within the field of view of the virtual camera.

11. The method according to claim 1, wherein the camera is an articulatable camera instrument extending out an entry guide and the controlling of the position and orientation of the camera tip comprises:

defining a Cartesian reference frame having an origin at the position of the virtual camera and oriented such that one axis of the reference frame points toward a focal point on a longitudinal axis extending between proximal and distal ends of the entry guide; and

controlling the position and orientation of the camera tip in response to the operator manipulation of the input device such that movement of the input device in a Cartesian reference frame associated with the input device results in corresponding scaled movement of the camera tip relative to the Cartesian reference frame of the virtual camera.

12. The method according to claim 1, wherein the camera is an articulatable camera instrument extending out an entry guide and the controlling of the position and orientation of the camera tip comprises:

defining a Cartesian reference frame having an origin where a central longitudinal axis of the entry guide intersects an orthogonal plane at a distal end of the entry guide where the camera instrument extends out of the entry guide and oriented such that one axis of the reference frame is parallel with the central longitudinal axis; and

controlling the position and orientation of the camera tip in response to the operator manipulation of the input device such that movement of the input device in a Cartesian reference frame associated with the input device results in corresponding scaled movement of the camera tip relative to the Cartesian reference frame defined at the distal end of the entry guide.

13. The method according to claim 1, further comprising prior to controlling the position and orientation of the camera tip:

disengaging control of an instrument associated with the input device so as to hold the instrument in place;

aligning an orientation of the input device relative to an orientation of the camera tip; and

engaging control of the camera tip with the input device.

14. The method according to claim 13, further comprising after controlling the position and orientation of the camera:

disengaging control of the camera tip with the input device so as to hold the camera tip in place;

aligning the orientation of the input device relative to an orientation of a tip of the instrument; and

engaging control of the instrument with the input device.

15. A medical robotic system comprising:

a camera;

mechanical elements used for positioning and orienting a tip of the camera;

a display screen;

an input device; and

a controller configured to determine positions of the mechanical elements, determine a position and orientation of the camera tip using the determined positions of the mechanical elements, generate a view of a computer model of the camera corresponding to a perspective of a virtual camera, display the view on the display screen, and control the positioning and orienting of the camera tip by moving the mechanical elements in response to manipulation of the input device so that the positioning and orienting of the camera tip intuitively appears to an operator who is manipulating the input device while viewing the display screen to correspond to the displayed view of the computer model of the camera.

16. The medical robotic system according claim 15, wherein the controller is configured to determine the positions of the mechanical elements using information provided by sensors that sense the positions.

17. The medical robotic system according claim 15, wherein the mechanical elements include joints of a manipulator used to position and orient the camera tip.

18. The medical robotic system according claim 15, wherein the mechanical elements include links of a manipulator used to position and orient the camera tip.

19. The medical robotic system according claim 15, further comprising:

actuators used to position the mechanical elements, wherein the controller is configured to determine the positions of the mechanical elements using information provided by encoders that sense actuation of the actuators.

20. The medical robotic system according claim 15, wherein the mechanical elements are included in a manipulator used for positioning and orienting the camera tip and the controller is configured to determine the positions of the mechanical elements by receiving information from a sensor disposed on the camera tip and calculating the positions of the mechanical elements by applying the information received from the sensor to inverse kinematics of the manipulator.

21. The medical robotic system according claim 15, wherein the mechanical elements are included in a manipulator used for positioning and orienting the camera tip and the controller is configured to calculate the position and orientation of the camera tip by applying the determined positions of the mechanical elements to forward kinematics of the manipulator.

22. The medical robotic system according claim 15, wherein the controller is configured to generate the view of the computer model of the camera corresponding to the perspective of the virtual camera by generating a three-dimensional computer model of the camera and translating the computer model to the perspective of the virtual camera.

23. The medical robotic system according to claim 22, further comprising:

an entry guide, wherein the camera is an articulatable camera instrument extending out the entry guide, the virtual camera has identical characteristics as the camera, and the perspective of the virtual camera is at a position and orientation from which the camera instrument extending out of the entry guide is within a field of view of the virtual camera.

24. The medical robotic system according to claim 23, further comprising:

at least one articulatable instrument, wherein the at least one articulatable instrument extends out of the entry guide along with the articulatable camera instrument and the perspective of the virtual camera is at a position and orientation from which the at least one articulatable instrument and the camera instrument extending out of the entry guide are at least partially within the field of view of the virtual camera.

25. The medical robotic system according to claim 15, further comprising an entry guide, wherein the camera is an articulatable camera instrument extending out the entry guide and the controller is configured to:

define a Cartesian reference frame having an origin at the position of the virtual camera and oriented such that one axis of the reference frame points to a focal point on a longitudinal axis extending between proximal and distal ends of the entry guide, and

control the position and orientation of the camera tip in response to the operator manipulation of the input device such that movement of the input device in a Cartesian reference frame associated with the input device results in corresponding scaled movement of the camera tip relative to the Cartesian reference frame of the virtual camera.

26. The medical robotic system according to claim 15, further comprising an entry guide, wherein the camera is an articulatable camera instrument extending out the entry guide and the controller is configured to:

define a Cartesian reference frame having an origin where a central longitudinal axis of the entry guide intersects an orthogonal plane at a distal end of the entry guide where the camera instrument extends out of the entry guide and oriented such that one axis of the reference frame is parallel with the central longitudinal axis, and

control the position and orientation of the camera tip in response to the operator manipulation of the input device such that movement of the input device in a Cartesian reference frame associated with the input device results in corresponding scaled movement of the camera tip relative to the Cartesian reference frame defined at the distal end of the entry guide.

27. The medical robotic system according to claim 15, wherein the controller is configured to:

disengage control of an instrument associated with the input device so as to hold the instrument in place,

align an orientation of the input device relative to an orientation of the camera tip, and

engage control of the camera tip with the input device prior to controlling the positioning and orienting of the camera tip.

28. The medical robotic system according to claim 27, wherein the controller is configured to:

disengage control of the camera tip with the input device so as to hold the camera tip in place,

align the orientation of the input device relative to an orientation of a tip of the instrument, and

engage control of the instrument with the input device after controlling the positioning and orienting of the camera tip.