Medical robotic system providing computer generated auxiliary views of a camera instrument for controlling the positioning and orienting of its tip

• US 8,864,652 B2
• Filed: 12/17/2008
• Issued: 10/21/2014
• Est. Priority Date: 06/27/2008
• Status: Active Grant

• Current Assignee: Intuitive Surgical Operations, Inc. (Intuitive Surgical Holdings, LLC)
• Original Assignee: Intuitive Surgical Operations, Inc. (Intuitive Surgical Holdings, LLC)
• Inventors: Diolaiti, Nicola, Larkin, David Q., Gomez, Daniel, Mohr, Paul W., Lilagan, Paul, Mustafa, Tabish
• Primary Examiner(s): Nguyen, Anhtuan T
• Assistant Examiner(s): CHOU, WILLIAM B
• Application Number:
  US12/336,713
• Publication Number:
  US 20090326556A1
• Time in Patent Office: 2,134 Days
• Field of Search: 600/103, 600/104, 600/117, 600/145, 600/173, 600/106, 600/109, 600/160
• US Class Current: 600/102
• CPC Class Codes: A61B 1/00006 of control signals
• CPC Class Codes: A61B 1/00009 of image signals during a use of endoscope
• CPC Class Codes: A61B 1/00042 for mechanical operation
• CPC Class Codes: A61B 1/00045 Display arrangement
• CPC Class Codes: A61B 1/0005 combining images e.g. side-by-side, superimposed or tiled
• CPC Class Codes: A61B 1/00087 Tools A61B17/00234 takes precedence
• CPC Class Codes: A61B 1/00149 using articulated arms
• CPC Class Codes: A61B 1/00154 using guiding arrangements for insertion
• CPC Class Codes: A61B 1/05 characterised by the image sensor, e.g. camera, being in the distal end portion
• CPC Class Codes: A61B 1/313 for introducing through surgical openings, e.g. laparoscopes
• CPC Class Codes: A61B 2017/00119 alarm; indicating an abnormal situation
• CPC Class Codes: A61B 2034/102 Modelling of surgical devices, implants or prosthesis
• CPC Class Codes: A61B 2034/2074 Interface software
• CPC Class Codes: A61B 2034/301 for introducing or steering flexible instruments inserted into the body, e.g. catheters or endoscopes
• CPC Class Codes: A61B 2034/305 Details of wrist mechanisms at distal ends of robotic arms
• CPC Class Codes: A61B 2090/364 Correlation of different images or relation of image positions in respect to the body
• CPC Class Codes: A61B 2090/365 augmented reality, i.e. correlating a live optical image with another image
• CPC Class Codes: A61B 34/20 Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
• CPC Class Codes: A61B 34/30 Surgical robots
• CPC Class Codes: A61B 34/37 Master-slave robots A61B34/35 takes precedence
• CPC Class Codes: A61B 34/74 : Manipulators with manual electric input means
• CPC Class Codes: A61B 34/77 : Manipulators with motion or force scaling
• CPC Class Codes: A61B 5/065 : Determining position of the probe employing exclusively positioning means located on or in the probe, e.g. using position sensors arranged on the probe
• CPC Class Codes: A61B 90/03 : Automatic limiting or abutting means, e.g. for safety
• CPC Class Codes: A61B 90/36 : Image-producing devices or illumination devices not otherwise provided for
• CPC Class Codes: A61B 90/361 : Image-producing devices, e.g. surgical cameras
• CPC Class Codes: G06T 7/70 : Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G09B 23/285 : for injections, endoscopy, bronchoscopy, sigmoidscopy, insertion of contraceptive devices or enemas

Abstract

A medical robotic system includes an entry guide with surgical tools and a camera extending out of its distal end. To supplement the view provided by an image captured by the camera, an auxiliary view including articulatable arms of the surgical tools and/or camera is generated from sensed or otherwise determined information about their positions and orientations and displayed on a display screen from the perspective of a specified viewing point. Intuitive control is provided to an operator with respect to the auxiliary view while the operator controls the positioning and orienting of the camera.

Claims

1. A method for positioning and orienting a distal end of an image capturing device that is controllably movable in response to operator manipulation of an input device by actuating a plurality of joints that is mechanically coupled to a plurality of links which is mechanically coupled to the distal end of the image capturing device, the method comprising:

receiving information of positions of the plurality of joints;

determining a position and orientation of the distal end of the image capturing-device by using the received information of the positions of the plurality of joints;

generating a virtual camera perspective view of a computer model of the distal end of the image capturing device, of at least one of the plurality of links, and of at least one of the plurality of joints by using the determined position and orientation of the distal end of the image capturing device, wherein the virtual camera perspective view is from a perspective of a virtual camera;

displaying the virtual camera perspective view of the computer model on a display screen; and

controlling the positioning and orienting of the distal end of the image capturing device by actuating the plurality of joints in response to operator manipulation of the input device such that movement of the input device results in corresponding scaled movement of the distal end of the image capturing device relative to a control reference frame so that the positioning and orienting of the distal end of the image capturing device appears to an operator who is manipulating the input device while viewing the display screen to be intuitively correct with respect to the displayed virtual camera perspective view of the computer model.

2. The method according to claim 1, wherein the plurality of joints is included in a manipulator used for positioning and orienting the distal end of the image capturing device;

wherein the receiving information of the positions of the plurality of joints comprises;

receiving information from a sensor disposed in proximity to the distal end of the image capturing device; and

wherein the determining the position and orientation of the distal end of the image capture device comprises;

calculating calculated positions of the plurality of joints by applying the information received from the sensor to inverse kinematics of the manipulator, and calculating the position and orientation of the distal end of the image capture device by applying the calculated positions of the plurality of joints to forward kinematics of the manipulator.

3. The method according to claim 1, wherein the plurality of joints is included in a manipulator used for positioning and orienting the distal end of the image capturing device;

wherein the determining the position and orientation of the distal end of the image capture device comprises;

calculating the position and orientation of the distal end of the image capturing device by applying the received information of the positions of the plurality of joints to forward kinematics of the manipulator.

4. The method according to claim 1, wherein the generating of the virtual camera perspective view of the computer model comprises:

generating a three-dimensional computer model of the distal end of the image capturing device, the plurality of joints, and the plurality of links; and

translating and orienting the generated three-dimensional computer model to the virtual camera perspective view.

5. The method according to claim 1, wherein the distal end of the image capture device and at least one articulatable instrument extends out of an entry guide, and the virtual camera perspective view is at a position and orientation from which a distal end of the at least one articulatable instrument, the distal end of the image capturing device, the at least one of the plurality of links, and the at least one of the plurality of joints are within the field of view of the virtual camera.

6. The method according to claim 5, wherein the controlling the positioning and orienting of the distal end of the image capturing device-comprises:

defining the control reference frame such that an origin of the control reference frame is at the position of the virtual camera and such that one axis of the control reference frame points toward a point on a longitudinal axis extending through proximal and distal ends of the entry guide.

7. The method according to claim 5, wherein the controlling the positioning and orienting of the distal end of the image capturing device comprises:

defining the control reference frame such that an origin of the control reference frame is at a point where the distal end of the image capturing device extends out of the entry guide and such that one axis of the control reference frame is parallel with a central longitudinal axis of the entry guide.

8. A medical system comprising:

an image capturing device having a distal end;

a plurality of links that is mechanically coupled to the distal end of the image capturing device for positioning and orienting the distal end of the image capturing device;

a plurality of joints that is mechanically coupled to the plurality of links;

a display screen;

an input device; and

a controller configured;

to receive information of positions of the plurality of joints;

to determine a position and orientation of the distal end of the image capturing device by using the received information of the positions of the plurality of joints;

to generate a virtual camera perspective view of a computer model of the distal end of the image capturing device, of at least one of the plurality of links, and of at least one of the plurality of joints by using the determined position and orientation of the distal end of the image capturing device, wherein the virtual camera perspective view is from a perspective of a virtual camera;

to display the virtual camera perspective view of the computer model on the display screen; and

to control the positioning and orienting of the distal end of the image capturing device by actuating the plurality of joints in response to manipulation of the input device such that movement of the input device results in corresponding scaled movement of the distal end of the image capturing device relative to a control reference frame so that the positioning and orienting of the distal end of the image capturing device appears to an operator who is manipulating the input device while viewing the display screen to be intuitively correct with respect to the displayed virtual camera perspective view of the computer model.

9. The medical system according claim 8, wherein the controller is configured to determine the positions of the plurality of joints by using information provided by sensors that sense the positions of the plurality of joints.

10. The medical system according claim 8, further comprising:

actuators used to actuate the plurality of joints, wherein the controller is configured to determine the positions of the plurality of joints by using information provided by encoders that sense actuation of the actuators.

11. The medical system according claim 8, wherein the plurality of joints is included in a manipulator used for positioning and orienting the distal end of the image capturing device, and the controller is configured to determine the positions of the joints by receiving information from a sensor disposed in proximity to the distal end of the image capturing device and by calculating the positions of the plurality of joints by applying the information received from the sensor to inverse kinematics of the manipulator.

12. The medical system according claim 8, wherein the plurality of joints is included in a manipulator used for positioning and orienting the distal end of the image capturing device, and the controller is configured to calculate the position and orientation of the distal end of the image capturing device by applying the received information of the positions of the plurality of joints to forward kinematics of the manipulator.

13. The medical system according claim 8, wherein the controller is configured to generate the virtual camera perspective view of the computer model by generating a three-dimensional computer model of the distal end of the image capturing device, the plurality of joints, and the plurality of links;

and by translating and orienting the three-dimensional computer model to the virtual camera perspective.

14. The medical system according to claim 8, further comprising:

at least one articulatable instrument, wherein the distal end of the image capturing device and a distal end of the at least one articulatable instrument extends out of an entry guide, and the virtual camera perspective view is at a position and orientation from which the distal end of the at least one articulatable instrument, the distal end of the image capturing device, the at least one of the plurality of links, and the at least one of the plurality of joints are within the field of view of the virtual camera.

15. The medical system according to claim 14, wherein the control reference frame has an origin at the position of the virtual camera and is oriented such that one axis of the control reference frame points to a point on a longitudinal axis extending through proximal and distal ends of the entry guide.

16. The medical system according to claim 14, wherein the control reference frame has an origin where the distal end of the image capturing device extends out of the entry guide and is oriented such that one axis of the control reference frame is parallel with a central longitudinal axis of the entry guide.