SYSTEM FOR MOVING FIRST INSERTABLE INSTRUMENT AND SECOND INSERTABLE INSTRUMENT, CONTROLLER, AND COMPUTER-READABLE STORAGE DEVICE

• US 20180256008A1
• Filed: 05/10/2018
• Published: 09/13/2018
• Est. Priority Date: 02/10/2016
• Status: Active Grant

• Current Assignee: Olympus Corporation
• Original Assignee: Olympus Corporation
• Inventors: NISHIZAWA, Koji
• Granted Patent: US 10,638,915 B2
• Time in Patent Office: Days
• CPC Class Codes: A61B 1/00006 of control signals
• CPC Class Codes: A61B 1/00149 using articulated arms
• CPC Class Codes: A61B 1/05 characterised by the image sensor, e.g. camera, being in the distal end portion
• CPC Class Codes: A61B 2034/2055 Optical tracking systems
• CPC Class Codes: A61B 2034/2072 Reference field transducer attached to an instrument or patient
• CPC Class Codes: A61B 34/20 Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
• CPC Class Codes: G06T 2207/10068 Endoscopic image
• CPC Class Codes: G06T 2207/30204 Marker
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: H04N 23/555 for picking-up images in sites, inaccessible due to their dimensions or hazardous conditions, e.g. endoscopes or borescopes
• CPC Class Codes: H04N 23/66 Remote control of cameras or camera parts, e.g. by remote control devices

Abstract

A medical system having a first insertable instrument having a first image sensor; a second insertable instrument having a second image sensor; a first arm configured to move the first insertable instrument; and a controller configured to: acquire a position and orientation of the second insertable instrument in a predetermined coordinate system; calculate a position and direction of an optical axis of the second image sensor based on the position and orientation of the second insertable instrument; acquire a position and orientation of the first insertable instrument in the predetermined coordinate system; calculate a first operation amount of the first arm to move the first image sensor such that the optical axis of the second image sensor substantially coincides with an optical axis of the first image sensor, based on the position and orientation of the first insertable instrument; and control the first arm based on the first operation amount.

Claims

1. A medical system comprising:

a first insertable instrument comprising a first image sensor;

a second insertable instrument comprising a second image sensor;

a first arm configured to support and move the first insertable instrument; and

a controller configured to;

acquire a position and orientation of the second insertable instrument in a predetermined coordinate system;

calculate a position and direction of an optical axis of the second image sensor based on the position and orientation of the second insertable instrument;

acquire a position and orientation of the first insertable instrument in the predetermined coordinate system;

calculate a first operation amount of the first arm to move the first image sensor such that the optical axis of the second image sensor substantially coincides with an optical axis of the first image sensor, based on the position and orientation of the first insertable instrument; and

control the first arm, based on the first operation amount of the first arm, to move the first insertable instrument.

2. The medical system according to claim 1,

wherein the controller is configured to calculate the first operation amount of the first arm to move the first image sensor such that a deviation between the direction of the optical axis of the second image sensor and the direction of the optical axis of the first image sensor falls within a predetermined threshold.

3. The medical system according to claim 1,

wherein the first insertable instrument comprises one or more joints, and

wherein the controller is configured to:

calculate a first operation amount of the one or more joints and the first operation amount of the first arm to move the first image sensor such that the optical axis of the second image sensor substantially coincides with the optical axis of the first image sensor, based on the position and orientation of the first insertable instrument; and

control the one or more joints, based on the first operation amount of the one or more joints, and control the first arm, based on the first operation amount of the first arm, to move the first insertable instrument.

4. The medical system according to claim 1,

wherein the controller is configured to:

acquire a movable range of the first arm in the predetermined coordinate system; and

calculate the first operation amount of the first arm to move the first image sensor based on the position and orientation of the first insertable instrument and the movable range of the first arm in the predetermined coordinate system such that;

a distance between the optical axis of the second image sensor and the optical axis of the first image sensor is minimized, in a case that the optical axis of the first image sensor and the optical axis of the second image sensor are parallel; and

an angle formed by the optical axis of the first image sensor and the optical axis of the second image sensor is minimized, in a case that the optical axis of the first image sensor and the optical axis of the second image sensor are not parallel.

5. The medical system according to claim 1, further comprising a second arm configured to support and move the second insertable instrument,

wherein the controller is configured to:

determine which of the first insertable instrument or the second insertable instrument is operated by an operator;

in response to determining that the first insertable instrument is being operated by the operator;

acquire the position and orientation of the second insertable instrument in the predetermined coordinate system;

calculate the position and direction of the optical axis of the second image sensor based on the position and orientation of the second endoscope;

acquire the position and orientation of the first endoscope in the predetermined coordinate system;

calculate the first operation amount of the first arm based on the position and orientation of the first insertable instrument such that the optical axis of the first image sensor substantially coincides with the optical axis of the second image sensor; and

control the first arm, based on the first operation amount, to move the first insertable instrument; and

in response to determining that the second insertable instrument is being operated by the operator;

acquire the position and orientation of the first insertable instrument in the predetermined coordinate system;

calculate a position and direction of the optical axis of the first image sensor based on the position and orientation of the first insertable instrument;

acquire the position and orientation of the second insertable instrument in the predetermined coordinate system;

calculate a second operation amount of the second arm based on the position and orientation of the second insertable instrument such that the optical axis of the second image sensor substantially coincides with the optical axis of the first image sensor; and

control the second arm, based on the second operation amount, to move the second insertable instrument.

6. The medical system according to claim 1, further comprising:

a first marker disposed on the first insertable instrument;

a second marker disposed on the second insertable instrument;

a camera configured to take an image of both the first marker and the second marker,

wherein the controller is configured to acquire the position and orientation of the second insertable instrument in the predetermined coordinate system, based on the image.

7. A controller for controlling a medical system comprising:

a first insertable instrument comprising a first image sensor;

a second insertable instrument comprising a second image sensor; and

a first arm configured to support and move the first insertable instrument, wherein the controller comprises one or more processors configured to;

acquire a position and orientation of the second insertable instrument in a predetermined coordinate system;

calculate a position and direction of an optical axis of the second image sensor based on the position and orientation of the second insertable instrument;

acquire a position and orientation of the first insertable instrument in the predetermined coordinate system;

calculate a first operation amount of the first arm to move the first image sensor such that the optical axis of the second image sensor substantially coincides with an optical axis of the first image sensor, based on the position and orientation of the first insertable instrument; and

control the first arm, based on the first operation amount of the first arm, to move the first insertable instrument.

8. The controller according to claim 7,

wherein the one or more processors are configured to calculate the first operation amount of the first arm to move the first image sensor such that a deviation between the direction of the optical axis of the second image sensor and the direction of the optical axis of the first image sensor falls within a predetermined threshold.

9. The controller according to claim 7,

wherein the first insertable instrument comprises one or more joints, and

wherein the one or more processors are configured to:

calculate a first operation amount of the one or more joints and the first operation amount of the first arm to move the first image sensor such that the optical axis of the second image sensor substantially coincides with the optical axis of the first image sensor, based on the position and orientation of the first insertable instrument; and

control the one or more joints, based on the first operation amount of the one or more joints, and control the first arm, based on the first operation amount of the first arm, to move the first insertable instrument.

10. The controller according to claim 7,

wherein the one or more processors are configured to:

acquire a movable range of the first arm in the predetermined coordinate system; and

calculate the first operation amount of the first arm to move the first image sensor based on the position and orientation of the first insertable instrument and the movable range of the first arm in the predetermined coordinate system such that;

a distance between the optical axis of the second image sensor and the optical axis of the first image sensor is minimized, in a case that the optical axis of the first image sensor and the optical axis of the second image sensor are parallel; and

an angle formed by the optical axis of the first image sensor and the optical axis of the second image sensor is minimized, in a case that the optical axis of the first image sensor and the optical axis of the second image sensor are not parallel.

11. The controller according to claim 7,

wherein the system further comprises a second arm configured to support and move the second insertable instrument, and

wherein the one or more processors are configured to:

determine which of the first insertable instrument or the second insertable instrument is operated by an operator;

in response to determining that the first insertable instrument is being operated by the operator;

acquire the position and orientation of the second insertable instrument in the predetermined coordinate system;

calculate the position and direction of the optical axis of the second image sensor based on the position and orientation of the second endoscope;

acquire the position and orientation of the first endoscope in the predetermined coordinate system;

calculate the first operation amount of the first arm based on the position and orientation of the first insertable instrument such that the optical axis of the first image sensor substantially coincides with the optical axis of the second image sensor; and

control the first arm, based on the first operation amount, to move the first insertable instrument; and

in response to determining that the second insertable instrument is being operated by the operator;

acquire the position and orientation of the first insertable instrument in the predetermined coordinate system;

calculate a position and direction of the optical axis of the first image sensor based on the position and orientation of the first insertable instrument;

acquire the position and orientation of the second insertable instrument in the predetermined coordinate system;

calculate a second operation amount of the second arm based on the position and orientation of the second insertable instrument such that the optical axis of the second image sensor substantially coincides with the optical axis of the first image sensor; and

control the second arm, based on the second operation amount, to move the second insertable instrument.

12. The controller according to claim 7,

wherein the system further comprises:

a first marker disposed on the first insertable instrument;

a second marker disposed on the second insertable instrument; and

a camera configured to take an image of both the first marker and the second marker, and

wherein the one or more processors are configured to acquire the position and orientation of the second insertable instrument in the predetermined coordinate system, based on the image.

13. A computer readable storage device storing instructions for controlling a medical system comprising:

a first insertable instrument comprising a first image sensor;

a second insertable instrument comprising a second image sensor; and

a first arm configured to support and move the first insertable instrument, wherein the instructions cause a computer to perform processes comprising;

acquiring a position and orientation of the second insertable instrument in a predetermined coordinate system;

calculating a position and direction of an optical axis of the second image sensor based on the position and orientation of the second insertable instrument;

acquiring a position and orientation of the first insertable instrument in the predetermined coordinate system;

calculating a first operation amount of the first arm to move the first image sensor such that the optical axis of the second image sensor substantially coincides with an optical axis of the first image sensor, based on the position and orientation of the first insertable instrument; and

controlling the first arm, based on the first operation amount of the first arm, to move the first insertable instrument.

14. The computer readable storage device according to claim 13,

wherein the instructions cause the computer to perform processes comprising calculating the first operation amount of the first arm to move the first image sensor such that a deviation between the direction of the optical axis of the second image sensor and the direction of the optical axis of the first image sensor falls within a predetermined threshold.

15. The computer readable storage device according to claim 13,

wherein the first insertable instrument comprises one or more joints, and

wherein the instructions cause the computer to perform processes comprising:

calculating a first operation amount of the one or more joints and the first operation amount of the first arm to move the first image sensor such that the optical axis of the second image sensor substantially coincides with the optical axis of the first image sensor, based on the position and orientation of the first insertable instrument; and

controlling the one or more joints, based on the first operation amount of the one or more joints, and control the first arm, based on the first operation amount of the first arm, to move the first insertable instrument.

16. The computer readable storage device according to claim 13,

wherein the instructions cause the computer to perform processes comprising:

acquiring a movable range of the first arm in the predetermined coordinate system; and

calculating the first operation amount of the first arm to move the first image sensor based on the position and orientation of the first insertable instrument and the movable range of the first arm in the predetermined coordinate system such that;

a distance between the optical axis of the second image sensor and the optical axis of the first image sensor is minimized, in a case that the optical axis of the first image sensor and the optical axis of the second image sensor are parallel; and

an angle formed by the optical axis of the first image sensor and the optical axis of the second image sensor is minimized, in a case that the optical axis of the first image sensor and the optical axis of the second image sensor are not parallel.

17. The computer readable storage device according to claim 13,

wherein the system further comprises a second arm configured to support and move the second insertable instrument, and

wherein the instructions cause the computer to perform processes comprising:

determining which of the first insertable instrument or the second insertable instrument is operated by an operator;

in response to determining that the first insertable instrument is being operated by the operator;

acquiring the position and orientation of the second insertable instrument in the predetermined coordinate system;

calculating the position and direction of the optical axis of the second image sensor based on the position and orientation of the second endoscope;

acquiring the position and orientation of the first endoscope in the predetermined coordinate system;

calculating the first operation amount of the first arm based on the position and orientation of the first insertable instrument such that the optical axis of the first image sensor substantially coincides with the optical axis of the second image sensor; and

controlling the first arm, based on the first operation amount, to move the first insertable instrument; and

in response to determining that the second insertable instrument is being operated by the operator;

acquiring the position and orientation of the first insertable instrument in the predetermined coordinate system;

calculating a position and direction of the optical axis of the first image sensor based on the position and orientation of the first insertable instrument;

acquiring the position and orientation of the second insertable instrument in the predetermined coordinate system;

calculating a second operation amount of the second arm based on the position and orientation of the second insertable instrument such that the optical axis of the second image sensor substantially coincides with the optical axis of the first image sensor; and

controlling the second arm, based on the second operation amount, to move the second insertable instrument.

18. The computer readable storage device according to claim 13,

wherein the system further comprises:

a first marker disposed on the first insertable instrument;

a second marker disposed on the second insertable instrument; and

a camera configured to take an image of both the first marker and the second marker, and

wherein the instructions cause the computer to perform processes comprising acquiring the position and orientation of the second insertable instrument in the predetermined coordinate system, based on the image.