Intelligent display

• US 11,188,285 B2
• Filed: 06/29/2015
• Issued: 11/30/2021
• Est. Priority Date: 07/02/2014
• Status: Active Grant

• Current Assignee: Covidien LP (Medtronic Plc)
• Original Assignee: Covidien LP (Medtronic Plc)
• Inventors: Kehat, Israel, Klein, Eyal, Greenburg, Benjamin, Averbuch, Dorian
• Primary Examiner(s): Mattson, Sean D
• Application Number:
  US14/754,058
• Publication Number:
  US 20160000517A1
• Time in Patent Office: 2,346 Days
• CPC Class Codes: A61B 1/0005 combining images e.g. side-by-side, superimposed or tiled
• CPC Class Codes: A61B 1/2676 Bronchoscopes
• CPC Class Codes: A61B 2034/2051 Electromagnetic tracking systems
• CPC Class Codes: A61B 2034/2065 Tracking using image or pattern recognition
• CPC Class Codes: A61B 2090/364 Correlation of different images or relation of image positions in respect to the body
• CPC Class Codes: A61B 2090/367 creating a 3D dataset from 2D images using position information
• CPC Class Codes: A61B 2090/374 NMR or MRI
• CPC Class Codes: A61B 2090/3762 using computed tomography systems [CT]
• CPC Class Codes: A61B 2560/0475 Special features of memory means, e.g. removable memory cards
• CPC Class Codes: A61B 34/20 Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
• CPC Class Codes: A61B 34/25 User interfaces for surgical systems
• CPC Class Codes: A61B 5/0013 Medical image data A61B1/00011, A61B6/56, A61B8/56 take precedence
• CPC Class Codes: A61B 5/066 Superposing sensor position on an image of the patient, e.g. obtained by ultrasound or x-ray imaging
• CPC Class Codes: A61B 5/7425 Displaying combinations of multiple images regardless of image source, e.g. displaying a reference anatomical image with a live image geometric transforms for image registration G06T3/14; enhancement using image fusion G06T5/50; algorithms for registration of images G06T7/30
• CPC Class Codes: G06F 3/14 Digital output to display device ; Cooperation and interconnection of the display device with other functional units
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 2211/412 Dynamic
• CPC Class Codes: G06T 2211/428 Real-time
• CPC Class Codes: G09G 2380/08 Biomedical applications

Abstract

A medical image display apparatus for displaying medical images of a lung on a screen includes a network interface receiving positional information of a navigation instrument from a position sensor of the navigation instrument, a video stream from an optical sensor of the navigation instrument, and medical images from an imaging device, a memory storing a plurality of medical images and instructions, a processor executing the instructions, and a display dynamically displaying images on the screen. The instructions, when executed by the processor, cause the medical image display apparatus to determine whether status information indicates a pathway reviewing mode, a target management mode, or a navigation mode. The instructions, when executed by the processor, further cause the display to dynamically select and update images, which are displayed on the screen, among the plurality of medical images based on the positional information of the navigation instrument and status information.

Claims

1. A medical image display apparatus for displaying medical images of a lung on a screen, comprising:

a network interface configured to receive positional information from a position sensor disposed on one of a catheter or a navigation instrument disposed within a working channel of the catheter, a live bronchoscopic video stream from an optical sensor of a bronchoscope through which the catheter and the navigation instrument are configured to be inserted for navigation within a patient'"'"'s lung, and images from an imaging device;

a memory storing instructions;

a processor configured to execute the instructions; and

a display configured to dynamically display windows on the screen,

wherein the instructions, when executed by the processor, cause the medical image display apparatus to;

determine endoscopic navigation information of the display, which indicates a mode among a pathway reviewing mode, a target management mode, or a navigation mode; and

dynamically vary a number of the windows to be displayed on the screen based on a change of the mode indicated by the determined endoscopic navigation information and the positional information, wherein;

the target management mode includes a first window displayed on the screen presenting a three-dimensional (3D) map of a patient'"'"'s lung and at least one additional window displayed on the screen that is generated in response to a selection of a target in the 3D map of the patient'"'"'s lung, the at least one additional window presenting an image of the selected target; and

the navigation mode further includes;

a central navigation mode wherein a first window and a second window are displayed on the screen, the first window presenting the 3D map of the patient'"'"'s lung and a current position indicator displayed on the 3D map representing a current position of the position sensor within the patient'"'"'s lung, and the second window presenting the live bronchoscopic video stream showing an actual position of the position sensor within the patient'"'"'s lung, wherein the current position indicator moves relative to the 3D map displayed on the screen as the position sensor is navigated within the patient'"'"'s lung such that a position of the current position indicator displayed on the 3D map is synchronized with the actual position of the position sensor shown in the live bronchoscopic video stream; and

a peripheral navigation mode wherein a third window presenting a virtual bronchoscopic video stream is caused to be displayed on the screen with the first and second windows in response to the positional information indicating that the position sensor is advancing distally from a distal end of the bronchoscope toward the target and past a threshold position within a bronchus of the patient, the virtual bronchoscopic video stream including graphical representations of the position sensor, the target, and a planned pathway to the target to depict the position sensor advancing distally to the distal end of the bronchoscope past the threshold position and along the planned pathway toward the target.

2. The medical image display apparatus according to claim 1, wherein the image of the selected target is selected from the group consisting of a sagittal image, a coronal image, and an axial image.

3. The medical image display apparatus according to claim 2, wherein the sagittal, coronal, or axial image is captured by computed tomography, fluoroscopy, computer aided tomography, positron emission tomography, or magnetic resonance imaging.

4. The medical image display apparatus according to claim 1, wherein changes in the positional information indicate movements of the position sensor in the patient'"'"'s lung.

5. The medical image display apparatus according to claim 1, wherein the at least one additional window of the target management mode comprises a first window presenting an axial image of the selected target, a second window presenting a coronal image of the selected target, and a third window presenting a sagittal image of the selected target.

6. The medical image display apparatus according to claim 1, wherein the 3D map is displayed with an orientation indicator.

7. The medical image display apparatus according to claim 6, wherein the orientation indicator shows an orientation of the 3D map.

8. The medical image display apparatus according to claim 1, wherein the display automatically orients the 3D map to show a current position of the position sensor in the 3D map.

9. A medical image display system comprising:

a catheter configured to be received through a working channel of a bronchoscope;

a navigation instrument configured to be received through the catheter and to navigate inside of a patient, at least one of the catheter or the navigation instrument having a position sensor configured to transmit positional information; and

a medical image display apparatus comprising;

a network interface configured to receive the positional information and a live bronchoscopic video stream from the bronchoscope;

a processor configured to execute instructions;

a memory storing the instructions; and

a display configured to display windows on a screen, wherein the instructions, when executed by the processor, cause the medical image display apparatus to;

dynamically vary a number of the windows to be displayed on the screen based on a change of a mode of operation of the medical image display apparatus;

operate in a target management mode to display a first window on the screen presenting a three-dimensional (3D) map of the patient'"'"'s lung and at least one additional window on the screen that is generated in response to a selection of a target in the 3D map of the patient'"'"'s lung, the at least one additional window presenting an image of the selected target;

operate in a central navigation mode to display a first window and a second window, the first window presenting the 3D map of the patient'"'"'s lung and a current position indicator displayed on the 3D map representing a current position of the position sensor within the patient'"'"'s lung, and the second window presenting the live bronchoscopic video stream showing an actual position of the position sensor within the patient'"'"'s lung, wherein the current position indicator moves relative to the presented 3D map as the position sensor is navigated within the patient'"'"'s lung such that a position of the current position indicator displayed on the 3D map is synchronized with the actual position of the position sensor shown in the live bronchoscopic video stream; and

operate in a peripheral navigation mode wherein a third window presenting a virtual bronchoscopic video stream is caused to be displayed on the screen with the first and second windows in response to the positional information indicating that the position sensor is advancing toward the target past a threshold position within the patient'"'"'s lung, the virtual bronchoscopic video stream including graphical representations of the position sensor, the target, and a planned pathway to the target to depict the position sensor advancing distally past the threshold position and along the planned pathway toward the target.

10. The medical image display system according to claim 9, wherein the at least one additional window of the target management mode comprises a first window presenting an axial image of the selected target, a second window presenting a coronal image of the selected target, and a third window presenting a sagittal image of the selected target.