Marking and tracking an area of interest during endoscopy

• US 9,295,372 B2
• Filed: 09/18/2013
• Issued: 03/29/2016
• Est. Priority Date: 09/18/2013
• Status: Active Grant

• Current Assignee: Cerner Innovation, Inc. (Oracle Corporation)
• Original Assignee: Cerner Innovation, Inc. (Oracle Corporation)
• Inventors: Staples, Alan Harris II, Ramsey, Karen Kaye, Hunt, Bryan Michael
• Primary Examiner(s): TSAI, TSUNG YIN
• Application Number:
  US14/030,653
• Publication Number:
  US 20150078615A1
• Time in Patent Office: 923 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: A61B 1/0004 for electronic operation
• CPC Class Codes: A61B 1/00045 Display arrangement
• CPC Class Codes: A61B 1/0005 combining images e.g. side-by-side, superimposed or tiled
• CPC Class Codes: A61B 2090/365 augmented reality, i.e. correlating a live optical image with another image
• CPC Class Codes: A61B 90/37 Surgical systems with images on a monitor during operation
• CPC Class Codes: G06T 2207/10068 Endoscopic image
• CPC Class Codes: G06T 2207/30032 Colon polyp
• CPC Class Codes: G06T 2207/30096 Tumor; Lesion
• CPC Class Codes: G06T 2207/30204 Marker
• CPC Class Codes: G06T 7/0012 Biomedical image inspection
• CPC Class Codes: G06T 7/13 Edge detection
• CPC Class Codes: G06T 7/248 involving reference images or patches
• CPC Class Codes: G06T 7/50 Depth or shape recovery
• CPC Class Codes: G06T 7/74 involving reference images or patches

Abstract

An area of interest of a patient'"'"'s organ may be identified based on the presence of a possible lesion during an endoscopic procedure. The location of the area of interest may then be tracked relative to the camera view being displayed to the endoscopist in real-time or near real-time during the endoscopic procedure. If the area of interest is visually marked on the display, the visual marking is moved with the area of interest as it moves within the camera view. If the area of interest moves outside the camera view, a directional indicator may be displayed to indicate the location of the area of interest relative to the camera view to assist the endoscopist in relocating the area of interest.

Claims

1. One or more computer storage media storing computer usable instructions that, when used by one or more computing devices, cause the one or more computing devices to perform operations comprising:

identifying an area of interest of a patient'"'"'s organ within images captured by an endoscope during an endoscopic procedure;

providing a visual marker identifying the area of interest within displayed images of a camera view being viewed by an endoscopist;

tracking a location of the area of interest relative to the camera view during the endoscopic procedure;

determining the location of the area of interest has changed relative to the camera view;

determining if the area of interest remains within the camera view;

if the area of interest remains within the camera view, moving the visual marker based on the change in the location of the area of interest;

if the area of interest has moved outside of the camera view, providing a visual directional marker based on the location of the area of interest relative to the camera view;

tracking the endoscopist'"'"'s eyes looking at the displayed images of the camera view; and

based on movements of the endoscopist'"'"'s eyes, controlling a directionality of the endoscope.

2. The one or more computer storage media of claim 1, wherein the area of interest is manually identified by an endoscopist.

3. The one or more computer storage media of claim 2, wherein the area of interest is manually identified by the endoscopist by:

receiving input from an endoscopist via an input device marking an area within the displayed images as the area of interest.

4. The one or more computer storage media of claim 1, wherein the area of interest is automatically identified.

5. The one or more computer storage media of claim 4, wherein the area of interest is automatically identified by:

providing a library of lesion shapes;

performing edge detection to determine a shape of an area within the patient'"'"'s organ;

comparing the shape with the library of lesion shapes;

determining the shape as matching a lesion shape from the library of lesion shapes; and

identifying the area as an area of interest based on the shape matching the lesion shape from the library of lesion shapes.

6. The one or more computer storage media of claim 4, wherein the area of interest is automatically identified by:

presenting the images in white light to the endoscopist;

analyzing the images using modified color conditions as a background process; and

identifying the area of interest based on analysis of the images using the modified color conditions.

7. The one or more computer storage media of claim 6, wherein the operations further comprise:

changing the images displayed to the endoscopist to modified color images based on a modified color channel setting used to identify the area of interest.

8. The one or more computer storage media of claim 1, wherein the visual marker comprises at least one selected from the following:

a box surround the area of interest, highlighting an edge of the area of interest, coloring the area of interest, and dimming areas outside of the area of interest.

9. The one or more computer storage media of claim 1, wherein tracking the location of the area of interest relative to a camera view during the endoscopic procedure comprises employing computer vision techniques to analyze images captured during the endoscopic procedure to determine movement of the area of interest relative to the camera view.

10. A method in a clinical computing environment for tracking an area of interest of a patient'"'"'s organ during an endoscopic procedure, the method comprising:

identifying the area of interest within a camera view of an endoscope during the endoscopic procedure;

determining the area of interest has moved outside the camera view during the endoscopic procedure;

displaying a directional indication on displayed images identifying a location of the area of interest relative to the camera view during the endoscopic procedure;

tracking the endoscopist'"'"'s eyes looking at the displayed images of the camera view; and

based on movements of the endoscopist'"'"'s eyes, controlling a directionality of the endoscope.

11. The method of claim 10, wherein identifying the area of interest comprises receiving input from an endoscopist via an input device marking an area within the displayed images as the area of interest.

12. The method of claim 10, wherein the area of interest is automatically identified by:

providing a library of lesion shapes;

performing edge detection to determine a shape of an area within the patient'"'"'s organ;

comparing the shape with the library of lesion shapes;

determining the shape as matching a lesion shape from the library of lesion shapes; and

identifying the area as an area of interest based on the shape matching the lesion shape from the library of lesion shapes.

13. The method of claim 10, wherein the area of interest is automatically identified by:

presenting the displayed images in white light to the endoscopist;

analyzing captured images using modified color conditions as a background process; and

identifying the area of interest based on analysis of the captured images using the modified color conditions.

14. The method of claim 13, wherein the method further comprises:

changing the displayed images to modified color images based on a modified color channel setting used to identify the area of interest.

15. The method of claim 10, wherein the method further comprises displaying a visual marker identifying the area of interest on the displayed images when the area of interest is within the displayed images, the visual marker comprising at least one selected from the following:

a box surround the area of interest, highlighting an edge of the area of interest, coloring the area of interest, and dimming areas outside of the area of interest.

16. The method of claim 10, wherein determining the area of interest has moved outside the camera view during the endoscopic procedure comprises employing computer vision techniques to analyze images captured during the endoscopic procedure to determine movement of the area of interest relative to the camera view.

17. A system in a clinical computing environment for tracking an area of interest of a patient'"'"'s organ during an endoscopic procedure, the system comprising:

one or more processors; and

one or more computer storage media storing instructions to cause the one or more processors to;

identify the area of interest within a camera view of an endoscope during the endoscopic procedure,

determine the area of interest has moved outside the camera view during the endoscopic procedure,

display a directional indication on displayed images identifying a location of the area of interest relative to the camera view during the endoscopic procedure,

tracking the endoscopist'"'"'s eyes looking at the displayed images of the camera view, and

based on movements of the endoscopist'"'"'s eyes, controlling a directionality of the endoscope.

18. The system of claim 17, wherein the area of interest is manually identified by an endoscopist.

19. The system of claim 17, wherein the area of interest is automatically identified by the system analyzing images captured during the endoscopic procedure.

20. The system of claim 17, wherein determining the area of interest has moved outside the camera view during the endoscopic procedure comprises employing computer vision techniques to analyze images captured during the endoscopic procedure to determine movement of the area of interest relative to the camera view.