Method and system for surgical tool localization during anatomical surgery

• US 10,147,188 B2
• Filed: 12/28/2017
• Issued: 12/04/2018
• Est. Priority Date: 02/19/2015
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Chou, Chen-Rui, Liu, Ming-Chang
• Primary Examiner(s): Desire, Gregory M
• Application Number:
  US15/856,298
• Publication Number:
  US 20180122070A1
• Time in Patent Office: 341 Days
• CPC Class Codes: A61B 2034/2065 Tracking using image or pattern recognition
• CPC Class Codes: A61B 2090/374 NMR or MRI
• CPC Class Codes: A61B 2090/3762 using computed tomography systems [CT]
• CPC Class Codes: A61B 34/20 Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
• CPC Class Codes: A61B 90/37 Surgical systems with images on a monitor during operation
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/20004 Adaptive image processing
• CPC Class Codes: G06T 2207/30004 Biomedical image processing
• CPC Class Codes: G06T 2207/30021 Catheter; Guide wire
• CPC Class Codes: G06T 5/20 using local operators
• CPC Class Codes: G06T 7/0012 Biomedical image inspection
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06V 10/56 relating to colour
• CPC Class Codes: G06V 10/755 Deformable models or variational models, e.g. snakes or active contours
• CPC Class Codes: G06V 2201/034 of medical instruments

Abstract

Various aspects of a method and system to localize surgical tools during anatomical surgery are disclosed herein. In accordance with an embodiment of the disclosure, the method is implementable in an image-processing engine, which is communicatively coupled to an image-capturing device that captures one or more video frames. The method includes determination of one or more physical characteristics of one or more surgical tools present in the one or more video frames, based on one or more color and geometric constraints. Thereafter, two-dimensional (2D) masks of the one or more surgical tools are detected, based on the one or more physical characteristics of the one or more surgical tools. Further, poses of the one or more surgical tools are estimated, when the 2D masks of the one or more surgical tools are occluded at tips and/or ends of the one or more surgical tools.

Claims

1. A system comprising:

one or more circuits configured to;

determine physical characteristics of a plurality of surgical tools based on color constraints of said plurality of surgical tools;

detect two-dimensional (2D) masks of said plurality of surgical tools in at least one video frame based on said physical characteristics of said plurality of surgical tools;

estimate poses of said plurality of surgical tools in said at least one video frame based on said 2D masks of said plurality of surgical tools; and

display masked at least one video frame based on said poses of said plurality of surgical tools.

2. The system of claim 1, wherein said one or more circuits are further configured to receive user input from a user terminal.

3. The system of claim 2, wherein said user input corresponds to at least one of a touch operation or a gesture by a user of said system.

4. The system of claim 1, wherein said one or more circuits are communicatively coupled with an image capturing device that is configured to capture said at least one video frame.

5. The system of claim 1, wherein said poses of said plurality of surgical tools are based on occlusion of said 2D masks of said plurality of surgical tools.

6. The system of claim 5, wherein said one or more circuits are further configured to generate a notification indicative of said occlusion of said plurality of surgical tools,

wherein said notification corresponds to at least one of an audio alert, a textual alert, a visual alert, or a haptic alert.

7. A method comprising:

determining physical characteristics of a plurality of surgical tools based on color constraints of said plurality of surgical tools;

detecting two-dimensional (2D) masks of said plurality of surgical tools in at least one video frame based on said physical characteristics of said plurality of surgical tools;

estimating poses of said plurality of surgical tools in said at least one video frame based on said 2D masks of said plurality of surgical tools; and

displaying masked at least one video frame based on said poses of said plurality of surgical tools.

8. The method of claim 7, further comprising receiving user input from a user terminal.

9. The method of claim 8, wherein said user input corresponds to at least one of a touch operation or a gesture by a user.

10. The method of claim 7, wherein said poses of said plurality of surgical tools are based on occlusion of said 2D masks of said plurality of surgical tools.

11. The method of claim 10, further comprising generating a notification indicative of said occlusion of said plurality of surgical tools, wherein said notification corresponds to at least one of an audio alert, a textual alert, a visual alert, or a haptic alert.

12. The method of claim 7, further comprising capturing said at least one video frame.

13. A non-transitory computer-readable medium having stored thereon computer-executable instructions that, when executed by a processor, cause the processor to execute operations, the operations comprising:

determining physical characteristics of a plurality of surgical tools based on color constraints of said plurality of surgical tools;

detecting two-dimensional (2D) masks of said plurality of surgical tools in at least one video frame based on said physical characteristics of said plurality of surgical tools;

estimating poses of said plurality of surgical tools in said at least one video frame based on said 2D masks of said plurality of surgical tools; and

displaying masked at least one video frame based on said poses of said plurality of surgical tools.

14. The non-transitory computer-readable medium of claim 13, further comprising receiving user input from a user terminal.

15. The non-transitory computer-readable medium of claim 14, wherein said user input corresponds to at least one of a touch operation or a gesture by a user.

16. The non-transitory computer-readable medium of claim 13, wherein said poses of said plurality of surgical tools are based on occlusion of said 2D masks of said plurality of surgical tools.

17. The non-transitory computer-readable medium of claim 16, further comprising generating a notification indicative of said occlusion of said plurality of surgical tools,

wherein said notification corresponds to at least one of an audio alert, a textual alert, a visual alert, or a haptic alert.