System and method for providing a contour video with a 3D surface in a medical navigation system

• US 10,543,045 B2
• Filed: 07/13/2015
• Issued: 01/28/2020
• Est. Priority Date: 07/13/2015
• Status: Active Grant

• Current Assignee: Synaptive Medical (Barbados) Inc.
• Original Assignee: Synaptive Medical (Barbados) Inc.
• Inventors: Sela, Gal, Chen, Sean Jy-Shyang, Mak, Siu Wai Jacky, Abhari, Kamyar, Richmond, Joshua Lee
• Primary Examiner(s): Du, Haixia
• Application Number:
  US15/575,552
• Publication Number:
  US 20180125586A1
• Time in Patent Office: 1,660 Days
• CPC Class Codes: A61B 2034/2055 Optical tracking systems
• CPC Class Codes: A61B 2034/2057 Details of tracking cameras
• CPC Class Codes: A61B 2090/103 Cranial plugs for access to brain
• CPC Class Codes: A61B 2090/363 Use of fiducial points
• CPC Class Codes: A61B 2090/367 creating a 3D dataset from 2D images using position information
• CPC Class Codes: A61B 2090/373 using light, e.g. by using optical scanners
• CPC Class Codes: A61B 2090/3735 Optical coherence tomography [OCT]
• CPC Class Codes: A61B 34/20 Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
• CPC Class Codes: A61B 5/00 Measuring for diagnostic purposes radiation diagnosis A61B6/00; diagnosis by ultrasonic, sonic or infrasonic waves A61B8/00; Identification of persons
• CPC Class Codes: A61B 90/37 Surgical systems with images on a monitor during operation
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 19/003 Navigation within 3D models or images
• CPC Class Codes: G06T 2207/30016 Brain
• CPC Class Codes: G06T 2207/30096 Tumor; Lesion
• CPC Class Codes: H04N 13/204 using stereoscopic image cameras stereoscopic photography G03B35/00
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/246 Calibration of cameras
• CPC Class Codes: H04N 13/254 in combination with electromagnetic radiation sources for illuminating objects
• CPC Class Codes: H04N 13/30 Image reproducers optical systems for producing stereoscopic or other three-dimensional effects G02B30/00
• CPC Class Codes: H04N 23/57 Mechanical or electrical details of cameras or camera modules specially adapted for being embedded in other devices

Abstract

A medical navigation system for displaying a three dimensional (3D) surface video of a target is provided. The medical navigation system comprises a 3D imaging device, a camera, a display, and a controller electrically coupled to the 3D imaging device, the camera, and the display. The controller has a processor coupled to a memory. The controller is configured to perform calibration of input devices; acquire 3D depth data of the target from a signal generated by the 3D imaging device; construct a 3D surface contour of the target based on the 3D depth data; acquire a video stream of the target from a signal generated by the camera; generate a 3D surface video based on the 3D surface contour and the video stream; and display the 3D surface video on the display.

Claims

1. A medical navigation system for displaying a three dimensional (3D) surface video of a target, the medical navigation system comprising:

a structured light 3D scanner;

a camera;

a display; and

a controller electrically coupled to the 3D scanner, the camera, and the display, the controller having a processor coupled to a memory, the controller being configured to;

acquire the 3D depth data of the target from a signal generated by the 3D scanner;

construct a 3D surface contour, including a 3D point cloud, of the target based on the 3D depth data;

acquire a video stream of the target from a signal generated by the camera;

generate a 3D surface video based on the 3D surface contour and the video stream; and

display the 3D surface video on the display;

wherein the 3D surface contour is used by the controller to exclude occlusion events such that an object passing between the target and at least one of the 3D scanner and the camera is not visible in the 3D surface video.

2. The medical navigation system according to claim 1, further comprising:

a positioning device having a positioning arm with an end effector at the end of the positioning arm, the positioning device electrically coupled to the controller and at least one of the 3D scanner and the camera being mountable on the end effector.

3. The medical navigation system according to claim 1, wherein the camera is selected from a group consisting of a video camera, an infrared camera, a visible light camera, and a non-visible light camera.

4. The medical navigation system according to claim 1, wherein the 3D surface video displayed on the display shows a 3D video that is dynamically rotatable about any axis.

5. The medical navigation system according to claim 1, wherein the display includes a two dimensional video display.

6. The medical navigation system according to claim 1, wherein the display includes a stereo display system.

7. The medical navigation system according to claim 1, wherein generating the 3D surface video includes coloring each point of the 3D point cloud using colour provided by the video stream.

8. The medical navigation system according to claim 1, wherein any objects having a depth that is beyond a threshold depth distance outside of the 3D surface contour are not shown in the 3D surface video.

9. The medical navigation system according to claim 1, wherein the target includes human tissue.

10. The medical navigation system according to claim 1, further comprising input devices including at least one of the 3D scanner, the camera, and a tracking system of the medical navigation system, the controller being further configured to:

perform calibration of input devices by mapping coordinates of the input devices into a common coordinate system.

11. The medical navigation system according to claim 1, wherein displaying the 3D surface video on the display includes overlaying the 3D surface video onto a corresponding portion of pre-operative images displayed on the display.

12. A method for displaying a three dimensional (3D) surface video of a target in a system having a structured light 3D scanner, a camera, a display, and a controller electrically coupled to the 3D scanner, the camera, and the display, the method comprising:

acquiring the 3D depth data of the target from a signal generated by the 3D scanner;

constructing a 3D surface contour, including a 3D point cloud, of the target based on the 3D depth data;

acquiring a video stream of the target from a signal generated by the camera;

generating a 3D surface video based on the 3D surface contour and the video stream;

displaying the 3D surface video on the display; and

excluding occlusion events using the 3D surface contour such that an object passing between the target and at least one of the 3D scanner and the camera is not visible in the 3D surface video.

13. The method according to claim 12, wherein the system further has a positioning device having a positioning arm with an end effector at the end of the positioning arm, the positioning device electrically coupled to the controller and at least one of the 3D scanner and the camera being mountable on the end effector.

14. The method according to claim 12, wherein the camera is selected from a group consisting of a video camera, an infrared camera, a visible light camera, and a non-visible light camera.

15. The method according to claim 12, wherein the 3D surface video displayed on the display shows a 3D video that is dynamically rotatable about any axis.

16. The method according to claim 12, wherein the display includes a two dimensional video display.

17. The method according to claim 12, wherein the display includes a stereo display system.

18. The method according to claim 12, wherein generating a 3D surface video includes coloring each point of the 3D point cloud using colour provided by the video stream.

19. The method according to claim 12, wherein any objects having a depth that is beyond a threshold depth distance outside of the 3D surface contour are not shown in the 3D surface video.

20. The method according to claim 12, wherein the target includes human tissue.

21. The method according to claim 12, the method further comprising performing calibration of input devices by mapping coordinates of the input devices into a common coordinate system, wherein the input devices include at least one of the 3D scanner, the camera, and a tracking system of the medical navigation system.

22. The method according to claim 12, wherein displaying the 3D surface video on the display includes overlaying the 3D surface video onto a corresponding portion of pre-operative images displayed on the display.