AUGMENTED REALITY IMAGE DISPLAY SYSTEM AND SURGICAL ROBOT SYSTEM COMPRISING THE SAME

• US 20140275760A1
• Filed: 12/18/2013
• Published: 09/18/2014
• Est. Priority Date: 03/13/2013
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Hwang, Won Jun, Roh, Kyung Shik, LEE, Hee Kuk, Choi, Jung Yun
• Granted Patent: US 9,767,608 B2
• Time in Patent Office: Days
• US Class Current: 600/102
• CPC Class Codes: A61B 1/00045 Display arrangement
• CPC Class Codes: A61B 1/00149 using articulated arms
• CPC Class Codes: A61B 1/04 combined with photographic or television appliances
• CPC Class Codes: A61B 2034/2065 Tracking using image or pattern recognition
• CPC Class Codes: A61B 2090/365 augmented reality, i.e. correlating a live optical image with another image
• CPC Class Codes: A61B 2090/3937 Visible markers
• CPC Class Codes: A61B 34/30 Surgical robots
• CPC Class Codes: A61B 34/37 Master-slave robots A61B34/35 takes precedence
• CPC Class Codes: A61B 5/055 involving electronic [EMR] or nuclear [NMR] magnetic resonance, e.g. magnetic resonance imaging
• CPC Class Codes: A61B 5/7445 Display arrangements, e.g. multiple display units
• CPC Class Codes: A61B 6/032 Transmission computed tomography [CT]
• CPC Class Codes: A61B 90/361 Image-producing devices, e.g. surgical cameras
• CPC Class Codes: A61B 90/92 coded with colour
• CPC Class Codes: A61B 90/94 coded with symbols, e.g. text
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2210/41 Medical
• CPC Class Codes: G16H 20/40 relating to mechanical, radiation or invasive therapies, e.g. surgery, laser therapy, dialysis or acupuncture
• CPC Class Codes: G16H 40/63 for local operation

Abstract

An augmented reality image display system may be implemented together with a surgical robot system. The surgical robot system may include a slave system performing a surgical operation, a master system controlling the surgical operation of the slave system, an imaging system generating a virtual image of the inside of a patient'"'"'s body, and an augmented reality image display system including a camera capturing a real image having a plurality of markers attached to the patient'"'"'s body or a human body model. The augmented reality image system may include an augmented reality image generator which detects the plurality of markers in the real image, estimates the position and gaze direction of the camera using the detected markers, and generates an augmented reality image by overlaying a region of the virtual image over the real image, and a display which displays the augmented reality image.

Claims

1. A surgical robot system, comprising:

a slave system to perform a surgical operation upon a object;

a master system to control the surgical operation of the slave system; and

an imaging system to generate the virtual image of an inside of the object; and

an augmented reality image display system comprising;

a camera to capture a real image of the object and a plurality of markers attached to the object,

an augmented reality image generator to detect a plurality of markers in the real image, to estimate a position and gaze direction of the camera using the detected markers, and to generate an augmented reality image by overlaying a region of a virtual image corresponding to the estimated position and gaze direction of the camera over the real image, and

a display to display the augmented reality image.

2. The surgical robot system according to claim 1, wherein the slave system comprises:

a surgical tool to perform the surgical operation;

an endoscope to capture an image of a region inside of the object;

a position sensor to detect a position of the surgical tool; and

a position calculator to calculate position information of the surgical tool using detected signals by the position sensor.

3. The surgical robot system according to claim 2, wherein the augmented reality image generator receives position information of the surgical tool from the slave system and generates a virtual surgical tool at a region matching the position information in the augmented reality image.

4. The surgical robot system according to claim 1, wherein:

the camera is attached to the display; and

the augmented reality image generator generates the augmented reality image by estimating the position and gaze direction of the camera changing in accordance with movement of the display in real-time and by compositing a region of the virtual image corresponding to the estimated position and gaze direction of the camera and the real image.

5. The surgical robot system according to claim 1, wherein the augmented reality image generator calculates the position information of each of the detected markers in the real image and estimates the position and gaze direction of the camera using the calculated position information of each of the markers.

6. The surgical robot system according to claim 5, wherein the position information of each of the markers comprises a distance between the markers and a connection angle between the markers.

7. The surgical robot system according to claim 5, wherein the position information of each of the markers comprises size information of the marker in the real image.

8. The surgical robot system according to claim 7, wherein the augmented reality image generator generates the augmented reality image by calculating a distance between the camera and the marker using the size information of the marker, and enlarges or contracts the virtual image in accordance with the calculated distance, and composites the virtual image and the real image.

9. The surgical robot system according to claim 1, wherein the imaging system comprises:

a three-dimensional (3D) image conversion unit to convert an image of the object captured in advance of the surgical operation, into a 3D image;

a virtual image generator to generate a virtual image by projecting the converted 3D image onto the image acquired by the endoscope; and

an image storage unit to store the 3D image and the virtual image.

10. The surgical robot system according to claim 9, wherein the image of the object captured in advance of the surgical operation includes at least one of a computed tomography (CT) image and a magnetic resonance imaging (MRI) image.

11. The surgical robot system according to claim 1, wherein the augmented reality image is generated to have a region corresponding to the position and gaze direction of the camera to face forward.

12. An augmented reality image display system comprising:

a camera capturing a real image of an object and a plurality of markers attached to the object;

an augmented reality image generator to detect a plurality of markers in the real image, to estimate a position and gaze direction of the camera using the detected markers, and to generate an augmented reality image by overlaying a region of a virtual image corresponding to the estimated position and gaze direction of the camera over the real image; and

a display to display the augmented reality image.

13. The augmented reality image display system according to claim 12, wherein:

the camera is attached to the display; and

the augmented reality image generator generates the augmented reality image by estimating the position and gaze direction of the camera changing in accordance with movement of the display in real-time and by compositing a region of the virtual image corresponding to the estimated position and gaze direction of the camera and the real image.

14. The augmented reality image display system according to claim 12, wherein the augmented reality image generator calculates position information of each of the detected markers in the real image and estimates position and gaze direction of the camera using the calculated position information of each of the markers.

15. The augmented reality image display system according to claim 14, wherein the position information of each of the markers comprises a distance between the markers and a connection angle between the markers.

16. The augmented reality image display system according to claim 14, wherein the position information of each of the markers comprises size information of the marker in the real image.

17. The augmented reality image display system according to claim 16, wherein the augmented reality image generator generates the augmented reality image by calculating a distance between the camera and the marker using the size information of the marker, and enlarges or contracts the virtual image in accordance with the calculated distance, and composites the virtual image and the real image.

18. The augmented reality image display system according to claim 12, wherein the augmented reality image comprises a virtual image of a surgical tool inserted into the object.

19. The augmented reality image display system according to claim 12, wherein the augmented reality image is generated to have a region corresponding to the position and gaze direction of the camera to face forward.