3D display system and method

• US 7,376,903 B2
• Filed: 06/29/2004
• Issued: 05/20/2008
• Est. Priority Date: 06/29/2004
• Status: Active Grant

• Current Assignee: GE Medical Systems Information Technologies Incorporated (General Electric Company)
• Original Assignee: GE Medical Systems Information Technologies Incorporated (General Electric Company)
• Inventors: Rai, Khal A., Morita, Mark M., Fors, Steven L.
• Primary Examiner(s): Lo; Weilun
• Assistant Examiner(s): Chuong; Truc T.
• Application Number:
  US10/880,772
• Publication Number:
  US 20050289472A1
• Time in Patent Office: 1,421 Days
• Field of Search: 715/723, 715/756, 715/757, 715/701, 715/702, 600/424
• US Class Current: 715/757
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00

Abstract

A 3D volumetric display system is configured to generate 3D diagnostic displays of 3D volumetric data acquired from a patient by an imaging system in a virtual-reality environment and to permit a user to conduct diagnostic interpretation of images in the virtual-reality environment and to permit the user to interact with the 3D diagnostic displays.

Claims

1. An apparatus comprising:

a 3D volumetric display system configured to generate 3D diagnostic displays of 3D volumetric data acquired from a patient by at least two of an imaging system, a picture archival and communication system (PACS) and an electronic medical record (EMR) system simultaneously in a virtual-reality environment, and wherein the display system is configured to permit a user to conduct diagnostic interpretation of images in the virtual-reality environment and to permit the user to interact with the 3D diagnostic displays, wherein the display system is configured to permit the user to conduct volumetric 3D diagnostic interpretation of images without post-processing the images, wherein post-processing includes having the images manipulated and compiled by a process of image review from the manner of acquisition and reconstruction of images,

wherein the display system comprises a haptics-enhanced virtual-reality device so that the user can touch and interact with a virtual object, wherein the haptics-enhanced virtual-reality device includes a projector, a transflective mirror positioned at an angle, and an overhead substantially opaque screen, which all are coupled to one another to display stereoscopic images that is projected on the overhead substantially opaque screen and is reflected in real-time 3D images on the transflective mirror,

wherein the display system further comprises a haptic force feedback device having robotic manipulators that applies force to the user'"'"'s hand corresponding to the environment that a virtual effector is in and wherein the haptics force feedback device is used to indicate whether the user'"'"'s hand is in contact with a virtual anatomical structure of a patient'"'"'s body.

2. The apparatus of claim 1 further comprising a graphical user interface configured to permit the user to access or navigate through the 3D volumetric data acquired from the patient.

3. The apparatus of claim 2, wherein the graphical user interface permits the user to manipulate the 3D volumetric data in any orientation, angle and zooming setting.

4. The apparatus of claim 2, wherein the graphical user interface enables the user to access, view, manipulate, and conduct diagnostic interpretation of the images.

5. The apparatus of claim 1, wherein the graphical user interface is executed in a virtual reality environment.

6. The apparatus of claim 1, wherein the stereoscopic images displayed in such that the displayed images track the user'"'"'s head movement and further wherein the stereoscopic images permit the user to view the imagery from more than one position.

7. The apparatus of claim 1, wherein the user'"'"'s hands are immersed in the 3D diagnostic displays, so the user is capable to manipulate the 3D volumetric data.

8. The apparatus of claim 1, wherein the user is capable to navigate through the 3D volumetric data to locate pathology in the virtual-reality environment.

9. The apparatus of claim 1, wherein the 3D diagnostic displays further comprises a 3D planner image wherein the 3D planner image is constructed from a plurality of 2D images to facilitate measurement of the diagnostic interpretation of images for anomaly.

10. The apparatus of claim 9, wherein the plurality of 2D images include a plurality of 2D viewports which is capable to simultaneously interact with the virtual anatomical structure of the patient'"'"'s body.

11. The apparatus of claim 9, wherein the 3D planner image is depicted in three orthogonal planes including transverse, sagittal, and coronal planes.

12. The apparatus of claim 9, wherein the 3D planner image and the volumetric 3D diagnostic images are displayed simultaneously.

13. The apparatus of claim 1, wherein the display system is capable of displaying the 2D images, 3D Planner images, and volumetric 3D diagnostic images simultaneously.

14. The apparatus of claim 1, wherein the imaging system includes a picture archiving and communication system (PACS) server and an image workstation coupled to one another.

15. The apparatus of claim 1, wherein the imaging system is a computed tomography (CT) imaging system.

16. The apparatus of claim 1, wherein the imaging system is a Magnetic Resonance (MR) imaging system.

17. The apparatus of claim 1, wherein the imaging system is an X-ray imaging system.

18. The apparatus of claim 1, wherein the imaging system is an ultrasonic imaging system.

19. A diagnostic apparatus comprising:

a display system including a graphical user interface (GUI) configured to be accessed simultaneously in at least two of a picture archiving and communication system (PACS), an image workstation, and an electronic medical record (EMR) system, and to navigate through a diagnostic image data and a patient'"'"'s data without post-processing of the diagnostic images, wherein the post-processing of the diagnostic images define having the images manipulated and compiled by a process of image review from the manner of acquisition and reconstruction of images, the display system including a haptics-enhanced virtual-reality device coupled to the PACS and the image workstation to display stereoscopic images so that a user can touch and interact with a virtual object,

wherein the haptics-enhanced virtual-reality device includes a projector, a transflective mirror positioned at an angle, and an overhead substantially opaque screen, which all are coupled to one another to display stereoscopic images that is projected on the overhead substantially opaque screen and is reflected in real-time 3D images on the transflective mirror,

wherein the display system further comprises a haptic force feedback device having robotic manipulators that applies force to the user'"'"'s hand corresponding to the environment that a virtual effector is in and wherein the haptics force feedback device is used to indicate whether the user'"'"'s hand is in contact with a virtual anatomical structure of a patient'"'"'s body.

20. The apparatus of claim 19, wherein the graphical user interface further comprising an interface tool configured by the user to customize the user interface on the display system.

21. The apparatus of claim 19, wherein graphical user interface further comprising a tool palette window to display a plurality of images information icons.

22. The apparatus of claim 19, wherein the stereoscopic images are viewed by the user wearing 3D goggles.

23. The apparatus of claim 19, wherein the haptics-enhanced virtual-reality device includes a user input device that is coupled to the display system which allows the user to navigate and to interact with the diagnostic images and the patient'"'"'s data.

24. A method of assisting medical diagnostic interpretation of images and data, the method comprising the steps of:

generating diagnostic images of a portion of patient'"'"'s body by using a display system during examination of the portion of patient'"'"'s body in accordance to symptoms;

replicating the portion of patient'"'"'s body from the diagnostic images by using a haptics-enhanced virtual-reality device of the display system, wherein the replicating step includes retrieving the diagnostic images from at least two of a picture archival and communication system (PACS), an imaging system and an electronic medical record (EMR) system simultaneously without post-processing the diagnostic images, wherein the display system is configured to permit the user to conduct volumetric 3D diagnostic interpretation of images without having the images manipulated and compiled by a process of image review from the manner of acquisition and reconstruction of images; and

displaying the replicated portion of the patient'"'"'s body in a virtual environment, wherein the haptics-enhanced virtual-reality device includes a projector, a transflective mirror positioned at an angle, and an overhead substantially opaque screen, which all are coupled to one another to display stereoscopic images that is projected on the overhead substantially opaque screen and is reflected in real-time 3D images on the transflective mirror,

wherein the display system further comprises a haptic force feedback device having robotic manipulators that applies force to the user'"'"'s hand corresponding to the environment that a virtual effector is in and wherein the haptics force feedback device is used to indicate whether the user'"'"'s hand is in contact with a virtual anatomical structure of a patient'"'"'s body.

25. The method of claim 24, wherein the step of generating diagnostic images include volumetric real-time images to locate pathology in the virtual-reality environment.

26. The method of claim 24, wherein the step of generating diagnostic images includes a step of enabling a user to simultaneously navigating among 2D and 3D images and a patient'"'"'s data.

27. The method of claim 24, wherein the step of generating diagnostic images includes a step of permitting a user to navigate through the diagnostic real-time images and the patient'"'"'s data without post-processing of the diagnostic images.

28. The method of claim 24 further comprising displaying a 3D planner image constructed from 2D images to facilitate diagnostic interpretation of images.

29. The method of claim 24, wherein the stereoscopic images displayed in such that the displayed images track the user'"'"'s head movement and further wherein the stereoscopic images permit the user to view the imagery from more than one position.

30. The method of claim 24 further comprising a haptic feedback device having robotic manipulators that push back against the user'"'"'s hand with a force that correspond to the environment that a virtual effector is in and wherein the haptics feedback device is used to indicate whether or not the user'"'"'s hand is in contact with a virtual anatomical structure of the patient'"'"'s body.

31. A diagnostic system comprising:

means for generating diagnostic real-time images of a portion of patient'"'"'s body during examination of the portion of patient'"'"'s body in accordance to symptoms;

means for replicating the portion of patient'"'"'s body from the real-time images, wherein the means for replicating includes retrieving the real time images from any number of a picture archival and communication system (PACS), an imaging system and an electronic medical record (EMR) system simultaneously without post-processing the diagnostic images, wherein the display system is configured to permit the user to conduct volumetric 3D diagnostic interpretation of images without having the images manipulated and compiled by a process of image review from the manner of acquisition and reconstruction of images; and

means for displaying the replicating portion of patient'"'"'s body in a virtual environment, wherein the means for displaying includes a haptics-enhanced virtual-reality device so that the user can touch and interact with a virtual object, wherein the haptics-enhanced virtual-reality device includes a projector, a transflective mirror positioned at an angle, and an overhead substantially opaque screen, which all are coupled to one another to display stereoscopic images that is projected on the overhead substantially opaque screen and is reflected in real time 3D images on the reflective mirror,

wherein the means for displaying further include a haptic force feedback device having robotic manipulators that applies force to the user'"'"'s hand corresponding to the environment that a virtual effector is in and wherein the haptics force feedback device is used to indicate whether the user'"'"'s hand is in contact with a virtual anatomical structure of a patient'"'"'s body.

32. The diagnostic system of claim 31, wherein means for generating diagnostic real-time images include volumetric real-time images to locate pathology in the virtual-reality environment.