Interactive digital arrow (d'arrow) three-dimensional (3D) pointing

• US 5,694,142 A
• Filed: 11/17/1994
• Issued: 12/02/1997
• Est. Priority Date: 06/21/1993
• Status: Expired due to Fees

• Current Assignee: General Electric Company
• Original Assignee: General Electric Company
• Inventors: Adams, William John, Darrow, Robert David, Dumoulin, Charles Lucian
• Primary Examiner(s): SARAS, STEVEN J
• Application Number:
  US08/340,783
• Time in Patent Office: 1,111 Days
• Field of Search: 128/653.1, 128/653.2, 364/413.13, 364/413.22, 178/18, 345/7-9, 345/113, 345/114, 345/115, 345/133, 345/156, 345/157, 345/158, 345/167, 353/28, 348/51-53, 348/65, 348/77, 395/119, 395/124, 395/125, 395/127, 395/128, 382/128, 382/131, 382/132, 359/65, 235/1 R
• US Class Current: 345/9
• CPC Class Codes: A61B 2090/365 augmented reality, i.e. correlating a live optical image with another image
• CPC Class Codes: A61B 2090/371 with simultaneous use of two cameras
• CPC Class Codes: A61B 2090/372 Details of monitor hardware
• CPC Class Codes: A61B 2090/502 Headgear, e.g. helmet, spectacles
• CPC Class Codes: A61B 90/36 Image-producing devices or illumination devices not otherwise provided for
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G16H 20/30 relating to physical therapies or activities, e.g. physiotherapy, acupressure or exercising
• CPC Class Codes: G16H 30/20 for handling medical images, e.g. DICOM, HL7 or PACS
• CPC Class Codes: G16H 30/40 for processing medical images, e.g. editing
• CPC Class Codes: G16H 40/63 for local operation
• CPC Class Codes: G16H 50/50 for simulation or modelling of medical disorders

Abstract

An interactive three-dimensional (3D) pointing device for selecting points within a subject employs a tracking device which determines the position of the operator, a semi-transparent screen positioned by the operator and the subject and provides this information to a model workstation. The model workstation superimposes computer graphic images of internal structures of the subject on a the semi-transparent screen through which the operator is viewing the subject. The superimposed image is derived from image data either previously generated and stored or obtained with an imaging system. The images of the internal structures are registered with the operator'"'"'s view of the external structures of the operator. The operator interactively views internal and external structures and the relation between them simultaneously, while moving the screen to select 3D target points at an image depth within the subject. Optionally other input devices may be used to identify current `target points` as selected points. The 3D points are then provided to an output device which utilizes them. Another embodiment employs stereoscopic viewing methods to provide 3D representations of the internal images superimposed on external structures to allow the operator to employ parallax to select 3D points.

Claims

1. A real-time three dimensional (3D) pointing device for identifying 3D locations within a subject comprising:

a) a semi-transparent screen interposed between said subject and an operator for displaying an image provided to it to appear superimposed on external structures of said subject seen through the screen;

b) a moveable mechanical arm for holding the semi-transparent screen in a position and orientation selected by said operator between said operator and said subject such that said operator may view said subject through said semi-transparent screen;

c) touch sensors for indicating a two dimensional (2D) position of the semi-transparent screen selected by said operator and identifying this position as a screen location;

d) a tracking device for repeatedly measuring the location and orientation of the operator, said subject, and the semi-transparent screen;

e) a symbol generation unit coupled to the tracking device for determining a depth based upon the screen distance from the subject, and for displaying a symbol at the `target location` defined as the screen location and depth on the semi-transparent screen in proper relation to the internal and external structures; and

f) a workstation coupled to the tracking device, for receiving the locations and orientations of said subject, operator and the semi-transparent screen, creating an image of internal structures of said subject from a set of imaging data on the semi-transparent screen consistent with the locations and orientations of the operator, said subject and the semi-transparent screen.

2. The real-time three dimensional (3D) pointing device of claim 1 wherein the semi-transparent screen is one capable of adjustable the relative degree of transparency of the images of the internal and external structures.

3. The real-time three dimensional (3D) pointing device of claim 1 further comprising touch sensors for indicating a two dimensional (2D) position of the semi-transparent screen selected by said operator and identifying this position as the screen location to the workstation.

4. The real-time three dimensional (3D) pointing device of claim 1 further comprising a symbol generation unit coupled to the tracking device for displaying a symbol representing the `target location` on the semi-transparent in proper relation to the internal and external structures.

5. The real-time three dimensional (3D) pointing device of claim 1 further comprising a moveable mechanical arm for holding the semi-transparent screen in a position and orientation selected by said operator between said operator and said subject such that said operator may view said subject through said semi-transparent screen.

6. The real-time three dimensional (3D) pointing device of claim 1 further comprising an imaging device for obtaining a 3D set of imaging data of internal structures of said subject and for providing the 3D set of imaging data to the workstation.

7. The real-time three dimensional (3D) pointing device of claim 1 wherein the workstation creates stereoscopic pairs of images of internal structures, and further comprising:

stereoscopic display means synchronized with the semi-transparent screen for providing each of the stereoscopic pairs of images to each eye of an operator thereby simulating a 3D image of internal structures superimposed upon and external structures of said subject.

8. The real-time three dimensional (3D) pointing device of claim 1 wherein the semi-transparent screen has a surface constructed of a material which may be written upon and erased.

9. A method of aiding an operator in identifying three-dimensional (3D) locations within a subject comprising the steps of:

a) acquiring multi-dimensional imaging data defining internal structures of said subject;

b) positioning a semi-transparent screen at the selected location and orientation between an operator and said subject allowing the operator to view external structures of said subject through the semi-transparent screen;

c) measuring locations and orientations of said subject, said operator and the semi-transparent screen;

d) superimposing a computer generated image of the internal structures on the semi-transparent screen consistent with the measured locations and orientations from the imaging data;

e) determining a position of a semi-transparent screen touched by the operator;

f) identifying this position as the selected screen location;

g) calculating a depth within the subject being a function of the distance between the screen and subject; and

h) creating a symbol on the screen representing a 3D `target location`, being the depth from the selected screen location.

10. The method of aiding an operator in identifying 3D locations within a subject of claim 9 further comprising, after the step of superimposing a computer generated image, the steps of:

a) determining a position of a semi-transparent screen touched by the operator;

b) identifying this position as the selected screen location.

11. The method of aiding an operator in identifying 3D locations within a subject of claim 9 further comprising, after the step of superimposing a computer generated image, the step of operating an input device to identify a currently `target location` as a `selected point`.

12. The method of aiding an operator in identifying 3D locations within a subject of claim 9 wherein steps "b" through "e" are repeated to allow the operator to interactively view internal and external structures while identifying 3D points within said subject.

13. A real-time three dimensional (3D) pointing device for interacting with an operator to select 3D locations of stored computer graphic models of structures within a subject comprising:

a) a semi-transparent screen allowing said operator to see external structures of said subject through the screen and also for displaying images of internal structures, superimposed upon the external structures;

b) a mechanical arm coupled to the screen and fixed at a second end for adjustably holding the semi-transparent screen between said subject and said operator in an operator-selected position;

c) touch sensors for interactively determining 2D screen locations touched by the operator;

d) tracking device for measuring locations and orientation of the semi-transparent screen, said operator and subject;

e) a workstation coupled to the semi-transparent screen, the touch sensors, tracking device, for receiving the locations and orientations of the screen, said subject, and operator, for displaying computer graphic models of said internal structures of said subject correctly registered with said subject'"'"'s external structures from the operator'"'"'s location and position;

f) symbol generation device coupled to the tracking device for determining a distance between the screen and said subject, determining a depth perpendicular to the screen within said subject being a function of this distance, and displaying a symbol on the screen representing a 3D location defined by the 2D screen location selected and the depth, as viewed from operator location and orientation through the screen.