Augmented reality system controlled by probe position

• US 7,493,153 B2
• Filed: 06/13/2001
• Issued: 02/17/2009
• Est. Priority Date: 06/13/2001
• Status: Expired due to Term

• Current Assignee: Volume Interactions Pte Ltd. (Bracco SPA)
• Original Assignee: Volume Interactions Pte Ltd. (Bracco SPA)
• Inventors: Ahmed, Syed Nadeem, Kockro, Ralf Alfons
• Primary Examiner(s): Smith; Ruth S
• Assistant Examiner(s): MEHTA, PARIKHA SOLANKI
• Application Number:
  US10/480,490
• Publication Number:
  US 20050203367A1
• Time in Patent Office: 2,806 Days
• Field of Search: 600/411, 600/417, 600/427, 600/429, 600/424, 600/407, 606/130, 382/128, 345/7
• US Class Current: 600/407
• CPC Class Codes: A61B 2034/102 Modelling of surgical devices, implants or prosthesis
• CPC Class Codes: A61B 2034/2055 Optical tracking systems
• CPC Class Codes: A61B 2034/2068 using pointers, e.g. pointers having reference marks for determining coordinates of body points
• CPC Class Codes: A61B 2090/365 augmented reality, i.e. correlating a live optical image with another image
• CPC Class Codes: A61B 2090/368 changing the image on a display according to the operator's position
• CPC Class Codes: A61B 34/20 Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
• CPC Class Codes: A61B 34/25 User interfaces for surgical systems
• CPC Class Codes: A61B 90/36 Image-producing devices or illumination devices not otherwise provided for
• CPC Class Codes: G02B 2027/0134 of stereoscopic type
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted

Abstract

A guide system for use by a user who performs an operation in a defined three-dimensional region is disclosed, the system including a data processing apparatus for generating images of the subject of the operation in co-registration with the subject, a display for displaying the images to the user, a probe having a position which is visible to the user, and a tracking unit for tracking the location of the probe by the system and transmitting that location to the data processing apparatus, the data processing apparatus being arranged, upon the user moving the probe to a selection region outside and surrounding the defined region, to generate one or more virtual buttons, each of the buttons being associated with a corresponding instruction to the system, the data processing apparatus being arranged to register a selection by the user of any of the virtual buttons, the selection including positioning of the probe in relation to the apparent position of that virtual button, and to modify the computer-generated image based on the selection.

Claims

1. A guide system, comprising:

a data processing apparatus for generating one or more images of a subject, said images being co-registered with the subject;

a display for displaying the images to the user;

a probe; and

a tracking unit for tracking the location of the probe and transmitting said location to the data processing apparatus,

wherein, said data processing apparatus generates said co-registered images of the subject in a main display area when the probe is within a defined three-dimensional visualization region, and upon a user moving the probe to a defined three-dimensional interface region outside of the visualization region, the data processing apparatus ceases to generate the co-registered images and causes the display in the main display area of an icon representing the probe and one or more virtual buttons, each of said virtual buttons being associated with a corresponding instruction to the system,

and wherein the data processing apparatus is arranged to obtain a selection by the user of any of the virtual buttons, said selection effected by positioning the physical probe to position said icon relative to the apparent position of said virtual button, and to modify the generated images based on the selection.

2. A system according to claim 1 wherein the data processing apparatus is arranged to generate images of the subject of the operation overlaid on one of images of the subject and the subject itself.

3. A system according to claim 1 in which the positioning of the icon relative to the apparent position of said virtual button includes aligning a longitudinal axis of the probe in the direction of the button.

4. A system according to claim 3, wherein the data processing apparatus is arranged, while the virtual buttons are displayed, to include in the image a line extending from said icon.

5. The system of claim 4, wherein said icon is a virtual probe and is displayed as moving analogously to movements of the probe.

6. A system according to claim 2 in which the display is adapted to be mounted on the head of a user, the user being able to view the subject through the display, so as to see the one or more images generated by the data processing apparatus superimposed on a true image of the subject, the tracking unit monitoring the position of the display and transmitting the position of the display to the processing apparatus, which is arranged to modify the one or more images generated by the data processing apparatus according to the position of the display to maintain the one or more images generated by the data processing apparatus and the real image in stereoscopic registration.

7. A system according to claim 1 wherein the display is adapted to be mounted on a microscope, the user being able to view the microscope image through the display so as to see the one or more images generated by the data processing apparatus superimposed on the microscope image, the tracking unit monitoring the position of the microscope and transmitting the monitored position of the microscope to the processing apparatus, which is arranged to modify the one or more images generated by the data processing apparatus according to the position of the microscope to maintain the one or more images generated by the data processing apparatus and the real image in stereoscopic registration.

8. The system of claim 1, wherein the interface region is at least one of surrounding the visualization region, adjacent to the visualization region and distant from the visualization region.

9. The system of claim 8, wherein the interface region is within the tracking volume of the tracking unit.

10. The system of claim 1, wherein when the probe is within the visualization region the probe is co-registered to the subject and the icon is displayed in correct relative spatial position to the subject and said generated images of the subject.

11. The system of claim 1, wherein the image guided system is a surgical navigation system and the co-registered images are generated from scans of a human body.

12. A method of controlling an image guided system used in a defined three-dimensional visualization region, said system including a data processing apparatus for generating images of a subject in co-registration with the subject, a display for displaying the images to a user, a probe, and a tracking unit for tracking the location of the probe and transmitting that location to the data processing apparatus, said method comprising:

interacting with the image guided system in a display mode when the probe is within the visualization region, wherein images of the subject are displayed in a main display region; and

interacting with the image guided system in a control mode when the probe is out of the visualization region but within a defined three-dimensional interface region;

wherein, when the probe is within the interface region the data processing apparatus obtains the position of the probe, ceases to generate the co-registered images, and generates in the main display area an image of an icon representing the probe and one or more virtual buttons, each of the virtual buttons being associated with a corresponding instruction to the system,

wherein the user selects one of said buttons by positioning the physical probe in 3D space so as to position the representative icon relative to the apparent position of the button, and wherein the data processing apparatus sets or modifies display parameters for said display mode in response to the instruction corresponding to the selected button.

13. A method according to claim 12 wherein the data processing apparatus generates images of the subject overlaid on the subject.

14. A method according to claim 12 wherein while the data processing apparatus displays the icon and the virtual buttons, it further displays a line extending from said icon, and wherein the positioning of said icon includes aligning said line with the button.

15. A method according to claim 12 wherein the display is mounted on the head of a user, the user can view the subject through the display so as to see the generated images superimposed on a real image of the subject, the tracking unit monitors the position of the display and transmits said display position to the processing apparatus, which modifies the generated images according to the display position to maintain the generated images and the real image in stereoscopic registration.

16. A method according to claim 12 wherein the display is mounted on a microscope, the user can view the microscope image through the display so as to see the one or more images generated by the data processing apparatus superimposed on the microscope image, the tracking unit monitors the position of the microscope and transmits the microscope'"'"'s position to the processing apparatus, which modifies the one or more images generated by the data processing apparatus according to the position of the microscope to maintain the one or more images generated by the data processing apparatus and the microscope image in stereoscopic registration.

17. The method of claim 12, wherein the image guided system is a surgical navigation system and the co-registered images are generated from scans of a human body.