AUGMENTED REALITY APPARATUS

• US 20140022283A1
• Filed: 07/22/2013
• Published: 01/23/2014
• Est. Priority Date: 07/20/2012
• Status: Abandoned Application

• Current Assignee: University Health Network
• Original Assignee: University Health Network
• Inventors: Chan, Harley Hau-Lam, Daly, Michael John, IRISH, Jonathan Crawford
• Application Number:
  US13/947,263
• Publication Number:
  US 20140022283A1
• Time in Patent Office: Days
• US Class Current: 345/633
• CPC Class Codes: A61B 2034/2055 Optical tracking systems
• CPC Class Codes: A61B 2090/363 Use of fiducial points
• CPC Class Codes: A61B 2090/366 using projection of images directly onto the body
• CPC Class Codes: A61B 2090/372 Details of monitor hardware
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/147 using display panels
• CPC Class Codes: G09G 2340/12 Overlay of images, i.e. displayed pixel being the result of switching between the corresponding input pixels
• CPC Class Codes: G09G 2354/00 Aspects of interface with display user
• CPC Class Codes: G09G 2380/08 Biomedical applications
• CPC Class Codes: G09G 3/002 to project the image of a two-dimensional display, such as an array of light emitting or modulating elements or a CRT
• CPC Class Codes: G09G 5/377 for mixing or overlaying two or more graphic patterns G09G5/02, G09G5/397 take precedence
• CPC Class Codes: G16H 30/20 for handling medical images, e.g. DICOM, HL7 or PACS
• CPC Class Codes: H04N 9/3185 Geometric adjustment, e.g. keystone or convergence optical or mechanical adjustment of convergence H04N9/317; using scanning means H04N3/22; optical or mechanical adjustments of projectors not peculiar to the presence of an electronic spatial light modulator G03B21/14

Abstract

Augmented reality apparatus (10) for use during intervention procedures on a trackable intervention site (32) are disclosed herein. The apparatus (10) comprises a data processor (28), a trackable projector (14) and a medium (30) including machine-readable instructions executable by the processor (28). The projector (14) is configured to project an image overlaying the intervention site (32) based on instructions from the data processor (28). The machine readable instructions are configured to cause the processor (28) to determine a spatial relationship between the projector (14) and the intervention site (32) based on a tracked position and orientation of the projector (14) and on a tracked position and orientation of the intervention site (32). The machine readable instructions are also configured to cause the processor to generate data representative of the image projected by the projector based on the determined spatial relationship between the projector and the intervention site.

Claims

1. An augmented reality apparatus for use during an intervention procedure on a trackable intervention site of a subject, the apparatus comprising:

at least one data processor;

at least one trackable projector in communication with the at least one data processor and configured to project an image overlaying the intervention site based on instructions from the at least one data processor; and

a medium including machine-readable instructions executable by the at least one processor and configured to cause the at least one processor to;

determine a spatial relationship between the trackable projector and the trackable intervention site based on a tracked position and orientation of the projector and on a tracked position and orientation of the intervention site;

access stored image data associated with the intervention site;

using the stored image data, generate data representative of the image to be projected by the trackable projector based on the determined spatial relationship between the trackable projector and the trackable intervention site; and

instruct the trackable projector to project the image.

2. The augmented reality apparatus as defined in claim 1, wherein the stored image data comprises data associated with a plurality of imaging modalities.

3. The augmented reality apparatus as defined in claim 1, wherein the projected image is based on a combination of data from plurality of imaging modalities.

4. The augmented reality apparatus as defined in claim 3, wherein the imaging modalities comprise at least two of computed tomography (CT), magnetic resonance imaging (MRI), cone beam computed tomography (CBCT), positron emission tomography (PET), single-photon emission computed tomography (SPECT), ultrasound and optical images.

5. The augmented reality apparatus as defined in claim 1, comprising a movable implement where the at least one processor is caused to generate data representative of the image to be projected by the trackable projector based on a position of the movable implement.

6. The augmented reality apparatus as defined in claim 5, wherein the movable implement comprises a tracked handheld pointer.

7. The augmented reality apparatus as defined in claim 5, wherein the position of the movable implement is used to select a depth of the intervention site and the image to be projected is based on stored image data at the selected depth.

8. The augmented reality apparatus as defined in claim 1, wherein the at least one processor is caused to generate data representative of the image to be projected by the trackable projector based on a tracked position of a user.

9. The augmented reality apparatus as defined in claim 1, wherein the at least one processor is caused to generate data representative of the image to be projected by the trackable projector based on a tracked position and orientation of a head of a user.

10. The augmented reality apparatus as defined in claim 1, comprising a translucent screen onto which the projector projects the image.

11. The augmented reality apparatus as defined in claim 10, wherein the translucent screen and the trackable projector are in fixed spatial relationship to one another and form a mobile handheld unit.

12. The augmented reality apparatus as defined in claim 9, wherein the at least one processor is caused to generate data representative of a proximity alert when a tracked position of a tool is proximate or within a region of interest in the stored image data.

13. An augmented reality apparatus comprising:

at least one data processor;

at least one trackable projector in communication with the at least one data processor and configured to project a medical image overlaying a trackable intervention site based on instructions from the at least one data processor;

at least one tracking device configured to;

track the position and orientation of the trackable projector;

track the position and orientation of the trackable intervention site; and

provide data representative of the position and orientation of the trackable projector and of the trackable intervention site to the data processor; and

a medium including machine-readable instructions executable by the at least one processor and configured to cause the at least one processor to;

determine a spatial relationship between the trackable projector and the trackable intervention site based on the tracked position and orientation of the projector and of the tracked position and orientation of the intervention site;

access volumetric medical image data associated with the intervention site;

using the volumetric medical image data, generate data representative of the medical image to be projected by the trackable projector based on the determined spatial relationship between the trackable projector and the trackable intervention site; and

instruct the trackable projector to project the medical image.

14. The augmented reality apparatus as defined in claim 13, comprising a translucent screen onto which the trackable projector projects the medical image, the translucent screen and the trackable projector forming a mobile handheld unit.

15. The augmented reality apparatus as defined in claim 14, wherein the at least one tracking device is configured to track a position of a head or an eye of a user and the at least one processor is caused to generate data representative of the image to be projected by the trackable projector based on the tracked position of the head or eye of the user.

16. The augmented reality apparatus as defined in claim 13, wherein the volumetric image data comprises data associated with a plurality of imaging modalities comprising at least two of computed tomography (CT), magnetic resonance imaging (MRI), cone beam computed tomography (CBCT), positron emission tomography (PET), single-photon emission computed tomography (SPECT), ultrasound and optical images.

17. The augmented reality apparatus as defined in claim 16, wherein the projected image is based on a combination of the plurality of imaging modalities.

18. The augmented reality apparatus as defined in claim 13, comprising a movable implement and the at least one processor is caused to generate data representative of the image to be projected by the trackable projector based on a position of the movable implement.

19. The augmented reality apparatus as defined in claim 18, wherein the position of the movable implement is used to select a region of the intervention site and the image to be projected is based on volumetric medical image data at the selected region.

20. The augmented reality apparatus as defined in claim 18, wherein the at least one processor is caused to select a slice of volumetric medical image data to be used as a basis for the projected image based on the position of the movable implement.