Augmented surgical reality environment

• US 10,607,345 B2
• Filed: 12/05/2018
• Issued: 03/31/2020
• Est. Priority Date: 07/25/2014
• Status: Active Grant

• Current Assignee: Covidien LP (Medtronic Plc)
• Original Assignee: Covidien LP (Medtronic Plc)
• Inventors: Carnes, Tony, McKenna, Edward, Pack, Stephen
• Primary Examiner(s): Pontius, James M
• Application Number:
  US16/210,686
• Publication Number:
  US 20190114768A1
• Time in Patent Office: 482 Days
• CPC Class Codes: A61B 2090/365 augmented reality, i.e. correlating a live optical image with another image
• CPC Class Codes: A61B 2576/00 Medical imaging apparatus involving image processing or analysis A61B1/00009, A61B6/52 and A61B8/52 take precedence; image processing per se G06T; details of algorithms for analysing medical images G06T7/0012
• CPC Class Codes: A61B 90/361 Image-producing devices, e.g. surgical cameras
• CPC Class Codes: A61B 90/37 Surgical systems with images on a monitor during operation
• CPC Class Codes: G06T 11/008 Specific post-processing after tomographic reconstruction, e.g. voxelisation, metal artifact correction
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2207/10048 Infrared image
• CPC Class Codes: G06T 2207/20024 Filtering details
• CPC Class Codes: G06T 2207/20221 Image fusion; Image merging
• CPC Class Codes: G06T 2207/30004 Biomedical image processing
• CPC Class Codes: G06T 5/10 using non-spatial domain filtering
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 7/0012 Biomedical image inspection
• CPC Class Codes: G06T 7/13 Edge detection
• CPC Class Codes: H04N 23/20 for generating image signals from infrared radiation only
• CPC Class Codes: H04N 23/60 Control of cameras or camera modules
• CPC Class Codes: H04N 23/63 by using electronic viewfinders
• CPC Class Codes: H04N 5/33 Transforming infrared radiation cameras or camera modules for generating image signals from infrared radiation H04N23/20; circuitry of SSIS for transforming infrared radiation into image signals H04N25/20

Abstract

The present disclosure is directed to an augmented reality surgical system for viewing an augmented image of a region of interest during a surgical procedure. The system includes an image capture device that captures an image of the region of interest. A controller receives the image and applies at least one image processing filter to the image. The image processing filter includes a spatial decomposition filter that decomposes the image into spatial frequency bands. A temporal filter is applied to the spatial frequency bands to generate temporally filtered bands. An adder adds each band spatial frequency band to a corresponding temporally filtered band to generate augmented bands. A reconstruction filter generates an augmented image by collapsing the augmented bands. A display displays the augmented image to a user.

Claims

1. An augmented reality surgical system for viewing an augmented image of a region of interest, the system comprising:

an image capture device configured to capture an image of the region of interest inside a patient;

a sensor array configured to capture data of a surgical environment;

a controller configured to receive the image and data from the sensor array and apply at least one image processing filter to the image and data to generate the augmented image, the image processing filter including;

a spatial decomposition filter configured to decompose the image into a plurality of spatial frequency bands;

a temporal filter configured to be applied to the plurality of spatial frequency bands to generate a plurality of temporally filtered bands;

an adder configured to add each band in the plurality of spatial frequency bands to a corresponding band in the plurality of temporally filtered bands to generate a plurality of augmented bands; and

a reconstruction filter configured to generate the augmented image by collapsing the plurality of augmented bands; and

a display configured to display the augmented image to a user.

2. The augmented reality surgical system of claim 1, wherein the image capture device captures a video having a plurality of image frames and the controller applies the at least one image processing filter to each image frame of the plurality of image frames.

3. The augmented reality surgical system of claim 1, wherein the temporal filter includes a bandpass filter.

4. The augmented reality surgical system of claim 3, wherein a bandpass frequency of the bandpass filter is set by a clinician.

5. The augmented reality surgical system of claim 1, wherein the temporal filter isolates at least one spatial frequency band from the plurality of spatial frequency bands to generate the plurality of temporally filtered bands.

6. The augmented reality surgical system of claim 1, wherein the plurality of temporally filtered bands are amplified by an amplifier before each band in the plurality of spatial frequency bands is added to the corresponding band in the plurality of temporally filtered bands to generate the plurality of augmented bands.

7. The augmented reality surgical system of claim 1, wherein the image processing filter uses an edge detection algorithm configured to highlight one or more edges in the image,

and the one or more highlighted edges is added to the augmented image.

8. The augmented reality surgical system of claim 1, further comprising at least one hyper-spectral sensor configured to obtain a plurality of hyper-spectral images,

wherein the image processing filter uses a hyper-spectral algorithm to combine the plurality of spectral images to generate a three dimensional hyper-spectral image cube that is added to the augmented image.

9. The augmented reality surgical system of claim 1, further comprising an infrared camera configured to capture at least one image,

wherein the image processing filter generates an infrared image from the captured image, the infrared image is added to the augmented image.

10. A method for generating an augmented image of a region of interest during a surgical procedure, the method comprising:

capturing at least one image of the region of interest using an image capture device;

capturing data of a surgical environment using a sensor array;

combining the at least one image and data of the surgical environment the generate an augmented image;

decomposing the at least one image to generate a plurality of spatial frequency bands;

applying a temporal filter to the plurality of spatial frequency bands to generate a plurality of temporally filtered bands;

adding each band in the plurality of spatial frequency bands to a corresponding band in the plurality of temporally filtered bands to generate a plurality of augmented bands;

collapsing the plurality of augmented bands to generate the augmented image; and

displaying the augmented image on a display.

11. The method of claim 10, further comprising isolating at least one spatial frequency band from the plurality of spatial frequency bands.

12. The method of claim 11, further comprising amplifying the temporally filtered bands before adding each band in the plurality of spatial frequency bands to a corresponding band in the plurality of temporally filtered bands to generate a plurality of augmented bands.

13. The method of claim 10, further comprising:

applying an edge detection algorithm configured to highlight one or more edges in the image; and

adding the one or more highlighted edges to the augmented image.

14. The method of claim 10, further comprising:

obtaining a plurality of hyper-spectral images;

combining the plurality of hyper-spectral images to generate a three dimensional hyper-spectral image cube; and

adding the three dimensional hyper-spectral image cube to the augmented image.

15. The method of claim 10, further comprising:

obtaining an infrared image; and

adding the infrared image to the augmented image.

16. A non-transitory computer readable medium storing instructions that, when executed by a processing device, cause the processing device to:

decompose image data having data from a sensor array including an imperceptible property of an object;

generate a plurality of spatial frequency bands from the decomposed image data;

temporally filter the plurality of spatial frequency bands;

generate a plurality of temporally filtered bands;

add each of the spatial frequency bands to a corresponding band of the temporally filtered bands;

collapse the added bands;

generate an augmented image with improved perceptibility of the imperceptible property from the collapsing of the added bands; and

output the augmented image.

17. The non-transitory computer readable medium of claim 16, further comprising additional stored instructions that, when executed by a processing device, cause the processing device to isolate at least one spatial frequency band containing information about the imperceptible property from the plurality of spatial frequency bands.

18. The non-transitory computer readable medium of claim 17, further comprising additional stored instructions that, when executed by a processing device, cause the processing device to amplify the temporally filtered bands before adding each band in the plurality of spatial frequency bands to a corresponding band in the plurality of temporally filtered bands.

19. The non-transitory computer readable medium of claim 16, further comprising additional stored instructions that, when executed by a processing device, cause the processing device to:

apply an edge detection algorithm configured to highlight one or more edges in the image data, the one or more edges being the imperceptible property; and

add the one or more highlighted edges to the augmented image.

20. The non-transitory computer readable medium of claim 16, further comprising additional stored instructions that, when executed by a processing device, cause the processing device to:

obtain a plurality of hyper-spectral images;

combine the plurality of hyper-spectral images to generate a three dimensional hyper-spectral image cube; and

add the three dimensional hyper-spectral image cube to the augmented image.