Surgery guidance system

• US 10,639,104 B1
• Filed: 11/07/2014
• Issued: 05/05/2020
• Est. Priority Date: 11/07/2014
• Status: Active Grant

• Current Assignee: Verily Life Sciences LLC (Alphabet Inc.)
• Original Assignee: Verily Life Sciences LLC (Alphabet Inc.)
• Inventors: Barral, Joelle K., Hannaford, Blake, Rephaeli, Eden, Ching, Christine Denise
• Primary Examiner(s): Peng, Bo Joseph
• Application Number:
  US14/536,271
• Time in Patent Office: 2,006 Days
• CPC Class Codes: A61B 1/04 combined with photographic or television appliances
• CPC Class Codes: A61B 2034/2057 Details of tracking cameras
• CPC Class Codes: A61B 2034/2074 Interface software
• CPC Class Codes: A61B 2034/252 indicating steps of a surgical procedure
• CPC Class Codes: A61B 2090/363 Use of fiducial points
• CPC Class Codes: A61B 2090/365 augmented reality, i.e. correlating a live optical image with another image
• CPC Class Codes: A61B 2090/366 using projection of images directly onto the body
• CPC Class Codes: A61B 2090/371 with simultaneous use of two cameras
• CPC Class Codes: A61B 2090/372 Details of monitor hardware
• CPC Class Codes: A61B 2090/376 using X-rays, e.g. fluoroscopy
• CPC Class Codes: A61B 2090/378 using ultrasound
• CPC Class Codes: A61B 2090/392 Radioactive markers
• CPC Class Codes: A61B 2090/502 Headgear, e.g. helmet, spectacles
• CPC Class Codes: A61B 34/20 Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
• CPC Class Codes: A61B 34/25 User interfaces for surgical systems
• CPC Class Codes: A61B 90/36 Image-producing devices or illumination devices not otherwise provided for
• CPC Class Codes: A61B 90/37 Surgical systems with images on a monitor during operation
• CPC Class Codes: A61B 90/50 Supports for surgical instruments, e.g. articulated arms

Abstract

Image guided surgery includes capturing a primary modality image of a surgical field of a patient with a camera system, obtaining a secondary modality image of the surgical field registered to the primary image, generating a surgical guidance image based at least in part upon the secondary modality image, and projecting the surgical guidance image onto the patient. The surgical guidance image presents visual augmentations on or over the patient to inform a medical practitioner during a surgical procedure.

Claims

1. A method for image-guided surgery, comprising:

capturing a primary modality image of a real surgical field with a camera system, wherein the camera system includes an endoscope camera to capture at least the primary modality image as an internal image of a patient;

obtaining a secondary modality image of the real surgical field registered to the primary modality image;

tracking a position of the endoscope camera within a body cavity of the patient;

generating a surgical guidance image based at least in part upon the secondary modality image, wherein the surgical guidance image indicates if the secondary modality image ceases to be registered to the primary modality image, by discontinuing the secondary modality image or by providing a separate warning within the surgical guidance image; and

projecting the surgical guidance image from a projector onto the real surgical field of the patient, wherein the surgical guidance image presents visual augmentations directly over one or more anatomical structures in the real surgical field to inform a medical practitioner during a surgical procedure, wherein the surgical guidance image includes a visual indication, projected on or over the patient, of the position of the endoscope camera within the body cavity of the patient, wherein the surgical guidance image is projected with an identification overlay that automatically identifies and labels the one or more anatomical structures based upon registration, by a controller communicatively coupled to the camera system, of the secondary modality image to the primary modality image.

2. The method of claim 1, wherein the primary modality image comprises at least one of a color image or a grayscale image captured in real time during the surgical procedure by the camera system.

3. The method of claim 1, further comprising:

capturing the secondary modality image with the camera system in real time during the surgical procedure by the camera system, wherein the secondary modality image is different from the primary modality image and comprises at least one of a fluorescence image from a fluorescent dye, an ultrasound image, a hyper-spectral image, a speckle image, an x-ray image, a Cherenkov image, or a radiation image from a radioactive tracer.

4. The method of claim 1, wherein the secondary modality image comprises a pre-operative image of the real surgical field acquired prior to the surgical procedure using a second imaging modality that is different than a first imaging modality used to capture the primary modality image.

5. The method of claim 1, further comprising:

registering multiple secondary modality images of the real surgical field to the primary modality image, wherein each of the multiple secondary modality images is acquired using a different imaging modality;

generating multiple surgical guidance images based at least in part upon the multiple secondary modality images; and

projecting the multiple surgical guidance images, each with a different color or a distinct appearance, onto the real surgical field of the patient.

6. The method of claim 1, wherein the capturing, the generating, and the projecting are performed in real time during the surgical procedure such that the surgical guidance image tracks movements of the one or more anatomical structures.

7. The method of claim 1,

wherein a location of the visual indication of the position of the endoscope camera within the body cavity of the patient moves within the surgical guidance image based at least in part upon the position of the endoscope camera within the body cavity of the patient.

8. The method of claim 1, further comprising:

identifying boundaries of the one or more anatomical structures in the real surgical field, wherein the identification overlay comprises shape traces of the one or more anatomical structures to visually distinguish the boundaries of the one or more anatomical structures.

9. The method of claim 1, wherein the identification overlay comprises annotations for labeling the one or more anatomical structures.

10. The method of claim 9, further comprising:

projecting the surgical guidance image with an annotation interface; and

recognizing voice or gesture commands for annotating the one or more anatomical structures with reference to the annotation interface.

11. The method of claim 1, further comprising:

projecting the surgical guidance image with one or more visual guidance cues while the medical practitioner is performing the surgical procedure, wherein the one or more visual guidance cues are selected from a catalog of visual guidance cues indexed to surgical procedures.

12. The method of claim 1, wherein the position of the endoscope camera is at least partially determined by registering the primary modality image with the secondary modality image in real time.

13. A surgical guidance system, comprising:

a camera system repositionable relative to a patient, wherein the camera system includes an endoscope camera;

a tracking system configured to locate and track a position of the endoscope camera within a body cavity of the patient;

a projector repositionable relative to the patient; and

a controller communicatively coupled to the camera system, the tracking system, and the projector, wherein the controller includes one or more processors and logic that, when executed by the one or more processors cause the surgical guidance system to perform operations including;

capturing a primary modality image of a real surgical field of a patient with the camera system, wherein the primary modality image includes as an internal image of the patient;

obtaining a secondary modality image of the real surgical field registered to the primary modality image;

generating a surgical guidance image based at least in part upon the secondary modality image and the position of the endoscope camera within the body cavity of the patient; and

projecting the surgical guidance image from the projector onto the patient, wherein the surgical guidance image indicates, by discontinuing the secondary modality image or by providing a separate warning within the surgical guidance image, if the secondary modality image ceases to be registered to the primary modality image, wherein the surgical guidance image presents visual augmentations over the patient to inform a medical practitioner during a surgical procedure, wherein the surgical guidance image includes a visual indication projected on or over the patient of a location of the endoscope camera within the body cavity of the patient, wherein a position of the visual indication moves within the surgical guidance image based at least in part upon the location of the endoscope camera within the body cavity of the patient.

14. The surgical guidance system of claim 13, wherein at least a portion of the camera system and the projector are mounted to one or more armatures that are repositionable over an operating table.

15. The surgical guidance system of claim 13, wherein one or both of a portion of the camera system or the projector system are head mountable for wearing on a head.

16. The surgical guidance system of claim 15, wherein the projector system is integrated into a see-through head wearable display.

17. The surgical guidance system of claim 13, wherein the surgical guidance image is projected with an identification overlay that automatically identifies and labels one or more anatomical structures based upon registration of the secondary modality image to the primary modality image.

18. The surgical guidance system of claim 13, wherein the capturing, the generating, and the projecting are performed in real time during the surgical procedure such that the surgical guidance image tracks movements of one or more anatomical structures.

19. The surgical guidance system of claim 13, wherein the operations further comprise:

projecting the surgical guidance image directly on one or more anatomical structures with an identification overlay that identifies the one or more anatomical structures within the real surgical field.

20. The surgical guidance system of claim 19, wherein the operations further comprise:

identifying boundaries of the one or more anatomical structures in the real surgical field, wherein the identification overlay comprises shape traces of the one or more anatomical structures to visually distinguish the boundaries of the one or more anatomical structures.

21. The surgical guidance system of claim 19, wherein the identification overlay comprises annotations for labeling the one or more anatomical structures.

22. The surgical guidance system of claim 21, wherein the operations further comprise:

projecting the surgical guidance image with an annotation interface; and

recognizing voice or gesture commands for annotating the one or more anatomical structures with reference to the annotation interface.

23. The surgical guidance system of claim 13, wherein the operations further comprise:

projecting the surgical guidance image with one or more visual guidance cues while the medical practitioner is performing the surgical procedure, wherein the one or more visual guidance cues are selected from a catalog of visual guidance cues indexed to surgical procedures.

24. A method for image-guided surgery, comprising:

capturing a primary modality image of a real surgical field with a camera system, wherein the camera system includes an endoscope camera to capture at least the primary modality image as an internal image of a patient;

obtaining a secondary modality image of the real surgical field registered to the primary modality image;

tracking a position of the endoscope camera within a body cavity of the patient;

generating a surgical guidance image based at least partially upon the secondary modality image; and

projecting the surgical guidance image from a projector onto the real surgical field of the patient,

wherein the surgical guidance image presents visual augmentations directly over one or more anatomical structures in the real surgical field to inform a medical practitioner during a surgical procedure, wherein the surgical guidance image includes a visual indication, projected on or over the patient, of a location of the endoscope camera within the body cavity of the patient, wherein the surgical guidance image also includes the internal image from the endoscope camera projected on the real surgical field, wherein a position of the internal image from the endoscope camera moves within the surgical guidance image based at least in part upon the position of the endoscope camera within the body cavity of the patient, and wherein the surgical guidance image is projected with an identification overlay that automatically identifies and labels the one or more anatomical structures based upon registration of the secondary modality image to the primary modality image.

25. The method of claim 24, wherein the surgical guidance image indicates if the secondary modality image ceases to be registered to the primary modality image, by discontinuing the secondary modality image or by providing a separate warning within the surgical guidance image.

26. The method of claim 24, wherein the position of the endoscope camera is used to modify at least one display parameter of the internal image projected onto the patient'"'"'s skin.