Hybrid image/scene renderer with hands free control

• US 10,056,012 B2
• Filed: 05/24/2013
• Issued: 08/21/2018
• Est. Priority Date: 05/25/2012
• Status: Active Grant

• Current Assignee: Surgical Theater Inc.
• Original Assignee: Surgical Theater LLC
• Inventors: Geri, Alon Yakob, Avisar, Mordechai
• Primary Examiner(s): Utama, Robert J
• Application Number:
  US14/402,746
• Publication Number:
  US 20150140535A1
• Time in Patent Office: 1,915 Days
• Field of Search: 434262
• CPC Class Codes: A61B 17/12 for ligaturing or otherwise compressing tubular parts of the body, e.g. blood vessels, umbilical cord specially adapted for vas deferens or fallopian tubes A61F6/20; materials for ligaturing blood vessels A61L17/00
• CPC Class Codes: A61B 17/1227 Spring clips
• CPC Class Codes: A61B 17/1285 for minimally invasive surgery
• CPC Class Codes: A61B 2017/00707 Dummies, phantoms; Devices simulating patient or parts of patient
• CPC Class Codes: A61B 2034/104 Modelling the effect of the tool, e.g. the effect of an implanted prosthesis or for predicting the effect of ablation or burring
• CPC Class Codes: A61B 34/10 Computer-aided planning, simulation or modelling of surgical operations
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G09B 23/28 for medicine
• CPC Class Codes: G16H 50/50 for simulation or modelling of medical disorders

Abstract

A system and method for converting static/still medical images of a particular patient into dynamic and interactive images interacting with medical tools including medical devices by coupling a model of tissue dynamics and tool characteristics to the patient specific imagery for simulating a medical procedure in an accurate and dynamic manner by coupling a model of tissue dynamics to patient specific imagery for simulating surgery on the particular patient. The method includes a tool to add and/or to adjust the dynamic image of tissues and ability to draw any geometric shape on the dynamic image of tissues and to add the shape into the modeling system.

Claims

1. A modeling system for enabling a user to perform a simulated medical procedure, said system comprising:

one or more computers;

a display for displaying images to the user;

a database for storing physical characteristics of the tissues of a particular patient;

an image generator using one or more of said computers for executing software for generating a dynamic realistic image of the tissues of the particular patient for displaying on said display, wherein said realistic image of the tissues is provided showing an appearance including shadowing and textures indicative of the tissues of the particular patient;

a database comprising a user tool library for providing a plurality of user tool models of actual user tools used in medical procedures;

a user interface for accepting inputs from the user for selecting one of the user tool models;

a user tool generator using one or more of said computers for executing software for generating a realistic tool image of the selected user tool model for displaying on said display;

a user interface for accepting inputs from the user, said inputs for dynamically manipulating said selected user tool image for dynamically interacting with said realistic image of the tissues during the simulated medical procedure for display to the user on said display in real-time; and

a user interface providing a tool to adjust the dynamic image of the tissues displayed on said display by adding or modifying features of said tissues for updating the display of tissues to compensate for anatomical structures that are in the actual biological tissue of the particular patient but are missing from the dynamic image of tissues originally displayed such that the dynamic image of tissues displayed are subsequently displayed on the display as integrated tissue with the added or modified features integrated with the dynamic image of the tissues originally displayed, wherein

dynamic interaction between the user tool image and the image of the tissues, as added or modified, is displayed on said display providing realistic visual features exhibiting realistic mechanical interactions in the tissues including the added or modified tissues, said mechanical interactions based on the stored physical characteristics of the the tissues originally displayed.

2. The modeling system of claim 1, wherein the user interface includes a touchscreen display.

3. The modeling system of claim 1, wherein the tool to adjust the dynamic image of tissues includes a feature to provide the ability to draw any geometric shape on the dynamic image of tissues.

4. The modeling system of claim 1, wherein the tool to adjust the dynamic image of tissues includes a feature to provide the ability to complete an incomplete anatomical structure of the dynamic image of tissues.

5. The modeling system of claim 1, wherein the tool to adjust the dynamic image of tissues includes a feature to provide the ability to modify the texture, lighting, shadow and/or shading of a portion of the dynamic image of tissues.

6. The modeling system of claim 1, wherein said medical images of the particular patient include an image of an aneurysm, and wherein said dynamic image includes an image of the aneurysm, and further wherein said user tool includes an aneurysm clip applier for applying an aneurysm clip model for dynamically interacting with the image of tissues.

7. The modeling system of claim 1, wherein the tool to adjust the dynamic image of tissues includes a feature to provide the ability to command the tool to interact with one or more portions of the dynamic image of tissues.

8. The modeling system of claim 1, wherein the tool to adjust the dynamic image of tissues includes a feature to provide the ability to select elements the tool model and/or the dynamic image of tissues for removal from the displayed image.

9. The modeling system of claim 1, wherein the tool to adjust the dynamic image of tissues includes a feature to provide the ability to reposition objects in the displayed image by selecting the objects and dragging the objects to a desired position for display in the image.

10. The modeling system of claim 1, wherein at least one user interface includes a camera for hands-free operations by the user.

11. A modeling system for performing a surgical simulation, comprising:

a database for storing patient tissue image information that are taken from, or derived from, medical images of a particular patient;

a database for storing standard characteristics of said tissue;

a display;

an image generator for generating a dynamic image of tissues of the particular patient for display on said display, said generating utilizing said patient image information such that said dynamic image of tissues is displayed on said display realistically representing corresponding actual tissues of the particular patient;

a user tool generator for generating a tool model of a user tool for dynamically interacting with said dynamic image of tissues via manipulations provided by a user for display on said display; and

a user interface providing a tool to adjust the dynamic image of tissues displayed on said display by adding or modifying features of said tissues for updating the display of tissues to compensate for anatomical structures that are in the actual biological tissue of the particular patient, but are missing from the dynamic image of tissues originally displayed, for display as integrated tissue such that the dynamic image of tissues displayed are subsequently displayed on the display with the added or modified features integrated with the dynamic image of the tissues originally displayed and having uniform dynamic properties based on the tissues originally displayed, wherein

said tool model is displayed on said display dynamically interacting with said dynamic image of tissues as added or modified for realistically simulating the medical procedure.

12. A modeling system for enabling a user to perform a simulated medical procedure, said system comprising:

one or more computers;

a display for displaying images to the user;

a database for storing characteristics of the tissues of a particular patient;

an image generator using one or more of said computers for executing software for generating a dynamic realistic image of the tissues of the particular patient based on the stored characteristics of the particular patient for displaying on said display, wherein said realistic image of the tissues is provided showing an appearance including shadowing and textures indicative of the tissues of the particular patient;

a database for storing a user tool library for providing a plurality of user tool models of actual user tools used in medical procedures;

a user interface for accepting inputs from the user for selecting one of the user tool models;

a user tool generator using one or more of said computers for executing software for generating a realistic tool image of the selected user tool model for displaying on said display; and

a user interface including a camera for accepting inputs from the user based on motions of the user'"'"'s hands, said inputs for dynamically manipulating said selected user tool image and/or the image of the tissues for dynamically interacting with the realistic image of the tissues during the simulated medical procedure for display to the user on said display in real-time,

a user interface providing a tool to adjust the dynamic image of the tissues displayed on said display by adding or modifying features of said tissues for updating the display of tissues to compensate for anatomical structures that are in the actual biological tissue of the particular patient but are missing from the dynamic image of tissues originally displayed such that the dynamic image of tissues displayed are subsequently displayed on the display as integrated tissue with the added or modified features integrated with the dynamic image of the tissues originally displayed, wherein

dynamic interaction between the user tool image and the image of the tissues is displayed on said display using images with realistic visual features exhibiting realistic mechanical interactions.

13. The modeling system of claim 12, wherein said medical images of the particular patient include an image of an anatomical structure and wherein said dynamic image includes an image of the anatomical structure, and further wherein said user tool includes an instrument for dynamically interacting with the anatomical structure.

14. A method of performing a surgical simulation, comprising the steps of:

providing a computer system;

providing a display connected to said computer device;

obtaining patient image information about the biological tissues of a particular patient for storing in said computer system;

generating, using the computer system, a dynamic image of the biological tissues of the particular patient for display on said display, said generating utilizing said patient image information such that said dynamic image of tissues is displayed on said display realistically representing corresponding actual tissues of the particular patient;

generating, using said computer system, a user tool model for dynamically interacting with said dynamic image of tissues via manipulations input by a user for display on said display;

adjusting, using a user input to said computer system, the dynamic image of tissues displayed on said display by adding or modifying features of said tissues for updating the display of tissues to compensate for anatomical structures that are in the actual biological tissue of the particular patient but are missing from the dynamic image of tissues originally displayed such that the dynamic image of tissues displayed are subsequently displayed on the display as integrated tissue with the added or modified features integrated with the dynamic image of the tissues originally displayed and having uniform dynamic properties based on the tissues originally displayed; and

generating, using the computer system, a realistic simulation of the medical procedure for display on said display showing interactions between the dynamic image of tissues and the user tool model according to inputs by the user.

15. A modeling system for enabling a user to perform a simulated medical procedure, said system comprising:

one or more computers;

a display for displaying images to the user;

an image generator using one or more of said computers for executing software for generating a dynamic realistic image of the tissues for particular patient for displaying on said display, wherein said realistic image of the tissues is provided showing an appearance including shadowing and textures indicative of actual tissues;

a database for storing a user tool library for providing a plurality of user tool models of actual user tools used in medical procedures;

a user interface for accepting inputs from the user for selecting one of the user tool models;

a user tool generator using one or more of said computers for executing software for generating a realistic tool image of the selected user tool model for displaying on said display;

a user interface providing a tool to adjust the dynamic image of the tissues displayed on said display by adding or modifying features of said tissues for updating the display of tissues to compensate for anatomical structures that are in the actual biological tissue of the particular patient but are missing from the dynamic image of tissues originally displayed such that the dynamic image of tissues displayed are subsequently displayed on the display as integrated tissue with the added or modified features integrated with the dynamic image of the tissues originally displayed; and

a user interface that can track the motions of an actual surgical instrument being used by the user with the particular patient, such that said motions of the actual surgical instrument are used for dynamically manipulating said selected user tool image and/or the image of the tissues for dynamically interacting with the realistic image of the tissues during the simulated medical procedure for display to the user on said display in real-time.

16. A modeling system for enabling a user to perform a simulated medical procedure, said

system comprising:

a display;

an image generator for generating a dynamic realistic image of the tissues of the particular patient for displaying on said display, wherein said realistic image of the tissues is provided showing an appearance indicative of the tissues of a patient;

a user tool generator for generating a realistic tool image of the selected user tool model for displaying on said display;

a user interface for accepting inputs from the user, said inputs for dynamically manipulating said selected user tool image for dynamically interacting with said realistic image of the tissues during the simulated medical procedure for display to the user on said display in real-time; and

a user interface providing a tool to adjust the dynamic image of the tissues displayed on said display by adding or modifying features of said tissues for updating the display of tissues to compensate for anatomical structures that are in the actual biological tissue of the patient but are missing from the dynamic image of tissues originally displayed such that the dynamic image of tissues displayed are subsequently displayed as integrated tissue on the display with the added or modified features integrated with the dynamic image of the tissues originally displayed and having uniform dynamic properties based on the tissues originally displayed.

17. The modeling system of claim 16, wherein the user interface includes a touchscreen display.

18. The modeling system of claim 16, wherein the tool to adjust the dynamic image of tissues includes a feature to provide the ability to draw any geometric shape on the dynamic image of tissues.

19. The modeling system of claim 16, wherein the tool to adjust the dynamic image of tissues includes a feature to provide the ability to complete an incomplete anatomical structure of the dynamic image of tissues.

20. The modeling system of claim 16, wherein the tool to adjust the dynamic image of tissues provides the ability to modify the texture, lighting, shadow and/or shading of a portion of the dynamic image of tissues.

21. The modeling system of claim 16, wherein said dynamic image of tissues include an image of an anatomical structure and wherein said user tool includes an instrument for dynamically interacting with the anatomical structure.

22. The modeling system of claim 16, further comprising:

a database for storing a library of a plurality of models of different implants; and

a user interface for selecting one implant model from said plurality of models for use with said user tool model for dynamically interacting with said image of tissues.

23. The modeling system of claim 16, wherein the tool to adjust the dynamic image of tissues includes a feature to provide the ability to command the tool to interact with one or more portions of the dynamic image of tissues.

24. The modeling system of claim 16, wherein the tool to adjust the dynamic image of tissues includes a feature to provide the ability to select elements of a model of the tool and/or the dynamic image of tissues for removal from the displayed image.

25. The modeling system of claim 16, wherein the tool to adjust the dynamic image of tissues includes a feature to provide the ability to reposition objects in the displayed image by selecting the objects and dragging the objects to a desired position for display in the image.

26. The modeling system of claim 16, wherein the tool to adjust the dynamic image of tissues includes a feature to provide the ability to enhance and integrate anatomical structure in the dynamic image.

27. The modeling system of claim 16, wherein the tool to adjust the dynamic image of tissues includes a feature to provide the ability to a draw any geometric shape for adding to the dynamic image of tissues.

28. The modeling system of claim 16, further comprising a camera for providing user input for hands-free operations by the user.