Environment property estimation and graphical display

• US 9,867,668 B2
• Filed: 04/09/2014
• Issued: 01/16/2018
• Est. Priority Date: 10/20/2008
• Status: Active Grant

• Current Assignee: Johns Hopkins University
• Original Assignee: Johns Hopkins University
• Inventors: Okamura, Allison M., Yamamoto, Tomonori, Vagvolgyi, Balazs P.
• Primary Examiner(s): Luong, Peter
• Application Number:
  US14/248,794
• Publication Number:
  US 20140222025A1
• Time in Patent Office: 1,378 Days
• Field of Search: 600407, 600476-478
• CPC Class Codes: A61B 2034/101 Computer-aided simulation of surgical operations
• CPC Class Codes: A61B 2090/064 for measuring force, pressure or mechanical tension
• CPC Class Codes: A61B 2090/065 for measuring contact or contact pressure
• CPC Class Codes: A61B 2090/371 with simultaneous use of two cameras
• CPC Class Codes: A61B 34/10 Computer-aided planning, simulation or modelling of surgical operations
• CPC Class Codes: A61B 34/30 Surgical robots
• CPC Class Codes: A61B 90/06 Measuring instruments not otherwise provided for
• CPC Class Codes: A61B 90/361 Image-producing devices, e.g. surgical cameras
• CPC Class Codes: A61B 90/37 Surgical systems with images on a monitor during operation
• CPC Class Codes: B25J 9/1671 characterised by simulation, either to verify existing program or to create and verify new program, CAD/CAM oriented, graphic oriented programming systems
• CPC Class Codes: G05B 19/4207 in which a model is traced or scanned and corresponding data recorded
• CPC Class Codes: G05B 2219/35506 Camera images overlayed with graphics, model
• CPC Class Codes: G05B 2219/37065 Map of stiffness, compliance of object
• CPC Class Codes: G05B 2219/37357 Force, pressure, weight or deflection
• CPC Class Codes: G05B 2219/45117 Medical, radio surgery manipulator

Abstract

A surgical robot including an imaging system comprising at least one camera, a processor in communication with the imaging system, a manipulation system in communication with the processor, and a visual display in communication with the processor. The processor is operable to calculate a mechanical property estimate for an area of an environment based on an environment model of tool-environment interaction data, create a composite image comprising a mechanical property map of the mechanical property estimate overlaid on an environment image from the at least one camera, and output the composite image on the visual display.

Claims

1. A method, comprising:

determining, with a processor, tool-environment interaction data from a palpation of an area of an environment;

calculating, with the processor, a mechanical property estimate of the area of the environment based on the tool-environment interaction data;

receiving, with the processor, an environment image;

generating, with the processor, a composite image comprising a mechanical property map of the mechanical property estimate overlaid on the environment; and

outputting the composite image on a visual display.

2. The method of claim 1, wherein the mechanical property map is generated based on translation of the mechanical property estimate to the environment image.

3. The method of claim 1, wherein an area of said mechanical property map overlaid on the environment image corresponds to the area of the environment shown overlaid by said area of said mechanical property map in the environment image.

4. The method of claim 3, wherein said mechanical property map comprises a hue-saturation-luminance colored mechanical property map, wherein the hue corresponds to a mechanical property value for an area of said mechanical property map and the saturation corresponds to a confidence level for the area of said mechanical property map.

5. The method of claim 1, wherein said mechanical property map comprises a blended mechanical property map.

6. The method of claim 1, wherein said tool-environment interaction data comprise an amount of force applied by a manipulation system to the environment.

7. The method of claim 1, wherein said tool-environment interaction data comprises an amount of displacement in the environment caused by a manipulation system.

8. The method of claim 1, wherein the composite image is generated in real time.

9. The method of claim 1, wherein said mechanical property map comprises a 3-D mechanical property map.

10. The method of claim 1, wherein said environment comprises at least one of tissue or an organ.

11. The method of claim 1, wherein said mechanical property comprises stiffness.

12. The method of claim 1, wherein the mechanical property estimate of the area of the environment is calculated based on an environmental model of the tool-environment interaction data.

13. The method of claim 12, wherein said environment model comprises a linear or nonlinear equation using the mechanical property of the environment to relate force and movement variables.

14. The method of claim 12, wherein the mechanical property estimate is calculated using an algorithm including the environment model and said tool-environment interaction data.