Environment property estimaton and graphical display

• US 8,700,123 B2
• Filed: 10/20/2009
• Issued: 04/15/2014
• Est. Priority Date: 10/20/2008
• Status: Active Grant

• Current Assignee: Johns Hopkins University
• Original Assignee: Johns Hopkins University
• Inventors: Okamura, Allison Mariko, Yamamoto, Tomonori, Vagvolgyi, Peter Balazs
• Primary Examiner(s): Luong, Peter
• Application Number:
  US13/124,350
• Publication Number:
  US 20110201885A1
• Time in Patent Office: 1,638 Days
• Field of Search: 600/407, 600/476
• US Class Current: 600/407
• CPC Class Codes: A61B 2034/101 Computer-aided simulation of surgical operations
• CPC Class Codes: A61B 2090/064 for measuring force, pressure or mechanical tension
• CPC Class Codes: A61B 2090/065 for measuring contact or contact pressure
• CPC Class Codes: A61B 2090/371 with simultaneous use of two cameras
• CPC Class Codes: A61B 34/10 Computer-aided planning, simulation or modelling of surgical operations
• CPC Class Codes: A61B 34/30 Surgical robots
• CPC Class Codes: A61B 90/06 Measuring instruments not otherwise provided for
• CPC Class Codes: A61B 90/361 Image-producing devices, e.g. surgical cameras
• CPC Class Codes: A61B 90/37 Surgical systems with images on a monitor during operation
• CPC Class Codes: B25J 9/1671 characterised by simulation, either to verify existing program or to create and verify new program, CAD/CAM oriented, graphic oriented programming systems
• CPC Class Codes: G05B 19/4207 in which a model is traced or scanned and corresponding data recorded
• CPC Class Codes: G05B 2219/35506 Camera images overlayed with graphics, model
• CPC Class Codes: G05B 2219/37065 Map of stiffness, compliance of object
• CPC Class Codes: G05B 2219/37357 Force, pressure, weight or deflection
• CPC Class Codes: G05B 2219/45117 Medical, radio surgery manipulator

Abstract

A surgical robot including an imaging system comprising at least one camera, a processor in communication with the imaging system, a manipulation system in communication with the processor, and a visual display in communication with the processor. The processor is operable to calculate a stiffness estimate for an area of an environment based on an environment model of tool-environment interaction data, create a composite image comprising a stiffness map of the stiffness estimate overlaid on an environment image from the at least one camera, and output the composite image on the visual display.

Claims

1. A surgical robot, comprising:

an imaging system comprising at least one camera;

a processor in communication with said imaging system;

a manipulation system in communication with said processor; and

a visual display in communication with said processor,

wherein said processor is operable to;

calculate a mechanical property estimate for an area of an environment based on an environment model of tool-environment interaction data;

create a composite image comprising a mechanical property map of the mechanical property estimate overlaid on an environment image from said at least one camera; and

output the composite image on said visual display.

2. The surgical robot of claim 1, wherein said processor is operable to create the mechanical property map based on translation of the mechanical property estimate to the environment image.

3. The surgical robot of claim 1, wherein an area of said mechanical property map overlaid on the environment image corresponds to the area of the environment shown overlaid by said area of said mechanical property map in the environment image.

4. The surgical robot of claim 3, wherein said mechanical property map comprises a hue-saturation-luminence colored mechanical property map, wherein the hue corresponds to a mechanical property value for an area of said mechanical property map and the saturation corresponds to a confidence level for the area of said mechanical property map.

5. The surgical robot of claim 1, wherein said mechanical property map comprises a blended mechanical property map.

6. The surgical robot of claim 1, wherein said manipulation system further comprises a tool-environment interaction sensor.

7. The surgical robot of claim 6, wherein said tool-environment interaction sensor comprises a force sensor, wherein said tool-environment interaction data comprise an amount of force applied by said manipulation system to the environment.

8. The surgical robot of claim 1, wherein said tool-environment interaction data comprise an amount of displacement in the environment caused by said manipulation system.

9. The surgical robot of claim 1, wherein said processor is further operable to create the composite image in real time.

10. The surgical robot of claim 1, wherein said environment model comprises a linear or nonlinear equation using the mechanical property of the environment to relate force and movement variables.

11. The surgical robot of claim 1, wherein said processor is further operable to apply an algorithm to estimate the mechanical property based on the environment model and said tool-environment interaction data.

12. The surgical robot of claim 1, wherein said mechanical property map comprises a surface mechanical property map embedded in three-dimensional space.

13. The surgical robot of claim 1, wherein said environment comprises at least one of tissue or an organ.

14. The surgical robot of claim 1, wherein said mechanical property comprises stiffness.

15. A data processing unit for use with a surgical robot, comprising:

at least one input port adapted to receive an environment image from the surgical robot and to receive tool-environment interaction data from the surgical robot;

an overlay component in communication with said at least one input port; and

an output port in communication with said overlay component,

wherein said overlay component is adapted to calculate a mechanical property estimate for an area of an environment based on an environment model of tool-environment interaction data, create a composite image comprising a mechanical property map of the mechanical property estimate overlaid on the environment image, and output the composite image to said output port.

16. The data processing unit of claim 15, wherein said overlay component is operable to create the mechanical property map based on translation of the mechanical property estimate to the environment image.

17. The data processing unit of claim 15, wherein an area of said mechanical property map overlaid on the environment image corresponds to the area of the environment shown overlaid by said area of said mechanical property map in the environment image.

18. The data processing unit of claim 17, wherein said mechanical property map comprises a hue-saturation-luminesce colored mechanical property map, wherein the hue corresponds to a mechanical property value for an area of said mechanical property map and the saturation corresponds to a confidence level for the area of said mechanical property map.

19. The data processing unit of claim 15, wherein said mechanical property map comprises a blended mechanical property map.

20. The data processing unit of claim 15, wherein said tool-environment interaction data comprise an amount of force applied by a manipulation system to the environment.

21. The data processing unit of claim 15, wherein said tool-environment interaction data comprises an amount of displacement in the environment caused by a manipulation system.

22. The data processing unit of claim 15, wherein said overlay component is further adapted to create the composite image in real time.

23. The data processing unit of claim 15, wherein said environment model comprises a linear or nonlinear equation using the mechanical property of the environment to relate force and movement variables.

24. The data processing unit of claim 15, wherein said overlay component is further adapted to apply an algorithm to estimate the mechanical property using the environment model and said tool-environment interaction data.

25. The data processing unit of claim 15, wherein said mechanical property map comprises a 3-D mechanical property map.

26. The data processing unit of claim 15, wherein said environment comprises at least one of tissue or an organ.

27. The data processing unit of claim 15, wherein said mechanical property comprises stiffness.

28. A tangible machine readable storage medium that provides instructions, which when executed by a computing platform, cause said computing platform to perform operations comprising a method comprising:

determining tool-environment interaction data from a palpation of an area of an environment;

calculating a mechanical property estimate of the area of the environment based on the tool-environment interaction data;

receiving an environment image;

generating a composite image comprising a mechanical property map of the mechanical property estimate overlaid on the environment; and

outputting the composite image on a visual display.