Anthro-centric multisensory interface for sensory augmentation of telesurgery

• US 8,880,223 B2
• Filed: 07/16/2012
• Issued: 11/04/2014
• Est. Priority Date: 07/16/2012
• Status: Active Grant

• Current Assignee: Florida Institute For Human And Machine Cognition, Inc.
• Original Assignee: Florida Institute For Human and Machine Cognition
• Inventors: Carif, Roger W., Hutcheson, Timothy L., Raj, Anil K., Moucheboeuf, Adrien M.
• Primary Examiner(s): HOLLOWAY, JASON R
• Application Number:
  US13/549,622
• Publication Number:
  US 20140018819A1
• Time in Patent Office: 841 Days
• US Class Current: 700/264
• CPC Class Codes: A61B 2017/00207 with hand gesture control or hand gesture recognition
• CPC Class Codes: A61B 2034/741 Glove like input devices, e.g. "data gloves"
• CPC Class Codes: A61B 2090/064 for measuring force, pressure or mechanical tension
• CPC Class Codes: A61B 34/25 User interfaces for surgical systems
• CPC Class Codes: A61B 34/30 Surgical robots
• CPC Class Codes: A61B 34/37 Master-slave robots A61B34/35 takes precedence
• CPC Class Codes: A61B 34/70 Manipulators specially adapted for use in surgery
• CPC Class Codes: A61B 34/76 Manipulators having means for providing feel, e.g. force or tactile feedback
• CPC Class Codes: A61B 90/00 Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges protective face masks A41D13/11; surgeon's or patient's gowns or dresses A41D13/12; devices for carrying-off, for treatment of, or for carrying-over, body liquids A61M1/00
• CPC Class Codes: A61B 90/361 Image-producing devices, e.g. surgical cameras
• CPC Class Codes: A61B 90/60 Supports for surgeons, e.g. chairs or hand supports seats for dentists or surgeons associated with dental chairs or operating chairs A61G15/00
• CPC Class Codes: Y10S 901/09 Closed loop, sensor feedback controls arm movement

Abstract

A multisensory interface for a tele-robotic surgical control system. The invention allows the surgeon to use natural gestures and motions to control the actions of end effectors in the robotic surgical apparatus. Multiple feedback mechanisms are provided to allow the physician a more intuitive understanding of what is being controlled, along with a greater situational awareness. Prior art robotic end effectors are inserted into the patient through a small incision—as is already known in the art. The invention presents an improved method of controlling these effectors.

Claims

1. A method for allowing a surgeon to control a robotic surgical apparatus including an end effector, comprising:

a. providing control software running on a computer;

b. providing a gesture capture system, said gesture capture system capturing present location information for at least said surgeon s fingers and forearms;

c. said gesture capture system providing said present location information for at least said surgeon'"'"'s fingers and forearms to said control software;

d. said control software controlling operation of said end effector;

e. said control software using said present location information for at least said surgeon'"'"'s fingers and forearms to control said operation of said end effector;

f. providing a depth-sensing camera system, said camera system providing data sufficient to generate a three dimensional software model of structures within said camera system'"'"'s field of view;

g. using said camera system data to create a three dimensional software model of said structures within said camera system'"'"'s field of view;

h. providing a video display, including a pan function;

i. displaying at least a portion of said three dimensional software model of said structures on said video display;

j. providing a head motion capture system, said head motion capture system capturing present head location information fur said surgeon'"'"'s head; and

k. controlling said pan function on said video display using said head location for information for said surgeon'"'"'s head.

2. A method for allowing a surgeon to control a robotic surgical apparatus as recited in claim 1, further comprising:

a. providing an apparent viewpoint for said depth-sensing camera system; and

b. providing a simulated parallax function as part of said video display, wherein when said pan function is activated, objects that are closer to said apparent viewpoint move a greater distance on said display that objects that are farther away.

3. A method for allowing a surgeon to control a robotic surgical apparatus as recited in claim 1, further comrising:

a. providing a zoom function in said video display; and

b. controlling said zoom function on said video display using said head location capture information for said surgeon'"'"'s head.

4. A method for allowing a surgeon to control a robotic surgical apparatus as recite in claim 2, further comprising:

a. providing a zoom function in said video display; and

b. controlling said zoom function on said video display using said head location capture information for said surgeon'"'"'s head.

5. A method for allowing a surgeon to control a robotic surgical apparatus as recited in claim 1, further comprising:

a. panning said video display to the left when said surgeon'"'"'s head moves left;

b. panning said video display to the right when said surgeon'"'"'s head moves right;

c. panning said video display up when said surgeon'"'"'s head moves down; and

d. panning said video display down when said surgeon'"'"'s head moves up.

6. A method for allowing a surgeon to control a robotic surgical apparatus as recited in claim 2, further comprising:

a. panning said video display to the right when said surgeon'"'"'s head moves left;

b. panning said video display to the left when said surgeon'"'"'s head moves right;

c. panning said video display up when said surgeon'"'"'s head moves down;

d. panning, said video display down when said surgeon'"'"'s head moves up;

e. providing a zoom function in said video display; and

f. controlling said zoom function on said video display using said head location capture information for said surgeon'"'"'s head.

7. A method for allowing a surgeon to control a robotic surgical apparatus as recited in claim 1, further comprising:

a. providing a zoom function in said video display; and

b. initially setting said zoom function so that said end effector'"'"'s display on said monitor is scaled to approximately match a forearm and a hand of said surgeon.

8. A method for allowing a surgeon to control a robotic surgical apparatus as recited in claim 1, further comprising providing specific defined positions for said surgeon'"'"'s fingers which are interpreted by said control software as specific defined commands.

9. A method for allowing a surgeon to control a robotic surgical apparatus as recited in claim 8, wherein a clenched fist is defined as a command to lock a selected effector in position.

10. A method for allowing a surgeon to control a robotic surgical apparatus as recited in claim 1, further comprising:

a. allowing said surgeon to control a second robotic surgical apparatus;

b. wherein said robotic surgical apparatus is associated with a first patient and said second robotic surgical apparatus is associated with a second patient; and

c. allowing said surgeon to toggle control between said robotic surgical apparatus and said second robotic surgical apparatus.

11. A method for allowing a surgeon to control a robotic surgical apparatus as recited in claim 1, thriller comprising:

a. providing a sensory feedback display to said surgeon; and

b. using said sensory feedback display to display a state of said end effector.

12. method for allowing a surgeon to control a robotic surgical apparatus as recited in claim 11, wherein said sensory feedback display is an electro-tactile array.

13. A method for allowing, a surgeon to control a robotic surgical apparatus as recited in claim 11, wherein said sensory feedback display is a vibro-tactile torso vest.

14. A method for allowing a surgeon to control a robotic surgical apparatus as recited in claim 11, wherein said state or said end effector is a position of said end effector.

15. A method for allowing a surgeon to control a robotic surgical apparatus as recited in claim 11, wherein;

a. said end effector has a movable portion; and

b. said state of said end effector is a position of said movable portion.

16. A method for allowing a surgeon to control a robotic surgical apparatus as recited in claim 15, wherein:

a. said end effector is able to sense a force feedback for said movable portion; and

b. said sate of said end effector is an amount of force feedback for said movable portion.

17. A method for allowing a surgeon to control a robotic surgical apparatus including an end effector, comprising:

a. providing control software running on a computer;

b. providing a gesture capture system, said gesture capture system capturing present location information for at least said surgeon'"'"'s fingers and at least one wrist;

c. said gesture capture system providing said present location information for at least said surgeon'"'"'s fingers and said at least one wrist to said control software;

d. providing specific defined Positions for said surgeon'"'"'s fingers which are interpreted by said control software as specificdefined commands;

e. wherein the posititin of said surgeon'"'"'s fingers and said at least one wrist are used by said control software to control the veration of said end effector;

f. providing a depth-sensing camera system, said camera system providing data sufficient to generate a three dimensional software model of structures within said camera system'"'"'s field of view;

g. using said camera system data to create a three dimensional software model of said structures within said camera system'"'"'s field of view;

h. providing a video display;

i. displaying at least a portion of said three dimensional software model of said structures on said video display;

j. providing a pan function for said video display;

k. providing a head motion capture system, said head motion capture system capturing present head location information for said surgeon'"'"'s head; and

l. controlling said pan function on said video display using said head location information for said surgeon'"'"'s head.

18. A method for allowing a surgeon to control a robotic surgical apparatus as recited in claim 17, further comprising:

a. providing an apparent viewpoint for said depth-sensing camera system; and

b. providing a simulated parallax function as part of said video display, wherein when said pan function is activated, objects that are closer to said apparent viewpoint move a greater distance on said display that objects that are farther away.

19. A method for allowing a surgeon to control a robotic surgical apparatus as recited in claim 17, further comprising:

a. providing a zoom function in said video display; and

b. controlling said zoom function on said video display using said head location capture information for said surgeon'"'"'s head.

20. A method for allowing a surgeon to control a robotic surgical apparatus as recited in claim 17, further comprising:

a. panning said video display to the right when said surgeon'"'"'s head moves left;

b. panning said video display to the left when said surgeon'"'"'s head moves right;

c. panning said video display up when said surgeons head moves down; and

d. panning said video display down when said surgeon'"'"'s head moves up.

22. A method for allowing a surgeon to control a robotic surgical apparatus as recited in claim 17, further comprising:

a. providing a zoom function in said video display; and

b. initially setting said zoom function so that said end effector'"'"'s display on said monitor is scaled to approximately match a wrist and a hand of said surgeon.

23. A method for allowing a surgeon to control a robotic surgical apparatus as recited in claim 17, further comprising:

a. providing a zoom function in said video display;

b. initially setting said zoom function so that said end effector'"'"'s display on said monitor is scaled to approximately match a wrist and a band of said surgeon.

24. A method for allowing a surgeon to control a robotic surgical apparatus as recited in claim 17, wherein a clenched first is defined as a command to lock a selected effector in position.

25. A method for allowing as surgeon to control a robotic surgical apparatus as recited in claim 17, further comprising:

a. allowing said surgeon to control a second robotic surgical apparatus;

b. wherein said robotic surgical apparatus is associated with a first patient and said second robotic surgical apparatus is associated with is a second patient; and

c. allowing said surgeon to toggle control between said robotic surgical apparatus and said second robotic surgical apparatus.

26. A method for allowing a surgeon to control a robotic surgical apparatus as recited in claim 17, further comprising:

a. providing a sensory feedback display to said surgeon; and

b. using said sensory feedback display to display a state of said end effector.

27. A method for allowing a surgeon to control a robotic surgical apparatus as recited in claim 26, wherein said sensory feedback display is an electro-tactile array.

28. A method for allowing a surgeon to control a robotic surgical apparatus as recited in claim 26, wherein said sensory feedback display is a vibro-tactile torso vest.

29. A method for allowing a surgeon to control a robotic surgical apparatus as recited in claim 26, wherein said state of said end effector is a position of said end effector.

30. A method for allowing a surgeon to control a robotic surgical apparatus as recited in claim 26, wherein:

a. said end effector has a movable portion; and

b. said state of said end effector is a position of said movable portion.

31. A method for allowing a surgeon to control a robotic surgical apparatus as recited in claim 30, wherein:

a. said end effector is able to sense a force feedback for said movable portion; and

b. said sate of said end effector is an amount of force feedback for said movable portion.

21. A method for allowing a surgeon to control a robotic surgical apparatus as recited in claim l7, further comprising:

a. panning said video display to the right when said surgeon'"'"'s head moves left;

b. panning said video display to the left when said surgeon'"'"'s head moves right;

c. panning said video display up when said surgeons head moves down; and

d. panning said video display down when said surgeon'"'"'s head moves up;

e. providing a zoom function in said video display; and

f. controlling said zoom function on said video display using said head location capture information for said surgeon'"'"'s head.