Medical devices, systems, and methods using eye gaze tracking

• US 10,278,782 B2
• Filed: 03/18/2015
• Issued: 05/07/2019
• Est. Priority Date: 03/19/2014
• Status: Active Grant

• Current Assignee: Intuitive Surgical Operations, Inc. (Intuitive Surgical Holdings, LLC)
• Original Assignee: Intuitive Surgical Operations, Inc. (Intuitive Surgical Holdings, LLC)
• Inventors: Jarc, Anthony Michael, Lin, Henry C., Sorger, Jonathan Michael
• Primary Examiner(s): Sample, Jonathan L
• Application Number:
  US15/126,074
• Publication Number:
  US 20170172675A1
• Time in Patent Office: 1,511 Days
• Field of Search: 700245-264
• CPC Class Codes: A61B 1/00193 adapted for stereoscopic vision
• CPC Class Codes: A61B 17/00234 for minimally invasive surgery A61B17/0218, A61B17/0469, A61B17/12013, A61B17/1285, A61B17/29, A61B17/320016 take precedence
• CPC Class Codes: A61B 2017/00216 with eye tracking or head position tracking control
• CPC Class Codes: A61B 2090/371 with simultaneous use of two cameras
• CPC Class Codes: A61B 2090/372 Details of monitor hardware
• CPC Class Codes: A61B 34/35 for telesurgery
• CPC Class Codes: A61B 34/71 Manipulators operated by drive cable mechanisms
• CPC Class Codes: A61B 90/361 Image-producing devices, e.g. surgical cameras
• CPC Class Codes: A61B 90/37 Surgical systems with images on a monitor during operation
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G16H 20/40 relating to mechanical, radiation or invasive therapies, e.g. surgery, laser therapy, dialysis or acupuncture
• CPC Class Codes: G16H 40/67 for remote operation

Abstract

Described herein is a teleoperational medical system for performing a medical procedure in a surgical field. The teleoperational system comprises an eye tracking unit and a control unit. The eye tracking unit includes an image display configured to display to a user an image of the surgical field, at least one eye tracker configured to measure data about a gaze point of the user, and a processor configured to process the data to determine a viewing location in the displayed image at which the gaze point of the user is directed. The control unit is configured to control at least one function of the teleoperational medical system based upon the determined viewing location.

Claims

1. A teleoperational medical system for performing a medical procedure in a surgical field, the system comprising:

an eye tracking unit including;

an image display configured to display to a user an image of the surgical field;

at least one eye tracker configured to measure data about a gaze point of the user;

a processor configured to process the data to determine a viewing location in the displayed image at which the gaze point of the user is directed; and

a control unit configured to control at least one function of the teleoperational medical system based upon the determined viewing location.

2. The teleoperational medical system of claim 1 wherein the image display is a 3D image display configured to display to the user a 3D image of the surgical field.

3. The teleoperational medical system of claim 1 further comprising a surgical instrument, wherein the control unit is configured to control at least one function of the surgical instrument in the surgical field based upon the determined viewing location in the displayed image.

4. The teleoperational medical system of claim 3 wherein the surgical instrument is displayed in the displayed image at the viewing location.

5. The teleoperational medical system of claim 3, wherein the control unit is configured to calibrate a 3D position of the surgical instrument in the surgical field based on the determined viewing location in the displayed image.

6. The teleoperational medical system of claim 3 wherein the surgical instrument is an energy application instrument and the control unit is configured to operate the energy application instrument to deliver energy within the surgical field based upon the determined viewing location.

7. The teleoperational medical system of claim 3 wherein the surgical instrument is a fastener application instrument and the control unit is configured to operate the fastener application instrument to deliver a fastener within the surgical field based upon the determined viewing location.

8. The teleoperational medical system of claim 3 wherein the surgical instrument is a fluid management instrument and the control unit is configured to operate the fluid management instrument to convey fluid within the surgical field based upon the determined viewing location.

9. The teleoperational medical system of claim 3, wherein the control unit is configured to deactivate the surgical instrument if the processor determines that the viewing location does not match a position of the surgical instrument for a predetermined length of time.

10. The teleoperational medical system of claim 3, wherein the control unit is configured to move the surgical instrument within the surgical field based on the determined viewing location in the displayed image.

11. The teleoperational medical system of claim 3, wherein the control unit is configured to transfer the control of the surgical instrument from the user to a second user if a determined viewing location in the displayed image of the second user matches a position of the surgical instrument in the image display for a predetermined length of time.

12. The teleoperational medical system of claim 1 further comprising a marking application, wherein the control unit is configured to control the marking application to make a virtual mark in the surgical field that is visible on the image display, based upon the determined viewing location.

13. The teleoperational medical system of claim 1 further comprising an imaging device wherein the control unit is configured to control at least one function of the imaging device in the surgical field based upon the determined viewing location.

14. The teleoperational medical system of claim 13 wherein the at least one function of the imaging device includes movement of the imaging device.

15. The teleoperational medical system of claim 13 wherein the at least one function of the imaging device includes a zoom operation of the imaging device.

16. The teleoperational medical system of claim 13 wherein the at least one function of the imaging device includes a focusing operation of the imaging device.

17. The teleoperational medical system of claim 13 wherein the at least one function of the imaging device includes a mode change operation of the imaging device.

18. The teleoperational medical system of claim 13, wherein the at least one function controlled by the control unit based upon the determined viewing location includes a user log-in function.

19. The teleoperational medical system of claim 1, wherein the control unit is configured to lock out the user from the teleoperational medical system if the processor fails to determine the viewing location on the image display.

20. The teleoperational medical system of claim 1, wherein the control unit is configured to lock out the user from the teleoperational medical system if the at least one eye tracker fails to detect eyes of the user.

21. The teleoperational medical system of claim 1, wherein the image display is configured to display to the user an image of a user interface comprising a plurality of functional options.

22. The teleoperational medical system of claim 21, wherein the control unit is configured to initiate at least one of the plurality of functional options if the determined viewing location matches a position of at least one functional option of the plurality of functional options in the displayed image of the user interface.

23. The teleoperational medical system of claim 21, wherein the control unit is configured to adjust a positional depth of the user interface in the displayed image based on the determined viewing location.

24. The teleoperational medical system of claim 1, wherein the processor is configured to generate an evaluation image of the viewing location at which the gaze point of the user is directed, on an evaluation image display.

25. The teleoperational medical system of claim 1, wherein the image display is configured to display to the user a gaze point of another user viewing the surgical field.

26. The teleoperational medical system of claim 1, wherein the processor is configured to receive an instruction input from the user and to adjust the displayed image based upon the instruction input.

27. The teleoperational medical system of claim 26 wherein the processor is configured to adjust the displayed image by performing a zoom function.

28. The teleoperational medical system of claim 26 wherein the processor is configured to adjust the displayed image by displaying a secondary image of the viewing location, as an overlay, on the image display based upon the gaze point of the user.

29. The teleoperational medical system of claim 28, further comprising a first imaging module and a second imaging module, wherein the first imaging module is configured to obtain the displayed image and the second imaging module is configured to capture the secondary image based upon the gaze point of the user, the first imaging module and the second imaging module being independently controlled.

30. A method for operating a teleoperational medical system, the method comprising:

displaying an image, including a surgical field image of a surgical field, on an image display;

measuring a gaze point of a user in the image display;

determining a viewing location in the displayed image at which the gaze point of the user is directed; and

controlling at least one function of the teleoperational medical system based upon the determined viewing location.

31. The method of claim 30 wherein the at least one function of the teleoperational medical system is a three dimensional calibration function that includes comparing the viewing location with a predetermined target in the displayed image with known three dimensional location parameters.

32. The method of claim 30 wherein the at least one function of the teleoperational medical system is an evaluation function that includes displaying the viewing location to another user.

33. The method of claim 30 further comprising providing a teaching location in the displayed image at which a gaze point of another user is directed.

34. The method of claim 30 wherein displayed surgical field image is a 3D image.

35. The method of claim 30 wherein the at least one function of the teleoperational medical system is a surgical instrument operation function including controlling at least one function of a surgical instrument in the surgical field based upon the determined viewing location in the displayed image.

36. The method of claim 35 wherein the surgical instrument is displayed in the surgical field image at the viewing location.

37. The method of claim 34 wherein the at least one function of the teleoperational medical system is an instrument operation function.

38. The method of claim 37 wherein the instrument operation function includes operating an energy application instrument to deliver energy at a location within the surgical field based upon the determined viewing location.

39. The method of claim 37 wherein the instrument operation function includes operating a fastener application instrument to deliver a fastener at a location within the surgical field based upon the determined viewing location.

40. The method of claim 37 wherein the instrument operation function includes operating a fluid management instrument to convey fluid at a location within the surgical field based upon the determined viewing location.

41. The method of claim 35 wherein the instrument operation function includes activating the surgical instrument if the viewing location matches a position of the surgical instrument in the displayed image for a predetermined length of time.

42. The method of claim 35, wherein the instrument operation function includes transferring control of the surgical instrument from the user to a second user if a viewing location of the second user matches a position of the surgical instrument in the displayed image for a predetermined length of time.

43. The method of claim 35 wherein the instrument operation function includes deactivating the surgical instrument if the viewing location does not match a position of the surgical instrument in the displayed image for a predetermined length of time.

44. The method of claim 35 wherein the instrument operation function includes moving the surgical instrument within the surgical field based upon the determined viewing location.

45. The method of claim 30 wherein the at least one function of the teleoperational medical system is a marking function that includes creating a virtual mark in the surgical field that is visible on the image display, based upon the determined viewing location.

46. The method of claim 30 wherein the at least one function of the teleoperational medical system is an imaging device operation function including controlling at least one function of an imaging device in the surgical field based upon the determined viewing location.

47. The method of claim 46 wherein the at least one function of the imaging device includes movement of the imaging device.

48. The method of claim 46 wherein the at least one function of the imaging device includes a zoom operation of the imaging device.

49. The method of claim 46 wherein the at least one function of the imaging device includes a focusing operation of the imaging device.

50. The method of claim 30 wherein the at least one function of the teleoperational medical system is a user log-in function.

51. The method of claim 30 wherein the at least one function of the teleoperational medical system is a lock-out function to lock out the user from the teleoperational medical system if the gaze point of the user is directed away from the displayed image.

52. The method of claim 30 wherein the displayed image includes a user interface comprising a plurality of functional options.

53. The method of claim 52 further comprising initiating at least one of the plurality of functional options if the determined viewing location matches a position of at least one functional option in the displayed image of the user interface.

54. The method of claim 53, further comprising adjusting a positional depth of the user interface in the displayed image based on the determined viewing location.

55. The method of claim 30 further comprising receiving an instruction input from the user and adjusting the displayed image based upon the instruction.

56. The method of claim 55 wherein the displayed image is adjusted by performing a zoom function.

57. The method of claim 55 wherein the displayed image is adjusted by displaying an overlay image on the image display based upon the gaze point of the user.

58. A teleoperational medical system for performing a medical procedure in a surgical field, the surgical system comprising:

an eye tracking unit including;

an image display configured to display to a user an image of the surgical field;

at least one eye tracker configured to measure data about a gaze point of the user;

a processor configured to process the data to determine a viewing location in the displayed image at which the gaze point of the user is directed;

a control unit configured to control at least one function of the teleoperational medical system based upon the determined viewing location; and

a surgical instrument, wherein the control unit is configured to control at least one function of the surgical instrument in the surgical field based upon the determined viewing location in the displayed image, and wherein the control unit is configured to activate the surgical instrument if the processor determines that the viewing location matches a position of the surgical instrument in the image display for a predetermined length of time.