Robotic hair transplantation system with touchscreen interface for controlling movement of tool

• US 10,387,021 B2
• Filed: 07/31/2014
• Issued: 08/20/2019
• Est. Priority Date: 07/31/2014
• Status: Active Grant

• Current Assignee: Venus Concept, Inc.
• Original Assignee: Restoration Robotics, Inc. (Venus Concept, Inc.)
• Inventors: Shapter, Christopher C, Petrovic, Ognjen, Zingaretti, Gabriele
• Primary Examiner(s): Houston, Elizabeth
• Assistant Examiner(s): Wei, Charles M
• Application Number:
  US14/448,643
• Publication Number:
  US 20160030134A1
• Time in Patent Office: 1,846 Days
• Field of Search: 600562-571, 606133, 606187, 715863
• CPC Class Codes: A61B 17/32053 Punch like cutting instruments, e.g. using a cylindrical or oval knife A61B17/326 takes precedence; trepans A61B17/1695; for use on the cornea A61F9/007
• CPC Class Codes: A61B 2017/00752 Hair removal or transplantation
• CPC Class Codes: A61B 2034/258 providing specific settings for specific users
• CPC Class Codes: A61B 2090/371 with simultaneous use of two cameras
• CPC Class Codes: A61B 2090/3937 Visible markers
• CPC Class Codes: A61B 34/25 User interfaces for surgical systems
• CPC Class Codes: A61B 34/30 Surgical robots
• CPC Class Codes: A61B 90/361 Image-producing devices, e.g. surgical cameras
• CPC Class Codes: G06F 3/04847 Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06F 3/04886 by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus

Abstract

A system is disclosed for performing hair transplantation procedures. The system may include a robotic arm and a tool coupled to the robotic arm, the tool configured for hair harvesting, site making or hair implantation. The system further includes a touch screen user interface that supports touch gestures for efficiently creating and manipulating objects, such as reserved regions, that control the harvesting, site making, and/or implantation sites selected by the system. The user interface implements logic rules that use the context in which a touch gesture is performed to determine the type of operation intended. The touch interface substantially eliminates or reduces the need for a user to use a keyboard or mouse during the procedure.

Claims

1. A system for hair transplantation, the system comprising:

a computing system coupled to a camera, said computing system comprising a touch screen, said computing system programmed to provide, on the touch screen, a user interface that includes functionality for a user to view one or more images and to interactively control a location on a body surface from which follicular units are harvested, at which implantation sites are prepared, and/or at which implantation of follicular units is made;

said user interface including a reserved region drawing function that enables a user to interactively draw, on an image of the body portion via a shape drawing gesture on the touch screen, a shape representing a reserved region from which no follicular units are to be harvested, or at which no sites or implantations are to be made, said reserved region drawing function including logic rules for determining whether a touch gesture by the user on the image of the body surface is an attempt to draw a reserved region, said logic rules including at least one logic rule that compares a region corresponding to the touch gesture on the image of the body surface to at least one other region that is defined within the image of the body surface for controlling harvesting, site making, and/or implantation;

wherein the at least one logic rule determines a distance between (1) a location of the region corresponding to the touch gesture, and (2) a location corresponding to an interface object on the image of the body surface, and compares said distance to a threshold distance to determine whether to treat the touch gesture as an attempt to draw a reserved region, wherein the touch gesture is determined to be an attempt to draw a reserved region only if said distance is greater than the threshold distance;

wherein the interface object on the image of the body surface comprises a boundary indication, another reserved region, a virtual selection area indication, or a direction of travel indication; and

further comprising another logic rule that determines whether the touch gesture by the user on the image of the body surface corresponds to a valid region, wherein region validity is determined as a function of at least one of a dimensional threshold, a height to width ratio threshold, and an area threshold, and wherein the touch gesture is determined to be an attempt to draw a reserved region only if the touch gesture by the user on the image of the body surface is determined to correspond to a valid region;

wherein the reserved region drawing function is configured to create a reserved region of a predetermined size in response to a touch gesture by the user on the image of the body surface, wherein the predetermined size of the reserved region that is created in response to a two finger touch gesture is greater than the predetermined size of the reserved region that is created in response to a one finger touch gesture.

2. The system of claim 1, further comprising a robotic arm that controls a position of a tool relative to the body surface, and the camera is positioned on the robotic arm to capture images showing at least a portion of the body surface.

3. The system of claim 1, wherein the logic rules for determining whether a touch gesture is an attempt to draw a reserved region include a logic rule that determines whether an initial touch of the touch gesture falls within defined harvesting, site making, or implantation region.

4. The system of claim 1, wherein the logic rules for determining whether a touch gesture is an attempt to draw a reserved region include a logic rule that determines whether the system for hair transplantation is in site making or implantation mode.

5. The system of claim 1, wherein the another logic rule for determining whether a touch gesture is an attempt to draw a valid region include a logic rule that determines whether a ratio of a width and height of the region corresponding to the touch gesture satisfies a threshold ratio requirement for representing a reserved region.

6. The system of claim 1, wherein the logic rules for determining whether a touch gesture is an attempt to draw a reserved region include a logic rule that determines whether the region corresponding to the touch gesture encompasses a minimum number of follicular units.

7. The system of claim 1, wherein the logic rules for determining whether a touch gesture is an attempt to draw a reserved region include logic for determining whether a selected object is a previously drawn reserved region and imposing at least some restrictions on an attempt to modify the previously drawn reserved region.

8. The system of claim 1,

wherein said reserved region drawing function enables the user to use a multiple-finger gesture, and said reserved region drawing function includes logic for determining whether multiple concurrent touch events on the image of the body surface represent a request to create the reserved region.

9. The system of claim 8, further comprising a robotic arm that controls a position of the tool relative to the body surface and the camera mounted on the robotic arm.

10. The system of claim 8, wherein the logic determines whether the multiple concurrent touch events represent a request to create a reserved region based at least in part on a measure of spatial separation between the multiple concurrent touch events.

11. The system of claim 10, wherein logic determines whether the multiple concurrent touch events represent a request to create a reserved region based additionally on locations of the multiple concurrent touch events relative to a previously defined harvesting, site making, or implantation region representing a region of the body surface at which harvesting, site making, or implantation is permitted.

12. The system of claim 8, wherein the logic treats the multiple concurrent touch events as a request to create a reserved region only if spatial separation between the multiple concurrent touch events falls below a threshold, whereby the logic inhibits accidental creation of reserved regions.

13. The system of claim 8, wherein the reserved region drawing function is responsive to detection of a valid multiple-finger gesture by generating, and displaying a visual representation of, a reserved region of a predetermined size and shape.

14. The system of claim 8, wherein the user interface also implements a reserved region function that is responsive to a touch and drag gesture on a visual representation of a reserved region by moving a position of the reserved region.

15. The system of claim 1, wherein the logic rules comprise rule for determining both whether the touch gesture is 1) an attempt to draw the reserved region, or 2) an attempt to move or modify the reserved region, or perform another operation.

16. The system of claim 1, wherein the logic rules comprise determining a number of touch points.

17. The system of claim 16, wherein the determined number of touch points is one, and the processor dynamically determines the user'"'"'s drawing intent.

18. The system of claim 1, wherein the logic rules comprise determining whether there is a follicular unit or hair follicle near a touch point.

19. A non-transitory computer storage medium having stored thereon:

an executable code module that provides a touch interface for interactively controlling a hair transplantation procedure via a touch screen of a hair transplantation system, the executable code module comprising program instructions that direct the hair transplantation system to implement a process that comprises;

detecting a touch gesture in which a user touches, within a displayed image of a body surface, a starting location on the touch screen and drags a finger along the touch screen from the starting location; and

determining, based on a set of logic rules that consider a context of the touch gesture, whether to treat the touch gesture;

     1) either as an attempt by the user to draw a reserved region that defines an area on the body surface to be excluded from at least a portion of the hair transplantation procedure, or

     2) as an attempt by the user to indicate a modification or another operation,

wherein said logic rules include at least one logic rule that compares a region corresponding to the touch gesture on the image of the body surface to at least one other region that is defined within the image of the body surface for controlling harvesting, site making, and/or implantation;

wherein the at least one logic rule determines a distance between (1) a location of the region corresponding to the touch gesture, and (2) a location corresponding to an interface object on the image of the body surface, and compares said distance to a threshold distance to determine whether to treat the touch gesture as an attempt to draw a reserved region, wherein the touch gesture is determined to be an attempt to draw a reserved region only if said distance is greater than the threshold distance;

wherein the interface object on the image of the body surface comprises a boundary indication, another reserved region, a virtual selection area indication, or a direction of travel indication; and

further comprising another logic rule that determines whether the touch gesture by the user on the image of the body surface corresponds to a valid region, wherein region validity is determined as a function of at least one of a dimensional threshold, a height to width ratio threshold, and an area threshold, and wherein the touch gesture is determined to be an attempt to draw a reserved region only if the touch gesture by the user on the image of the body surface is determined to correspond to a valid region;

wherein the reserved region drawing function is configured to create a reserved region of a predetermined size in response to a touch gesture by the user on the image of the body surface, wherein the predetermined size of the reserved region that is created in response to a two finger touch gesture is greater than the predetermined size of the reserved region that is created in response to a one finger touch gesture.

20. The non-transitory computer storage medium of claim 19, wherein the reserved region is a zone from which no follicular units are to be harvested, or at which no site or implantations are to be made.

21. The non-transitory computer storage medium of claim 19, wherein the reserved region is a zone into which no follicular units are to be implanted or a zone where no incisions are to be made.

22. The non-transitory computer storage medium of claim 19, wherein the touch interface enables the user to draw the reserved region without first placing the system in a drawing mode.

23. The non-transitory computer storage medium of claim 19, wherein the logic rules for determining whether a touch gesture is an attempt to draw a reserved region include a logic rule that determines whether an initial touch of the touch gesture falls within defined harvesting, implanting, or site making region.

24. The non-transitory computer storage medium of claim 19, wherein the logic rules for determining whether a touch gesture is an attempt to draw a reserved region include a logic rule that determines whether the system for hair transplantation is in site making or implantation mode.

25. The non-transitory computer storage medium of claim 19, wherein the logic rules for determining whether a touch gesture is an attempt to draw a reserved region include logic for determining whether a selected object is a previously drawn reserved region and imposing at least some restrictions on an attempt to modify the previously drawn reserved region.

26. The non-transitory computer storage medium of claim 19, wherein the computing system is configured to apply the logic rules in real time as the touch gesture is performed, and to provide real time visual feedback on the touch screen indicative of whether the touch gesture is being interpreted as an attempt to draw a reserved region.

27. The non-transitory computer storage medium of claim 19, wherein the determining whether to treat the touch gesture as an attempt to draw the reserved region also comprises determining whether the touch gesture is an attempt to move or modify the reserved region, or perform another operation.

28. A method of controlling a hair transplantation system, comprising:

generating using a camera, on a touch screen, an image of a body surface;

detecting a touch gesture in which a user touches the touch screen at an initial location within the image of the body surface and then moves a finger along the touch screen; and

while the user performs the touch gesture;

applying logic rules to determine whether the touch gesture satisfies a set of requirements for drawing a reserved region that defines a portion of the body surface to be excluded from at least one phase of the hair transplantation procedure, said logic rules including at least one logic rule that compares a region corresponding to the touch gesture on the image of the body surface to at least one other region that is defined within the image of the body surface for controlling harvesting, site making, and/or implantation; and

based on application of the logic rules, providing visual feedback on the touch screen to the user indicative of whether touch gesture is being treated as a request to draw a reserved region;

wherein the at least one logic rule determines a distance between (1) a location of the region corresponding to the touch gesture, and (2) a location corresponding to an interface object on the image of the body surface, and compares said distance to a threshold distance to determine whether to treat the touch gesture as an attempt to draw a reserved region, wherein the touch gesture is determined to be an attempt to draw a reserved region only if said distance is greater than the threshold distance;

wherein the interface object on the image of the body surface comprises a boundary indication, another reserved region, a virtual selection area indication, or a direction of travel indication; and

further comprising another logic rule that determines whether the touch gesture by the user on the image of the body surface corresponds to a valid region, wherein region validity is determined as a function of at least one of a dimensional threshold, a height to width ratio threshold, and an area threshold, and wherein the touch gesture is determined to be an attempt to draw a reserved region only if the touch gesture by the user on the image of the body surface is determined to correspond to a valid region;

wherein the reserved region drawing function is configured to create a reserved region of a predetermined size in response to a touch gesture by the user on the image of the body surface, wherein the predetermined size of the reserved region that is created in response to a two finger touch gesture is greater than the predetermined size of the reserved region that is created in response to a one finger touch gesture.

29. The method of claim 28, said method performed under control of program instructions executed by a robotic hair transplantation system operatively attached to the camera.

30. The method of claim 28, wherein the logic rules include logic for determining whether the touch gesture represents a request to move a previously defined reserved region.

31. The method of claim 28, further comprising:

generating, on the touch screen, a visual representation of a boundary of a procedure region wherein the procedure region comprises a harvesting region, a site making region, or an implantation region, said procedure region capable of encompassing one or more reserved regions; and

applying a second set of logic rules to determine whether the touch gesture represents a request to move the boundary of the procedure region.

32. The method of claim 28, wherein said detecting the touch gesture occurs without first placing the system in a drawing mode.

33. The method of claim 28, wherein said applying logic rules comprises applying a logic rule that determines whether an initial touch of the touch gesture falls within defined harvesting, site making, or implantation region.

34. The method of claim 28, wherein said applying logic rules comprises applying a logic rule that determines whether an area of the region corresponding to the touch gesture satisfies a minimum requirement.

35. The method of claim 28, wherein said applying logic rules comprises applying a logic rule that determines whether a width and height of the region corresponding to the touch gesture satisfy a requirement for representing a touch gesture.

36. The method of claim 28, wherein applying logic rules comprises applying the logic rules in real time as the touch gesture is performed, and providing real time visual feedback on the touch screen indicative of whether the touch gesture is being interpreted as an attempt to draw a reserved region.

37. The method of claim 28, wherein applying the logic rules comprises determining whether the touch gesture is an attempt to move or modify the reserved region, or perform another operation.