Free-space user interface and control using virtual constructs

• US 9,501,152 B2
• Filed: 01/14/2014
• Issued: 11/22/2016
• Est. Priority Date: 01/15/2013
• Status: Active Grant

• Current Assignee: Lmi Liquidating Co., LLC
• Original Assignee: Leap Motion Incorporated (Ultraleap Limited)
• Inventors: Bedikian, Raffi, Marsden, Jonathan, Mertens, Keith, Holz, David
• Primary Examiner(s): Nguyen, Kevin
• Application Number:
  US14/154,730
• Publication Number:
  US 20140201689A1
• Time in Patent Office: 1,043 Days
• Field of Search: 715/863, 715/738, 715761-765, 715851-853
• US Class Current: 1/1
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

During control of a user interface via free-space motions of a hand or other suitable control object, switching between control modes may be facilitated by tracking the control object'"'"'s movements relative to, and its penetration of, a virtual control construct (such as a virtual surface construct). The position of the virtual control construct may be updated, continuously or from time to time, based on the control object'"'"'s location.

Claims

1. A computer-implemented method of controlling a machine user interface, comprising:

receiving information including motion information for a control object;

determining from the motion information whether a motion of the control object is an engagement gesture according to an occurrence of an engagement gesture applied to at least one virtual control construct defined within a field of view of an image capturing device, comprising;

determining whether an intersection occurred between control object and at least one virtual control construct, and when an intersection has occurred determining from the motion information whether the engagement includes continued motion after intersection;

otherwise

determining whether a dis-intersection of the control object from the at least one virtual control construct occurred;

otherwise

determining whether motion of the control object occurred relative to at least one virtual control construct;

determining from the motion information a set of engagement attributes defining an engagement gesture; and

identifying an engagement gesture by correlating motion information to at least one engagement gesture based at least upon one or more of motion of the control object, occurrence of any of an intersection, a dis-intersection or a non-intersection of the control object with the virtual control construct, and the set of engagement attributes;

determining a control to which the engagement gesture is applicable; and

manipulating the control according to at least the engagement gesture.

2. A computer-implemented method for controlling a user interface via free-space motions of a control object, the method comprising:

receiving motion information indicating positions of a control object being tracked in free space; and

using a processor, (i) defining a virtual control construct, at least a portion thereof having a spatial position determined based at least in part on the motion information such that the virtual control construct portion is positioned proximate to the control object;

(ii) determining from the motion information whether the tracked motions of the control object indicate that the control object has intersected the virtual control construct; and

(iii) switching from conducting control of a user interface in a first mode to conducting control of the user interface in a second mode based at least in part upon an occurrence of the control object intersecting the virtual control construct.

3. The method of claim 2, further comprising:

updating at least the spatial position of the virtual control construct portion based at least in part on the motion information such that the virtual control construct portion is enabled to follow the control object.

4. The method of claim 3, wherein the virtual control construct computationally follows the tracked motions of the control object with a time lag.

5. The method of claim 4, wherein the time lag is fixed.

6. The method of claim 4, wherein the time lag is computed by the processor and depends on a motion parameter of the control object.

7. The method of claim 3, wherein the spatial position of the virtual control construct is updated by the processor based on a current distance between the control object and the virtual control construct.

8. The method of claim 7, wherein the spatial position of the virtual control construct is updated in accordance with a virtual energy potential defined as a function of the distance between the control object and the virtual control construct.

9. The method of claim 8, wherein the virtual energy potential comprises minima at steady-state distances between the control object and the virtual control construct in the engaged mode and the disengaged mode.

10. The method of claim 9, wherein the steady-state distance in the engaged mode is equal to the steady-state distance in the disengaged mode.

11. The method of claim 9, wherein the steady-state distance in the engaged mode is larger than the steady-state distance in the disengaged mode.

12. The method of claim 2, further comprising computationally tracking the motions of the control object based on a temporal sequence of images of the control object.

13. The method of claim 12, further comprising acquiring the sequence of images with a camera system having depth-sensing capability.

14. The method of claim 2, further comprising computationally filtering the motions of the control object based, at least in part, on a velocity thereof.

15. The method of claim 2, wherein the first mode is an engaged mode and the second mode is a disengaged mode.

16. The method of claim 2, further comprising computationally determining, during a transition from the disengaged mode to the engaged mode, a degree of penetration of the virtual control construct by the control object, and controlling the user interface based at least in part thereon.

17. The method of claim 2, wherein conducting control of the user interface comprises updating screen content based, at least in part, on the mode and the tracked motions of the control object.

18. The method of claim 2, wherein conducting control of the user interface comprises operating a cursor associated with a position on a screen based, at least in part, on the mode and the tracked motions of the control object.

19. The method of claim 18, wherein operating the cursor comprises displaying a cursor symbol on the screen at the associated position.

20. The method of claim 19, wherein the cursor symbol is indicative of a distance between the control object and the virtual control construct.

21. The method of claim 2, wherein the virtual control construct comprises a virtual surface construct.

22. The method of claim 21, wherein the virtual surface construct is computationally defined as a plane oriented relative to at least one of a tracked orientation of the control object or an orientation of a screen displaying the user interface.

23. The method of claim 2, wherein determining whether the control object has intersected the virtual control construct comprises computing an intersection of a straight line through an axis of the control object with a screen displaying the user interface.

24. The method of claim 2, wherein determining whether the control object has intersected the virtual control construct comprises computationally projecting a tip of the control object perpendicularly onto a screen displaying the user interface.

25. A system including one or more processors coupled to memory, the memory loaded with computer instructions to control a user interface via free-space motions of a control object, the instructions, when executed on the processors, implement actions comprising:

receiving motion information indicating positions of a control object being tracked in free space;

defining a virtual control construct, at least a portion thereof having a spatial position determined based at least in part on the motion information such that the virtual control construct portion is positioned proximate to the control object;

determining from the motion information whether the tracked motions of the control object indicate that the control object has intersected the virtual control construct; and

switching from conducting control of a user interface in a first mode to conducting control of the user interface in a second mode based at least in part upon an occurrence of the control object intersecting the virtual control construct.

26. The system of claim 25, further implementing actions comprising:

updating at least the spatial position of the virtual control construct portion based at least in part on the motion information such that the virtual control construct portion is enabled to follow the control object.

27. The system of claim 26, wherein the spatial position of the virtual control construct is updated in accordance with a virtual energy potential defined as a function of a distance between the control object and the virtual control construct.

28. The system of claim 25, wherein the first mode is an engaged mode and the second mode is a disengaged mode.

29. The system of claim 28, further implementing actions comprising computationally determining, during a transition from the disengaged mode to the engaged mode, a degree of penetration of the virtual control construct by the control object, and controlling the user interface based at least in part thereon.

30. The system of claim 25, wherein determining whether the control object has intersected the virtual control construct comprises computationally projecting a tip of the control object perpendicularly onto a screen displaying the user interface.

31. A non-transitory computer readable storage medium impressed with computer program instructions to control a user interface via free-space motions of a control object, the instructions, when executed on a processor, implement a method comprising:

receiving motion information indicating positions of a control object being tracked in free space;

defining a virtual control construct, at least a portion thereof having a spatial position determined based at least in part on the motion information such that the virtual control construct portion is positioned proximate to the control object;

determining from the motion information whether the tracked motions of the control object indicate that the control object has intersected the virtual control construct; and

switching from conducting control of a user interface in a first mode to conducting control of the user interface in a second mode based at least in part upon an occurrence of the control object intersecting the virtual control construct.

32. The non-transitory computer readable storage medium of claim 31 implementing the method further comprising:

updating at least the spatial position of the virtual control construct portion based at least in part on the motion information such that the virtual control construct portion is enabled to follow the control object.

33. The non-transitory computer readable storage medium of claim 32, wherein the spatial position of the virtual control construct is updated in accordance with a virtual energy potential defined as a function of a distance between the control object and the virtual control construct.

34. The non-transitory computer readable storage medium of claim 31, wherein the first mode is an engaged mode and the second mode is a disengaged mode.

35. The non-transitory computer readable storage medium of claim 34 implementing the method further comprising computationally determining, during a transition from the disengaged mode to the engaged mode, a degree of penetration of the virtual control construct by the control object, and controlling the user interface based at least in part thereon.

36. The non-transitory computer readable storage medium of claim 31, wherein determining whether the control object has intersected the virtual control construct comprises computationally projecting a tip of the control object perpendicularly onto a screen displaying the user interface.

37. A non-transitory computer readable storage medium impressed with computer program instructions to control a machine user interface, the instructions, when executed on a processor, implement a method comprising:

receiving information including motion information for a control object;

determining from the motion information whether a motion of the control object is an engagement gesture according to an occurrence of an engagement gesture applied to at least one virtual control construct defined within a field of view of an image capturing device, comprising;

determining whether an intersection occurred between control object and at least one virtual control construct, and when an intersection has occurred determining from the motion information whether the engagement includes continued motion after intersection;

otherwise

determining whether a dis-intersection of the control object from the at least one virtual control construct occurred;

otherwise

determining whether motion of the control object occurred relative to at least one virtual control construct;

determining from the motion information a set of engagement attributes defining an engagement gesture; and

identifying an engagement gesture by correlating motion information to at least one engagement gesture based at least upon one or more of motion of the control object, occurrence of any of an intersection, a dis-intersection or a non-intersection of the control object with the virtual control construct, and the set of engagement attributes;

determining a control to which the engagement gesture is applicable; and

manipulating the control according to at least the engagement gesture.