Free-space user interface and control using virtual constructs

• US 10,042,430 B2
• Filed: 11/21/2016
• Issued: 08/07/2018
• Est. Priority Date: 01/15/2013
• Status: Active Grant

• Current Assignee: Lmi Liquidating Co., LLC
• Original Assignee: Leap Motion Incorporated (Ultraleap Limited)
• Inventors: Bedikian, Raffi, Marsden, Jonathan, Mertens, Keith, Holz, David
• Primary Examiner(s): Nguyen, Kevin
• Application Number:
  US15/358,104
• Publication Number:
  US 20170075428A1
• Time in Patent Office: 624 Days
• Field of Search: 715768, 715752, 715851-853, 715740
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

During control of a user interface via free-space motions of a hand or other suitable control object, switching between control modes can be facilitated by tracking the control object'"'"'s movements relative to, and its penetration of, a virtual control construct (such as a virtual surface construct). The position of the virtual control construct can be updated, continuously or from time to time, based on the control object'"'"'s location.

Claims

1. A computer-implemented method for controlling a user interface via free-space motions of a control object, the method comprising:

receiving motion information indicating positions of a control object being tracked in a region of free space; and

using a processor;

(i) defining a plurality of virtual control constructs, including at least a first virtual control construct defined at a spatial position determined based at least in part on the motion information for a corresponding first portion of the control object;

whereby the first virtual control construct is positioned proximate to the first portion of the control object, and a second virtual control construct defined at a spatial position determined based at least in part on the motion information for a corresponding second portion of the control object;

whereby the second virtual control construct is positioned proximate to the second portion of the control object;

(ii) determining an input gesture made by the control object based on a first portion state determined for the first portion of the control object and including any change in spatial position of the first portion of the control object relative to the first virtual control construct and a second portion state determined for the second portion of the control object and including any change in spatial position of the second portion of the control object relative to the second virtual control construct; and

(iii) switching from conducting control of a user interface in a first mode to conducting control of the user interface in a second mode based at least in part upon interpreting the input gesture determined from the first portion state and the second portion state.

2. The computer-implemented method of claim 1, wherein:

the control object includes a hand, the first portion includes a finger and the second portion includes a thumb, and wherein the determining an input gesture includes;

determining that the first portion state and the second portion state indicate that the finger and the thumb changed distance from their corresponding user-specific virtual planes;

thereby reducing a distance between the finger and the thumb; and

determining from the first portion state and the second portion state that the input gesture comprises a pinching gesture of the thumb and finger.

3. The computer-implemented method of claim 2, wherein the switching further comprises:

interpreting the pinching gesture to be a command indicating a zooming out of displayed content; and

conducting control of the user interface zooming out of displayed content.

4. The computer-implemented method of claim 2, further comprising:

determining that at least one of the finger and the thumb penetrated a corresponding virtual control construct; and

determining from the first portion state and the second portion state that the input gesture comprises a maximal pinching gesture of the thumb and finger.

5. The computer-implemented method of claim 3, wherein the switching further comprises:

interpreting the maximal pinching gesture to be a command indicating a maximum zooming out of displayed content; and

conducting control of the user interface to perform continued zooming out of displayed content.

6. The computer-implemented method of claim 1, wherein:

the control object includes a hand, the first portion includes a finger and the second portion includes a thumb, and wherein the determining an input gesture includes;

determining that the first portion state and the second portion state indicate that the finger and the thumb changed distance from their corresponding user-specific virtual planes;

thereby increasing a distance between the finger and the thumb; and

determining from the first portion state and the second portion state that the input gesture comprises a spreading gesture of the thumb and finger.

7. The computer-implemented method of claim 6, wherein the switching further comprises:

interpreting the spreading gesture to be a command indicating a zooming in of displayed content; and

conducting control of the user interface to zooming in of displayed content.

8. The computer-implemented method of claim 6, further comprising:

determining that at least one of the finger and the thumb are dis-engaged from a corresponding virtual control construct; and

determining from the first portion state and the second portion state that the input gesture comprises a maximal spreading gesture of the thumb and finger.

9. The computer-implemented method of claim 8, wherein the switching further comprises:

interpreting the maximal spreading gesture to be a command indicating a maximum zooming in of displayed content; and

conducting control of the user interface to continued zooming in of displayed content.

10. The computer-implemented method of claim 1, further comprising:

updating a spatial position of at least one virtual control construct based at least in part on the motion information of a corresponding portion of the control object such that the virtual control construct is enabled to follow the corresponding portion of the control object.

11. The computer-implemented method of claim 10, wherein the virtual control construct computationally follows tracked motions of the control object portion with a time lag.

12. The computer-implemented method of claim 11, wherein the time lag is fixed.

13. The computer-implemented method of claim 12, wherein the time lag is computed by the processor and depends on a motion parameter of the control object portion.

14. The computer-implemented method of claim 10, wherein the spatial position of the virtual control construct is updated by the processor based on a current distance between the control object portion and the virtual control construct.

15. The computer-implemented method of claim 14, wherein the spatial position of the virtual control construct is updated in accordance with a virtual energy potential defined as a function of a distance between the control object portion and the virtual control construct.

16. The computer-implemented method of claim 15, wherein the virtual energy potential comprises minima at steady-state distances between the control object portion and a corresponding virtual control construct at a time when the control object portion is engaged with the virtual control construct and a time when the control object portion is disengaged from the virtual control construct.

17. The computer-implemented method of claim 1, further comprising computationally tracking the motions of the control object portions based on a temporal sequence of images of the control object.

18. The computer-implemented method of claim 17, further comprising acquiring the sequence of images with a camera system having depth-sensing capability.

19. The method of claim 1, wherein the first mode is an engaged mode and the second mode is a disengaged mode.

20. The method of claim 1, further comprising computationally determining, during a transition from a disengaged mode to an engaged mode, a degree of penetration of the virtual control construct by the control object, and controlling the user interface based at least in part thereon.

21. The method of claim 1, wherein conducting control of the user interface comprises updating screen content based, at least in part, on the mode and tracked motions of the control object.

22. The method of claim 1, wherein conducting control of the user interface comprises operating a cursor associated with a position on a screen based, at least in part, on the mode and tracked motions of the control object.

23. The method of claim 22, wherein operating the cursor comprises displaying a cursor symbol on the screen at the associated position.

24. The method of claim 23, wherein the cursor symbol is indicative of a distance between the control object portion and a corresponding virtual control construct.

25. A system including one or more processors coupled to memory, the memory loaded with computer instructions to control a user interface via free-space motions of a control object, the instructions, when executed on the processors, implement actions comprising:

receiving motion information indicating positions of a control object being tracked in a region of free space;

defining a plurality of virtual control constructs, including at least a first virtual control construct defined at a spatial position determined based at least in part on the motion information for a corresponding first portion of the control object;

whereby the first virtual control construct is positioned proximate to the first portion of the control object, and a second virtual control construct defined at a spatial position determined based at least in part on the motion information for a corresponding second portion of the control object;

whereby the second virtual control construct is positioned proximate to the second portion of the control object;

determining an input gesture made by the control object based on a first portion state determined for the first portion of the control object and including any change in spatial position of the first portion of the control object relative to the first virtual control construct and a second portion state determined for the second portion of the control object and including any change in spatial position of the second portion of the control object relative to the second virtual control construct; and

switching from conducting control of a user interface in a first mode to conducting control of the user interface in a second mode based at least in part upon interpreting the input gesture determined from the first portion state and the second portion state.

26. A non-transitory computer readable storage medium impressed with computer program instructions to control a user interface via free-space motions of a control object, the instructions, when executed on a processor, implement a method comprising:

receiving motion information indicating positions of a control object being tracked in a region of free space;

defining a plurality of virtual control constructs, including at least a first virtual control construct defined at a spatial position determined based at least in part on the motion information for a corresponding first portion of the control object;

whereby the first virtual control construct is positioned proximate to the first portion of the control object, and a second virtual control construct defined at a spatial position determined based at least in part on the motion information for a corresponding second portion of the control object;

whereby the second virtual control construct is positioned proximate to the second portion of the control object;

determining an input gesture made by the control object based on a first portion state determined for the first portion of the control object and including any change in spatial position of the first portion of the control object relative to the first virtual control construct and a second portion state determined for the second portion of the control object and including any change in spatial position of the second portion of the control object relative to the second virtual control construct; and

switching from conducting control of a user interface in a first mode to conducting control of the user interface in a second mode based at least in part upon interpreting the input gesture determined from the first portion state and the second portion state.