Systems and methods of free-space gestural interaction

• US 10,831,281 B2
• Filed: 05/02/2019
• Issued: 11/10/2020
• Est. Priority Date: 08/09/2013
• Status: Active Grant

• Current Assignee: Ultrahaptics Ip Two Limited (Ultraleap Holdings Ltd.)
• Original Assignee: Ultrahaptics Ip Two Limited (Ultraleap Holdings Ltd.)
• Inventors: Yang, Hua, Kontsevich, Leonid, Donald, James, Holz, David S., Marsden, Jonathan, Durdik, Paul
• Primary Examiner(s): Vu, Kieu D
• Assistant Examiner(s): Borja, Roberto
• Application Number:
  US16/402,134
• Publication Number:
  US 20190258320A1
• Time in Patent Office: 558 Days
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/03545 Pens or stylus
• CPC Class Codes: G06F 3/0426 tracking fingers with respect to a virtual keyboard projected or printed on the surface virtual keyboards on touch screens G06F3/04886
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04847 Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
• CPC Class Codes: G06F 3/0486 Drag-and-drop
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

During control of a user interface via free-space motions of a hand or other suitable control object, switching between control modes can be facilitated by tracking the control object'"'"'s movements relative to, and its contact with a “virtual touch plane or surface” (i.e., a plane, portion of a plane, and/or surface computationally defined in space, or corresponding to any physical surface).

Claims

1. A method of disambiguating among input commands generated by free-space gestures, the method including:

detecting a compound free-space gesture in a three-dimensional (3D) sensory space, wherein the compound free-space gesture includes a sequence of gesture primitives;

setting a context for gesture interpretation responsive to a virtual interaction between a first gesture primitive of the compound free-space gesture and a virtual plane defined in the 3D sensory space that is within a 3D field of view, wherein an orientation and location of the virtual plane is dependent upon a current location and a tracked orientation of a pointing direction of a single point of the first gesture primitive extending along a length of the single point of the first gesture primitive without being dependent upon an orientation of a physical screen for displaying a proximate interface control;

responsive to a position of the virtual interaction with the virtual plane, tentatively selecting the proximate interface control displayed on the physical screen and visually indicating, on the physical screen, the tentative selection of the proximate interface control, the tentative selection of the proximate interface control representing a selection of the proximate interface control without activating a function corresponding to the proximate interface control;

detecting a second gesture primitive of the compound free-space gesture (i) anywhere in the 3D field of view, (ii) without maintaining a virtual contact between the first gesture primitive of the compound free-space gesture and the virtual plane and (iii) while the proximate interface control remains tentatively selected; and

initiating the function corresponding to the proximate interface control based on the second gesture primitive.

2. The method of claim 1, wherein gestures are made by a control object that includes a human hand, further including performing the first gesture primitive using a first finger of a control object and performing the second gesture primitive using a second finger of the control object.

3. The method of claim 1, further including producing affine mapping between the first gesture primitive and the second gesture primitive and resulting actuation of the proximate interface control by applying a scaling function that automatically proportions responsiveness of the proximate interface control to the first gesture primitive and the second gesture primitive.

4. A method of interpreting free-space gestures in an augmented reality environment, the method including:

detecting a compound free-space gesture in an augmented reality (AR) environment, wherein the compound free-space gesture includes a sequence of gesture primitives;

instantiating a free-floating interaction modality in a real world physical space in response to a first virtual interaction between a first gesture primitive of the compound free-space gesture and a virtual plane in a three-dimensional (3D) sensory space that is within a 3D field of view, wherein an orientation and location of the virtual plane is dependent upon a current location and a tracked orientation of a pointing direction of a single point of the first gesture primitive extending along a length of the single point of the first gesture primitive without being dependent upon an orientation of a physical screen for displaying a proximate modality control;

setting a context for free-space gestural interaction with the free-floating interaction modality responsive to a second virtual interaction between a second gesture primitive of the compound free-space gesture and the virtual plane;

responsive to a position of the second virtual interaction with the virtual plane, tentatively selecting the proximate modality control displayed on the physical screen and visually indicating, on the physical screen, the tentative selection of the proximate modality control, the tentative selection of the proximate modality control representing a selection of the proximate modality control without activating a function corresponding to the proximate modality control;

detecting a third gesture primitive of the compound free-space gesture (i) anywhere in the 3D field of view, (ii) without maintaining a virtual contact between the first gesture primitive of the compound free-space gesture and the virtual plane and (iii) while the proximate modality control remains tentatively selected; and

initiating the function corresponding to the proximate modality control the third gesture primitive.

5. The method of claim 4, wherein the first virtual interaction is a sweeping intersection of the virtual plane by the first gesture primitive.

6. The method of claim 4, wherein the second virtual interaction is a normal intersection of the virtual plane by the first gesture primitive.

7. The method of claim 4, wherein the third gesture primitive represents a confirmatory gesture.

8. A method of preventing false selections during free-space gestural interactions in a three-dimensional (3D) sensory space, the method including:

setting a context for gesture interpretation by detecting a virtual contact between a first control object and a virtual plane computationally defined as substantially perpendicular to an orientation of a pointing direction of the first control object extending along a length of the first control object in the 3D sensory space, wherein an orientation and location of the virtual plane is dependent upon a location and a tracked orientation of the pointing direction of the first control object extending along the length of the first control object without being dependent upon an orientation of a physical screen for displaying a proximate interface control;

responsive to a position of the virtual contact with the virtual plane, tentatively selecting the proximate interface control displayed on the physical screen and visually indicating, on the physical screen, the tentative selection of the proximate interface control, the tentative selection of the proximate interface control representing a selection of the proximate interface control without activating a function corresponding to the proximate interface control;

detecting a free-space confirmatory gesture of a second control object (i) anywhere in a 3D field of view, (ii) without maintaining the virtual contact between the first control object and the virtual plane and (iii) while the proximate interface control remains tentatively selected; and

initiating the function corresponding to the proximate interface control based on the free-space confirmatory gesture.