Non-tactile interface systems and methods

• US 10,452,151 B2
• Filed: 03/09/2018
• Issued: 10/22/2019
• Est. Priority Date: 04/26/2013
• Status: Active Grant

• Current Assignee: Lmi Liquidating Co., LLC
• Original Assignee: Ultrahaptics Ip Two Limited (Ultraleap Holdings Ltd.)
• Inventors: Zagorsek, Michael, Dabir, Avinash, Durdik, Paul, Mertens, Keith
• Primary Examiner(s): Eisen, Alexander
• Assistant Examiner(s): Teshome, Kebede T
• Application Number:
  US15/917,066
• Publication Number:
  US 20190018495A1
• Time in Patent Office: 592 Days
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06V 40/11 Hand-related biometrics; Hand pose recognition
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

Methods and systems for processing an input are disclosed that detect a portion of a hand and/or other detectable object in a region of space monitored by a 3D sensor. The method further includes determining a zone corresponding to the region of space in which the portion of the hand or other detectable object was detected. Also, the method can include determining from the zone a correct way to interpret inputs made by a position, shape or a motion of the portion of the hand or other detectable object.

Claims

1. A machine-implemented method for processing an input gesture, the method comprising:

tracking, using a 3D sensor, user movements including sensing positional information of a portion of a hand in a monitored region of space monitored by the 3D sensor;

using the sensed positional information of the portion of the hand, defining a plurality of distinct user-specific virtual planes, including at least a first user-specific virtual plane defined in space relative to a position of, and corresponding to, a first finger of the hand and a second user-specific virtual plane defined in space relative to a position of, and corresponding to, a second finger of the hand, in the monitored region of space;

detecting, by the 3D sensor, a first finger state of the first finger relative to the corresponding first user-specific virtual plane and a second finger state of the second finger relative to the corresponding second user-specific virtual plane, wherein a finger state for a finger relative to the corresponding user-specific virtual plane defined for the finger is one of;

the finger moving closer or further away from the corresponding user-specific virtual plane, and the finger moving on or against the corresponding user-specific virtual plane;

determining an input gesture made by the portion of the hand based on the first finger state and the second finger state;

interpreting the input gesture as a command using the input gesture determined from the first finger state and the second finger state; and

providing the command to a machine for executing an action appropriate to the command.

2. The machine-implemented method of claim 1, further comprising:

determining the input gesture to be a pinch gesture and interpreting the pinch gesture to be a command indicating a zoom-in command, the determining and interpreting including;

determining that the first finger state and the second finger state indicate that the first finger and the second finger changed distance from their corresponding user-specific virtual planes;

determining that the first finger and second finger penetrating their corresponding user-specific virtual planes;

using the sensed positional information of the hand, determining from a plurality of zones defined for the monitored region, a zone in which the portion of the hand is present at the time the first finger state and the second finger state are detected; and

determining that the portion of the hand was present in a zone specific to entering commands.

3. The machine-implemented method of claim 1, further comprising:

determining the input gesture to be a spreading gesture and interpreting spreading gesture to be a command indicating a zoom-out command, the determining and interpreting including;

determining that the first finger state and the second finger state indicate that the first finger and the second finger changed distance from their corresponding user-specific virtual planes;

determining that the first finger and second finger receded from their corresponding user-specific virtual planes;

using the sensed positional information of the hand, determining from a plurality of zones defined for the monitored region, a zone in which the portion of the hand is present at the time the first finger state and the second finger state are detected; and

determining that the portion of the hand was present in a zone specific to entering commands.

4. The machine-implemented method of claim 1, further comprising:

determining the input gesture to be a pressure gesture and interpreting the pressure gesture to be a command indicating a command to draw a thicker line on a display, the determining and interpreting including;

determining that the first finger state and the second finger state indicate that the first finger and the second finger changed distance from their corresponding user-specific virtual planes;

determining that the first finger and second finger penetrated their corresponding user-specific virtual planes;

using the sensed positional information of the hand, determining from a plurality of zones defined for the monitored region, a zone in which the portion of the hand is present at the time the first finger state and the second finger state are detected; and

determining that the portion of the hand was present in a zone specific to entering content.

5. The method of claim 1, further comprising:

determining the command to indicate movement of a displayed object on a display responsive to;

determining that at least one of the first finger state and the second finger state indicates that the corresponding finger moved along a surface of the corresponding user-specific virtual plane;

using the sensed positional information of the hand, determining from a plurality of zones defined for the monitored region, a zone in which the portion of the hand is present at the time the first finger state and the second finger state are detected; and

determining that the portion of the hand was in a zone specific to entering content.

6. The machine-implemented method of claim 1, further comprising:

determining from the sensed user movement a speed of the portion of the hand and adjusting gesture-recognition sensitivity based on the speed of the user movement.

7. The machine-implemented method of claim 1, further comprising:

dynamically personalizing one or more planes by characterizing trajectories of user gestures as vectors in 3-dimensional space, using the vectors to determine a maximal depth extent from gestural movement along an axis extending from the user to a display; and

defining a thickness of a plane based upon the maximal depth.

8. The machine-implemented method of claim 7, further comprising:

revising plane positions as more user movements are detected.

9. The machine-implemented method of claim 1 further comprising:

shifting a plane in space as the user changes position.

10. The machine-implemented method of claim 1,

wherein the portion of the hand or other detectable object can be found in more than one zone, the more than one zone comprising a set of zones, and

wherein the method further includes:

determining from the set of zones in which the portion of the hand or other detectable object can be found, a first preferred zone;

assigning the portion of the hand or other detectable object to the first preferred zone; and

reporting the first preferred zone for further processing;

wherein the determining the first preferred zone includes;

applying a set of one or more rules to the set of zones to determine, from the set of zones, the first preferred zone according to the one or more applied rules; and

reporting the first preferred zone for further processing.

11. The machine-implemented method of claim 10, wherein determining the first preferred zone includes:

applying a zone hierarchy to the set of zones to determine, from the set of zones, a zone highest on the zone hierarchy; and

reporting the zone highest on the zone hierarchy as the first preferred zone for further processing.

12. The machine-implemented method of claim 1, further comprising determining a correct way to interpret an input detected by the 3D sensor including:

using the sensed positional information of the hand, determining from a plurality of zones defined for the monitored region, a zone in which the portion of the hand is present at the time the first finger state and the second finger state are detected;

determining an active program and that the determined zone is a hover zone for the active program; and

interpreting a position or a motion of the portion of the hand or other detectable object based on the active program and the hover zone.

13. The machine-implemented method of claim 1, further comprising:

interpreting a motion of the portion of the hand or other detectable object as one or more gestures; and

determining from the one or more gestures an input to at least one of (i) an application and (ii) an operating system.

14. A system, including:

an image-capture device including at least one camera;

an image analyzer coupled to the at least one camera, the image analyzer being configured to;

track, using a 3D sensor, user movements including sensing positional information of a portion of a hand in a monitored region of space monitored by the 3D sensor;

using the sensed positional information of the portion of the hand, define a plurality of distinct user-specific virtual planes, including at least a first user-specific virtual plane defined in space relative to a position of, and corresponding to, a first finger of the hand and a second user-specific virtual plane defined in space relative to a position of, and corresponding to, a second finger of the hand, in the monitored region of space;

detect a first finger state of the first finger relative to the corresponding first user-specific virtual plane and a second finger state of the second finger relative to the corresponding second user-specific virtual plane, wherein a finger state for a finger relative to the corresponding user-specific virtual plane defined for the finger is one of;

the finger moving closer or further away from the corresponding user-specific virtual plane, and the finger moving on or against the corresponding user-specific virtual plane;

determine an input gesture made by the portion of the hand based on the first finger state and the second finger state;

interpret the input gesture as a command using the input gesture determined from the first finger state and the second finger state; and

provide the command to a machine for executing an action appropriate to the command.

15. The system of claim 14, wherein the image analyzer is further configured to:

determine the input gesture to be a pinch gesture and interpreting the pinch gesture to be a command indicating a zoom-in command, the determining and interpreting including;

determining that the first finger state and the second finger state indicate that the first finger and the second finger changed distance from their corresponding user-specific virtual planes;

determining that the first finger and second finger penetrating their corresponding user-specific virtual planes;

using the sensed positional information of the hand, determining from a plurality of zones defined for the monitored region, a zone in which the portion of the hand is present at the time the first finger state and the second finger state are detected; and

determining that the portion of the hand was present in a zone specific to entering commands.

16. The system of claim 14, wherein the image analyzer is further configured to:

determine the input gesture to be a spreading gesture and interpreting spreading gesture to be a command indicating a zoom-out command, the determining and interpreting including;

determining that the first finger state and the second finger state indicate that the first finger and the second finger changed distance from their corresponding user-specific virtual planes;

determining that the first finger and second finger receded from their corresponding user-specific virtual planes;

using the sensed positional information of the hand, determining from a plurality of zones defined for the monitored region, a zone in which the portion of the hand is present at the time the first finger state and the second finger state are detected; and

determining that the portion of the hand was present in a zone specific to entering commands.

17. The system of claim 14, wherein the image analyzer is further configured to:

determine the input gesture to be a pressure gesture and interpreting the pressure gesture to be a command indicating a command to draw a thicker line on a display, the determining and interpreting including;

determining that the first finger state and the second finger state indicate that the first finger and the second finger changed distance from their corresponding user-specific virtual planes;

determining that the first finger and second finger penetrated their corresponding user-specific virtual planes;

using the sensed positional information of the hand, determining from a plurality of zones defined for the monitored region, a zone in which the portion of the hand is present at the time the first finger state and the second finger state are detected; and

determining that the portion of the hand was present in a zone specific to entering content.

18. The system of claim 14, wherein the image analyzer is further configured to:

determine the command to indicate movement of a displayed object on a display responsive to;

determining that at least one of the first finger state and the second finger state indicates that the corresponding finger moved along a surface of the corresponding user-specific virtual plane;

using the sensed positional information of the hand, determining from a plurality of zones defined for the monitored region, a zone in which the portion of the hand is present at the time the first finger state and the second finger state are detected; and

determining that the portion of the hand was in a zone specific to entering content.

19. The system of claim 18, wherein the image analyzer is further configured to:

detect that a finger of the first finger or the second finger penetrated the corresponding user-specific virtual plane preceding detection that the finger state associated with the finger detected motion along the surface of the corresponding user-specific virtual plane, and

determine the command to indicate dragging of the displayed object across the display.

20. A non-transitory computer-readable storage medium storing instructions which when executed by a processor cause the processor to:

track, using a 3D sensor, user movements including sensing positional information of a portion of a hand in a monitored region of space monitored by the 3D sensor;

using the sensed positional information of the portion of the hand, define a plurality of distinct user-specific virtual planes, including at least a first user-specific virtual plane defined in space relative to a position of, and corresponding to, a first finger of the hand and a second user-specific virtual plane defined in space relative to a position of, and corresponding to, a second finger of the hand, in the monitored region of space;

detect, by the 3D sensor, a first finger state of the first finger relative to the corresponding first user-specific virtual plane and a second finger state of the second finger relative to the corresponding second user-specific virtual plane, wherein a finger state for a finger relative to the corresponding user-specific virtual plane defined for the finger is one of;

the finger moving closer or further away from the corresponding user-specific virtual plane, and the finger moving on or against the corresponding user-specific virtual plane;

determine an input gesture made by the portion of the hand based on the first finger state and the second finger state;

interpret the input gesture as a command using the input gesture determined from the first finger state and the second finger state; and

provide the command to a machine for executing an action appropriate to the command.