Touch-free gesture recognition system and method

• US 10,606,442 B2
• Filed: 08/14/2017
• Issued: 03/31/2020
• Est. Priority Date: 08/04/2011
• Status: Active Grant

• Current Assignee: eyeSight Mobile Technologies Ltd.
• Original Assignee: eyeSight Mobile Technologies Ltd.
• Inventors: Katz, Itay
• Primary Examiner(s): Bautista, Xiomara L
• Application Number:
  US15/675,981
• Publication Number:
  US 20170344222A1
• Time in Patent Office: 960 Days
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback

Abstract

The present invention provides a system and method for interacting with a 3D virtual image containing activatable objects. The system of the invention includes a 3D display device that presents to a user a 3D image, an image sensor and a processor. The processor analyzes images obtained by the image sensor to determine when the user has placed an activating object such as a hand or a finger, or has performed a gesture related to an activatable object, in the 3D space, at the location where the user perceives an activatable object to be located. The user thus perceives that he is “touching” the activatable object with the activating object.

Claims

1. A touch-free gesture recognition system, comprising:

at least one processor configured to;

display a virtual image that includes at least one activatable object, the activatable object having at least two activating modes, wherein a first mode is associated with a first mode of activating object, and a second mode is associated with a second mode of activating object;

correlate a location of an activating object with a location of the activatable object; and

implement a first action when the location of the activating object correlates to the location of the activatable object and the activating object performs a first type of movement, and implement a second action when the location of the activating object correlates to the location of the activatable object and the activating object performs a second type of movement.

2. The touch-free gesture recognition system of claim 1, wherein the first action includes at least one of a first message or a first command, and the second action includes at least one of a second message or a second command.

3. The touch-free gesture recognition system of claim 1, wherein the first and second modes are each associated with at least one of a left side of the activatable object, a right side of the activatable object, a bottom side of the activatable object, and a top side of the activatable object.

4. The touch-free gesture recognition system of claim 3, wherein the at least one processor is further configured to:

detect a predetermined motion associated with the activating object including at least one of a right side click, a left side click, a bottom side click, a top side click, a grasp of the activatable object, a gesture toward the activatable object from the left side, a gesture toward the activatable object from the right side, and a pass through the activatable object.

5. The touch-free gesture recognition system of claim 1, wherein the at least one processor is further configured to:

detect a voice command and select the first action and the second action, based, at least in part, on the voice command.

6. The touch-free gesture recognition system of claim 1, wherein the first and second modes are each associated with at least one of a light touch to the activatable object and a deep touch to the activatable object.

7. The touch-free gesture recognition system of claim 1, wherein the at least one processor is further configured to:

detect the activatable object based, at least in part, on eye movements of the user.

8. The touch-free gesture recognition system of claim 1, wherein the at least one processor is further configured to:

adjust the display of the virtual image as the activating object moves closer to the activatable object.

9. The touch-free gesture recognition system of claim 1, wherein, to correlate the location of the activating object with the location of the activatable object, the at least one processor is further configured to:

determine whether the activating object is in a virtual cone that has a vertex associated with the image sensor and that extends through at least a portion of the activatable object.

10. The touch-free gesture recognition system of claim 9, wherein, to correlate the location of the activating object with the location of the activatable object, the at least one processor is further configured to:

determine whether the activating object is less than a predetermined distance from at least one of the activatable object or a display device associated with the activatable object.

11. The touch-free gesture recognition system of claim 1, wherein, to correlate the location of the activating object with the location of the activatable object, the at least one processor is further configured to:

determine whether the activating object is pointing at the activatable object and is less than a predetermined distance from the activatable object or a display device associated with the activatable object.

12. The touch-free gesture recognition system of claim 1, wherein, to correlate the location of the activating object with the location of the activatable object, the at least one processor is further configured to:

determine whether the activating object is performing a predetermined gesture and is less than a predetermined distance from the activatable object from a display device associated with the activatable object.

13. A non-transitory computer-readable medium including instructions that, when executed by at least one processor, cause the processor to perform operations, comprising:

displaying a virtual image that includes at least one activatable object, the activatable object having at least two activating modes simultaneously activatable, wherein a first mode is associated with a first mode of activating object and a second mode is associated with a second mode of activating object;

correlating a location of the activating object with a location of the activatable object; and

implementing a first action when the location of the activating object correlates to the activatable object and the activating object performs a first type of movement, and implementing a second action when the location of the activating object correlates to the activatable object and the activating object performs a second type of movement.

14. The non-transitory computer-readable medium of claim 13, wherein the first action includes at least one of a first message or a first command, and the second action includes at least one of a second message or a second command.

15. The non-transitory computer-readable medium of claim 13, wherein the instructions, when executed by at least one processor, further cause the processor to perform at least one operation comprising:

detecting a voice command; and

selecting at least one of the first action and the second action, based, at least in part, on the voice command.

16. The non-transitory computer-readable medium of claim 13, wherein the first and second activating modes are each associated with at least one of a light virtual touch to the activatable object and a deep virtual touch to the activatable object.

17. The non-transitory computer-readable medium of claim 13, wherein the instructions, when executed by at least one processor, further cause the processor to perform at least one operation comprising:

detecting the activatable object based, at least in part, on an eye movement of the user.

18. The non-transitory computer-readable medium of claim 13, wherein the instructions, when executed by at least one processor, further cause the processor to perform at least one operation comprising:

adjusting the display of the virtual image as the activating object moves closer to the activatable object.

19. The non-transitory computer-readable medium of claim 13, wherein, to correlate the location of the activating object with the location of the activatable object, the instructions, when executed by at least one processor, further cause the processor to perform at least one operation comprising:

determining whether the activating object is pointing at the activatable object and is less than a predetermined distance from the activatable object or from a display device associated with the activatable object.

20. A touch-free gesture recognition method, comprising:

displaying a virtual image that includes at least one activatable object, the activatable object having at least two

activating modes simultaneously, wherein a first mode is associated with a first mode of activating object, and a second mode is associated with a second mode of activating object;

correlating a location of the activating object with a location of the activatable object; and

implementing a first action when the location of the activating object correlates to the activatable object and the activating object performs a first type of movement, and implementing a second action when the location of the activating object correlates to the activatable object and the activating object performs a second type of movement.