Interfacing with a device via virtual 3D objects

• US 9,733,789 B2
• Filed: 08/02/2012
• Issued: 08/15/2017
• Est. Priority Date: 08/04/2011
• Status: Active Grant

• Current Assignee: eyeSight Mobile Technologies Ltd.
• Original Assignee: eyeSight Mobile Technologies Ltd.
• Inventors: Katz, Itay
• Primary Examiner(s): BAUTISTA, XIOMARA L
• Application Number:
  US14/236,492
• Publication Number:
  US 20150363070A1
• Time in Patent Office: 1,839 Days
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback

Abstract

The present invention provides a system and method for interacting with a 3D virtual image containing activatable objects. The system of the invention includes a 3D display device that presents to a user a 3D image, an image sensor and a processor. The processor analyzes images obtained by the image sensor to determine when the user has placed an activating object such as a hand or a finger, or has performed a gesture related to an activatable object, in the 3D space, at the location where the user perceives an activatable object to be located. The user thus perceives that he is “touching” the activatable object with the activating object.

Claims

1. A touch-free gesture recognition system, comprising:

at least one processor configured to;

display a virtual 3D image that includes at least one activatable object, the activatable object having at least two distinct sides that are simultaneously activatable, wherein a first distinct side of the at least two distinct sides is associated with a first action, and a second distinct side of the at least two distinct sides is associated with a second action;

receive, from at least one image sensor, at least one image including a depiction of an activating object associated with a user;

correlate a location of the activating object with a location of one of the first and second distinct sides; and

implement the first action when the location of the activating object correlates to the first distinct side and implement the second action when the location of the activating object correlates to the second distinct side.

2. The touch-free gesture recognition system of claim 1, wherein the first action includes at least one of a first message or a first command, and the second action includes at least one of a second message or a second command.

3. The touch-free gesture recognition system of claim 1, wherein the first and second distinct sides are each associated with at least one of a left side of the activatable object, a right side of the activatable object, a bottom side of the activatable object, and a top side of the activatable object.

4. The touch-free gesture recognition system of claim 3, wherein the at least one processor is further configured to:

detect a predetermined motion associated with the activating object including at least one of a right side click, a left side click, a bottom side click, a top side click, a grasp of the activatable object, a gesture toward the activatable object from the left side, a gesture toward the activatable object from the right side, and a pass through the activatable object.

5. The touch-free gesture recognition system of claim 1, wherein the at least one processor is further configured to:

detect a voice command and select the first action and the second action, based, at least in part, on the voice command.

6. The touch-free gesture recognition system of claim 1, wherein the first and second distinct sides are each associated with at least one of a light touch to the activatable object and a deep touch to the activatable object.

7. The touch-free gesture recognition system of claim 1, wherein the at least one processor is further configured to:

detect the activatable object based, at least in part, on eye movements of the user.

8. The touch-free gesture recognition system of claim 1, wherein the at least one processor is further configured to:

adjust the display of the virtual 3D image as the activating object moves closer to the activatable object.

9. The touch-free gesture recognition system of claim 1, wherein, to correlate the location of the activating object with one of the first and second distinct sides, the at least one processor is further configured to:

determine whether the activating object is in a virtual cone that has a vertex associated with the image sensor and that extends through at least a portion of the activatable object.

10. The touch-free gesture recognition system of claim 9, wherein, to correlate the location of the activating object with one of the first and second distinct sides, the at least one processor is further configured to:

determine whether the activating object is less than a predetermined distance from at least one of the activatable object or a display device associated with the activatable object.

11. The touch-free gesture recognition system of claim 1, wherein, to correlate the location of the activating object with one of the first and second distinct sides, the at least one processor is further configured to:

determine whether the activating object is pointing at the activatable object and is less than a predetermined distance from the activatable object.

12. The touch-free gesture recognition system of claim 1, wherein, to correlate the location of the activating object with one of the first and second distinct sides, the at least one processor is further configured to:

determine whether the activating object is pointing at the activatable object and is less than a predetermined distance from a display device associated with the activatable object.

13. The touch-free gesture recognition system of claim 1, wherein, to correlate the location of the activating object with one of the first and second distinct sides, the at least one processor is further configured to:

determine whether the activating object is performing a predetermined gesture and is less than a predetermined distance from the activatable object.

14. The touch-free gesture recognition system of claim 1, wherein, to correlate the location of the activating object with one of the first and second distinct sides, the at least one processor is further configured to:

determine whether the activating object is performing a predetermined gesture and is less than a predetermined distance from a display device associated with the activatable object.

15. A non-transitory computer-readable medium including instructions that, when executed by at least one processor, cause the processor to perform operations, comprising:

displaying a virtual 3D image that includes at least one activatable object, the activatable object having at least two distinct sides that are simultaneously activatable, wherein a first distinct side of the at least two distinct sides is associated with a first action and a second distinct side of the at least two distinct sides is associated with a second action;

receiving, from at least one image sensor, at least one image including a depiction of an activating object associated with a user;

correlating a location of the activating object with a location of one of the first and second distinct sides;

if the location of the activating object correlates to the first distinct side, implementing the first action, and

if the location of the activating object correlates to the second distinct side, implementing the second action.

16. The non-transitory computer-readable medium of claim 15, wherein the first action includes at least one of a first message or a first command, and the second action includes at least one of a second message or a second command.

17. The non-transitory computer-readable medium of claim 15, wherein the first and second distinct sides are each associated with at least one of a left side of the activatable object, a right side of the activatable object, a bottom side of the activatable object, and a top side of the activatable object.

18. The non-transitory computer-readable medium of claim 17, wherein the instructions, when executed by at least one processor, further cause the processor to perform at least one operation comprising:

detecting a predetermined motion associated with the activating object including at least one of a right side click, a left side click, a bottom side click, a top side click, a grasp of the activatable object, a gesture toward the activatable object from the left side, a gesture toward the activatable object from the right side, and a pass through the activatable object.

19. The non-transitory computer-readable medium of claim 15, wherein the instructions, when executed by at least one processor, further cause the processor to perform at least one operation comprising:

detecting a voice command; and

selecting at least one of the first action and the second action, based, at least in part, on the voice command.

20. The non-transitory computer-readable medium of claim 15, wherein the first and second distinct sides are each associated with at least one of a light virtual touch to the activatable object and a deep virtual touch to the activatable object.

21. The non-transitory computer-readable medium of claim 15, wherein the instructions, when executed by at least one processor, further cause the processor to perform at least one operation comprising:

detecting the activatable object based, at least in part, on an eye movement of the user.

22. The non-transitory computer-readable medium of claim 15, wherein the instructions, when executed by at least one processor, further cause the processor to perform at least one operation comprising:

adjusting the display of the virtual 3D image as the activating object moves closer to the activatable object.

23. The non-transitory computer-readable medium of claim 15, wherein, to correlate the location of the activating object with one of the first and second distinct sides, the instructions, when executed by at least one processor, further cause the processor to perform at least one operation comprising:

determining whether the activating object is in a virtual cone that has a vertex associated with the image sensor and that extends through at least a portion of the activatable object.

24. The non-transitory computer-readable medium of claim 23, wherein, to correlate the location of the activating object with one of the first and second distinct sides, the instructions, when executed by at least one processor, further cause the processor to perform at least one operation comprising:

determining whether the activating object is less than a predetermined distance from at least one of the activatable object or a display device associated with the activatable object.

25. The non-transitory computer-readable medium of claim 15, wherein, to correlate the location of the activating object with one of the first and second distinct sides, the instructions, when executed by at least one processor, further cause the processor to perform at least one operation comprising:

determining whether the activating object is pointing at the activatable object and is less than a predetermined distance from the activatable object.

26. The non-transitory computer-readable medium of claim 15, wherein, to correlate the location of the activating object with one of the first and second distinct sides, the instructions, when executed by at least one processor, further cause the processor to perform at least one operation comprising:

determining whether the activating object is pointing at the activatable object and is less than a predetermined distance from a display device associated with the activatable object.

27. The non-transitory computer-readable medium of claim 15, wherein, to correlate the location of the activating object with one of the first and second distinct sides, the instructions, when executed by at least one processor, further cause the processor to perform at least one operation comprising:

determining whether the activating object is performing a predetermined gesture and is less than a predetermined distance from the activatable object.

28. The non-transitory computer-readable medium of claim 15, wherein, to correlate the location of the activating object with one of the first and second distinct sides, the instructions, when executed by at least one processor, further cause the processor to perform at least one operation comprising:

determining whether the activating object is performing a predetermined gesture and is less than a predetermined distance from a display device associated with the activatable object.

29. A touch-free gesture recognition method, comprising:

displaying a virtual 3D image that includes at least one activatable object, the activatable object having at least two distinct sides that are simultaneously activatable, wherein a first distinct side of the at least two distinct sides is associated with a first action and a second distinct side of the at least two distinct sides is associated with a second action;

receiving, from at least one image sensor, at least one image including a depiction of an activating object associated with a user;

correlating a location of the activating object with a location of one of the first and second distinct sides; and

if the location of the activating object correlates to the first distinct side, implementing the first action, and

if the location of the activating object correlates to the second distinct side, implementing the second action.

30. The touch-free gesture recognition method of claim 29, wherein the first and second distinct virtual sides are each associated with at least one of a left side of the activatable object, a right side of the activatable object, a bottom side of the activatable object, and a top side of the activatable object.