System, method and apparatus of eye tracking or gaze detection applications including facilitating action on or interaction with a simulated object

• US 10,127,735 B2
• Filed: 01/05/2018
• Issued: 11/13/2018
• Est. Priority Date: 05/01/2012
• Status: Active Grant

• Current Assignee: Augmented Reality Holdings, LLC
• Original Assignee: Augmented Reality Holdings, LLC
• Inventors: Spivack, Nova
• Primary Examiner(s): Gray, Ryan M
• Application Number:
  US15/863,698
• Publication Number:
  US 20180158248A1
• Time in Patent Office: 312 Days
• Field of Search: None
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06Q 30/0623 Item investigation
• CPC Class Codes: G06Q 30/0639 Item locations
• CPC Class Codes: G06Q 30/0641 Shopping interfaces
• CPC Class Codes: G06Q 30/0643 Graphical representation of items or shoppers
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G09G 5/377 for mixing or overlaying two or more graphic patterns G09G5/02, G09G5/397 take precedence

Abstract

Techniques are disclosed for facilitating action by a user on a simulated object in an augmented reality environment. In some embodiments, a method includes, detecting a gesture of the user in a real environment via a sensor of the device; wherein, the gesture includes, movement of eye ball or eye focal point of one or more eyes of the user. The gesture can be detected by tracking: a movement of one or more eyes of the user, a non-movement of one or more eyes of the user, a location of a focal point of one or more eyes of the user, and/or a movement of an eye lid of one or more eyes of the user. The gesture can be captured to implement the action on the simulated object in the augmented reality environment.

Claims

1. A method to facilitate action by a user via a device on a simulated object in an augmented reality environment, the method comprising:

detecting a gesture of the user in a real environment via a sensor of the device;

wherein, the gesture includes, one or more of;

movement of one or more eyes of the user, and

movement of eye focal point of the one or more eyes of the user;

capturing the gesture to implement the action on the simulated object in the augmented reality environment;

wherein, the gesture is detected based on a given speed or velocity of the movement of the one or more eyes while the one or more eyes is in motion relative to the user;

wherein, the action to be performed on the simulated object is based on the gesture that is detected;

wherein, the gesture includes locating the eye focal point of the one or more eyes of the user to target the simulated object that is in the eye focal point;

wherein, the eye focal point is determined from the given speed or velocity.

2. The method of claim 1,

wherein, the user is associated with a physical location of the real environment;

wherein, the simulated object is associated with one or more of;

the physical location, location ranges in or near the physical location, a physical entity in or near the physical location, and another user in or near the physical location.

3. The method of claim 1,

wherein, the user is associated with a physical location of the real environment;

wherein, the simulated object includes information or data associated with one or more of;

the physical location, location ranges in or near the physical location, a physical entity in or near the physical location, and another user in or near the physical location.

4. The method of claim 1, wherein the gesture is detected via tracking by the sensor, of one or more of:

a movement of the one or more eyes of the user,

a direction of the movement of the one or more eyes of the user,

a non-movement of the one or more eyes of the user,

a location of the eye focal point of the one or more eyes of the user.

5. The method of claim 1, wherein the gesture is detected via tracking by the sensor, a movement of an eye lid of the one or more eyes of the user.

6. The method of claim 1, wherein, the gesture further includes, blinking of the one or more eyes of the user.

7. The method of claim 6, wherein, the action associated with the blinking of the one or more of the eyes of the user includes selection of the simulated object in the eye focal point of the one or more eyes.

8. The method of claim 6, wherein, the action associated with the blinking of the one or more eyes of the user includes one or more of, taking a photo in the augmented reality environment and selecting a button in the augmented reality environment.

9. The method of claim 1, wherein the action includes, one or more of:

a selection mechanism of the simulated object and pointing to the simulated object.

10. The method of claim 1, wherein, the action includes tracing an outline of the simulated object.

11. The method of claim 1, wherein, the sensor includes, one or more of:

an image sensor, a motion sensor, and a user stimulus sensor.

12. The method of claim 1, further comprising, performing search based on the simulated object that is in the eye focal point.

13. The method of claim 1, wherein, the gesture further includes a movement of, one or more of, a hand or a finger of the user.

14. The method of claim 1, wherein, the gesture further includes a movement of, one or more of, a head, an arm, a limb and a torso of the user.

15. The method of claim 1,

wherein, the simulated object includes an advertisement relevant to a physical entity or physical object in a physical location of the real environment.

16. The method of claim 1, wherein the simulated object that is targeted in the eye focal point is depicted in the augmented reality environment as a result of a search query.

17. A system to enable interaction with a simulated object by a user in a simulated environment, the system comprising:

a processor;

a memory having stored thereon instructions which, when executed by the processor, cause the system to;

detect a gesture of the user in a real environment;

wherein, the gesture includes, one or more of;

movement of eye ball motion of one or more eyes of the user, and

movement of eye focal point of the one or more eyes of the user;

capture the gesture to facilitate the interaction with the simulated object in the simulated environment;

wherein, the gesture is captured using a given speed or velocity of the movement of the one or more eyes while the one or more eyes is in motion relative to the user;

wherein, the interaction with the simulated objected that is enabled in the simulated environment is based on the gesture of the user;

wherein, the gesture includes locating the eye focal point of the one or more eyes of the user to target the simulated object that is in the eye focal point;

wherein, the eye focal point is determined from the given speed or velocity.

18. The system of claim 17, wherein the simulated object includes multimedia content.

19. The system of claim 17:

wherein the simulated object represents or is associated with a physical entity or physical object in a physical location of the real environment;

wherein, the physical entity or the physical object represented by or associated with the simulated object has been recognized or identified through recognition of the physical object or the physical entity using a recognition pattern, or identified through detection of physical characteristics of the physical object or the physical entity.

20. The system of claim 17:

wherein search is performed based on the simulated object that is in the eye focal point.

21. The system of claim 17:

wherein the simulated object represents or is associated with a physical object in a physical location of the real environment;

wherein, the physical object represented by or associated with the simulated object has been recognized through detection or identification of the physical object using one or more or, an augmented reality marker, a barcode and an RFID code.

22. The system of claim 17, wherein the gesture is detected via tracking, one or more of:

a movement of the one or more eyes of the user,

a non-movement of the one or more eyes of the user,

a location of the eye focal point of the one or more eyes of the user, and

a movement of an eye lid of the one or more eyes of the user.

23. The system of claim 17, wherein, the gesture further includes, blinking of the one or more eyes of the user.

24. The system of claim 17:

wherein the simulated object that is in the eye focal point is depicted in the simulated environment as a result of a search query.

25. The system of claim 17:

wherein search is performed based on the simulated object that is in the eye focal point;

wherein, advertisements are identified or detected based on the search that is performed.

26. An apparatus which facilitates action by a user on a virtual object in a digital environment, the apparatus comprising:

a sensor which detects a gesture of the user in a real environment;

wherein, the gesture includes, one or more of;

movement of eye ball of one or more eyes of the user, and

movement of eye focal point of the one or more eyes of the user;

the gesture being captured to implement the action to be performed on the virtual object in the digital environment;

wherein the gesture is detected based on a given speed or velocity of the movement of the one or more eyes while the one or more eyes is in motion relative to the user;

wherein, the action to be performed on the virtual object is based on the gesture that is detected;

wherein, the gesture includes locating the eye focal point of the one or more eyes of the user to target the virtual object that is in the eye focal point;

wherein, the eye focal point is determined from the given speed or velocity of the movement of the one or more eyes.

27. The apparatus of claim 26, further comprising:

an output module to present an updated rendering of the virtual object resulting from the action being performed on the virtual object

wherein, the output module includes, one or more of, an eye piece, goggles, or a mobile display.

28. The apparatus of claim 26:

wherein the virtual object that is in the eye focal point is depicted in the digital environment as a result of a search query.

29. The apparatus of claim 26:

wherein search is performed based on the virtual object that is in the eye focal point.

30. A non-transitory machine-readable storage medium, having stored thereon instructions which when executed by a processor, cause the processor to perform, a method to facilitate a transaction via an online environment, of a physical product in a real environment that is represented by a simulated object, the method comprising:

capturing a gesture performed with respect to the simulated object;

wherein, the gesture is performed by a user of the real environment to initiate the transaction of the physical product associated with the simulated object in the online environment;

wherein, the gesture performed by the user with respect to the simulated object, includes movement of one or more eyes of the user;

wherein, the gesture is captured using a given speed or velocity of the movement of the one or more eyes while the one or more eyes is in motion relative to the user;

wherein, the gesture includes locating an eye focal point of the one or more eyes of the user to target the simulated object that is in the eye focal point;

wherein, the eye focal point is determined from the given speed or velocity;

conducting the transaction of the physical product in the online environment, according to the gesture performed by the user with respect to the simulated object associated with the physical product.