Enabling augmented reality using eye gaze tracking

• US 10,474,233 B2
• Filed: 05/02/2018
• Issued: 11/12/2019
• Est. Priority Date: 12/19/2012
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Swaminathan, Ashwin, Ramachandran, Mahesh
• Primary Examiner(s): Nguyen, Chanh D
• Assistant Examiner(s): Truong, Nguyen H
• Application Number:
  US15/969,430
• Publication Number:
  US 20180253145A1
• Time in Patent Office: 559 Days
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06F 3/147 using display panels
• CPC Class Codes: G06V 10/25 Determination of region of interest [ROI] or a volume of interest [VOI]
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 30/142 using hand-held instruments; Constructional details of the instruments
• CPC Class Codes: G06V 40/19 Sensors therefor
• CPC Class Codes: G09G 2340/12 Overlay of images, i.e. displayed pixel being the result of switching between the corresponding input pixels

Abstract

Methods and apparatus relating to enabling augmented reality applications using eye gaze tracking are disclosed. An exemplary method according to the disclosure includes displaying an image to a user of a scene viewable by the user, receiving information indicative of an eye gaze of the user, determining an area of interest within the image based on the eye gaze information, determining an image segment based on the area of interest, initiating an object recognition process on the image segment, and displaying results of the object recognition process.

Claims

1. A method for providing augmented reality information, comprising:

displaying, on a display screen, a first image of a scene viewable by a user, the first image including an object and a virtual object tag associated with the object, the object being displayed at a first position relative to the display screen and the virtual object tag being located at a second position relative to the first position of the object, wherein a position of the virtual object tag is configured to change as a position of the object changes, the virtual object tag maintaining a distance relative to the object as the position of the object changes;

determining the object has been selected as a target object;

displaying, on the display screen, a second image of the scene, the second image including the object and the virtual object tag, the object being displayed in the second image at a third position relative to the display screen, wherein the virtual object tag automatically remains fixed in the second image at the second position relative to the first position of the object in the first image based on determining the object has been selected as the target object, and wherein the virtual object tag remains fixed at the second position in one or more images until an additional target object is detected in the one or more images;

determining whether the user has selected the virtual object tag; and

hiding the virtual object tag if it is determined that the user has not selected the virtual object tag.

2. The method of claim 1, wherein the virtual object tag indicates that augmented reality information related to the object is available.

3. The method of claim 1, further comprising:

determining the user has selected the virtual object tag; and

displaying, on the display screen, augmented reality information related to the object in response to determining the user has selected the virtual object tag.

4. The method of claim 3, wherein determining the user has selected the virtual object tag includes determining a gaze of the user is directed on or about the virtual object tag at the second position for a threshold amount of time.

5. The method of claim 3, wherein determining the user has selected the virtual object tag includes determining the user has selected a graphical element associated with the virtual object tag.

6. The method of claim 1, wherein the virtual object tag remains fixed at the second position in the one or more images for a duration of time based on a preset time period.

7. The method of claim 1, wherein the virtual object tag remains fixed at the second position in the one or more images as long as a gaze of the user is directed at the display screen.

8. The method of claim 1, wherein the virtual object tag is displayed at the second position for a plurality of images of the scene as the object associated with the virtual object tag is displayed at different positions in the plurality of images.

9. The method of claim 1, wherein the second position is based on a position of the object when the virtual object tag is initially displayed.

10. An apparatus for providing augmented reality information, comprising:

a display screen;

a memory configured to store images; and

a processor configured to;

display, on the display screen, a first image of a scene viewable by a user, the first image including an object and a virtual object tag associated with the object, the object being displayed at a first position relative to the display screen and the virtual object tag being displayed at a second position relative to the first position of the object, wherein a position of the virtual object tag is configured to change as a position of the object changes, the virtual object tag maintaining a distance relative to the object as the position of the object changes;

determine the object has been selected as a target object;

display, on the display screen, a second image of the scene, the second image including the object and the virtual object tag, the object being displayed in the second image at a third position relative to the display screen, wherein the virtual object tag automatically remains fixed in the second image at the second position relative to the first position of the object in the first image based on determining the object has been selected as the target object, and wherein the virtual object tag remains fixed at the second position in one or more images until an additional target object is detected in the one or more images;

determine whether the user has selected the virtual object tag; and

hide the virtual object tag if it is determined that the user has not selected the virtual object tag.

11. The apparatus of claim 10, wherein the virtual object tag indicates that augmented reality information related to the object is available.

12. The apparatus of claim 10, wherein the processor is further configured to:

determine the user has selected the virtual object tag; and

display, on the display screen, augmented reality information related to the object in response to determining the user has selected the virtual object tag.

13. The apparatus of claim 12, wherein determining the user has selected the virtual object tag includes determining a gaze of the user is directed on or about the virtual object tag at the second position for a threshold amount of time.

14. The apparatus of claim 12, wherein determining the user has selected the virtual object tag includes determining the user has selected a graphical element associated with the virtual object tag.

15. The apparatus of claim 10, wherein the virtual object tag remains fixed at the second position in the one or more images for a duration of time based on a preset time period.

16. The apparatus of claim 10, wherein the virtual object tag remains fixed at the second position in the one or more images as long as a gaze of the user is directed at the display screen.

17. The apparatus of claim 10, wherein the virtual object tag is displayed at the second position for a plurality of images of the scene as the object associated with the virtual object tag is displayed at different positions in the plurality of images.

18. The apparatus of claim 10, wherein the second position is based on a position of the object when the virtual object tag is initially displayed.

19. The apparatus of claim 10, further comprising a camera for capturing one or more images.

20. The apparatus of claim 10, further comprising a display for displaying one or more images.

21. The apparatus of claim 10, wherein the apparatus comprises a mobile device with a camera for capturing one or more images and a display for displaying the one or more images.

22. A non-transitory computer-readable medium having stored thereon instructions that, when executed by one or more processors, cause the one or more processors to:

display, on a display screen, a first image of a scene viewable by a user, the first image including an object and a virtual object tag associated with the object, the object being displayed at a first position relative to the display screen and the virtual object tag being displayed at a second position relative to the first position of the object, wherein a position of the virtual object tag is configured to change as a position of the object changes, the virtual object tag maintaining a distance relative to the object as the position of the object changes;

determine the object has been selected as a target object;

display, on the display screen, a second image of the scene, the second image including the object and the virtual object tag, the object being displayed in the second image at a third position relative to the display screen, wherein the virtual object tag automatically remains fixed in the second image at the second position relative to the first position of the object in the first image based on determining the object has been selected as the target object, and wherein the virtual object tag remains fixed at the second position in one or more images until an additional target object is detected in the one or more images;

determine whether the user has selected the virtual object tag; and

hide the virtual object tag if it is determined that the user has not selected the virtual object tag.

23. The non-transitory computer-readable medium of claim 22, wherein the virtual object tag indicates that augmented reality information related to the object is available.

24. The non-transitory computer-readable medium of claim 22, further comprising instructions that, when executed by the one or more processors, cause the one or more processors to:

determine the user has selected the virtual object tag; and

display, on the display screen, augmented reality information related to the object in response to determining the user has selected the virtual object tag.

25. The non-transitory computer-readable medium of claim 24, wherein determining the user has selected the virtual object tag includes determining a gaze of the user is directed on or about the virtual object tag at the second position for a threshold amount of time.

26. The non-transitory computer-readable medium of claim 24, wherein determining the user has selected the virtual object tag includes determining the user has selected a graphical element associated with the virtual object tag.

27. The non-transitory computer-readable medium of claim 22, wherein the virtual object tag remains fixed at the second position in the one or more images for a duration of time based on a preset time period.

28. The non-transitory computer-readable medium of claim 22, wherein the virtual object tag remains fixed at the second position in the one or more images as long as a gaze of the user is directed at the display screen.

29. The non-transitory computer-readable medium of claim 22, wherein the virtual object tag is displayed at the second position for a plurality of images of the scene as the object associated with the virtual object tag is displayed at different positions in the plurality of images.

30. The non-transitory computer-readable medium of claim 22, wherein the second position is based on a position of the object when the virtual object tag is initially displayed.