System and Method for Indicating a Presence of Supplemental Information in Augmented Reality

• US 20140267010A1
• Filed: 03/15/2013
• Published: 09/18/2014
• Est. Priority Date: 03/15/2013
• Status: Active Grant

• Current Assignee: Malikie Innovations Limited (Key Patent Innovations Limited)
• Original Assignee: Blackberry Limited
• Inventors: Bos, Jeffrey Charles, PASQUERO, Jerome
• Granted Patent: US 9,685,001 B2
• Time in Patent Office: Days
• US Class Current: 345/156
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

A method and system are provided for indicating a presence of supplemental information in augmented reality to a user. The method includes capturing a field of view of a camera, obtaining supplemental information for at least one object in the captured field of view, displaying the captured field of view on a display and tracking a point of regard of the user. The point of regard is indicative of an area on the display at which the gaze of the user is focused. The method also includes, for each object associated with supplemental information, displaying, overlaid the captured field of view, a respective indicator that the supplemental information is associated with the object if one or more criteria are satisfied. The one or more criteria are based on at least a proximity between an image of the object in the captured field of view and the point of regard.

Claims

1. A method of indicating a presence of supplemental information in augmented reality to a user of an electronic device, the electronic device comprising a camera and a display, the method comprising:

capturing a field of view of the camera;

obtaining supplemental information for at least one object in the captured field of view;

displaying the captured field of view on the display;

tracking a point of regard of the user, the point of regard being indicative of an area on the display at which the gaze of the user is focused; and

for each object associated with supplemental information;

displaying, overlaid the captured field of view, a respective indicator if one or more criteria are satisfied, the indicator being indicative that the supplemental information is associated with the object, the one or more criteria being based on at least a proximity between an image of the object in the captured field of view and the point of regard.

2. The method of claim 1, wherein displaying the indicator comprises any one or more of:

displaying an element adjacent to the image of the respective object; and

modifying the display of the image of the respective object.

3. The method of claim 1, wherein the indicator has a visibility based on a distance between the image of the respective object and the point of regard.

4. The method of claim 3, wherein the visibility of the indicator is inversely proportional to the distance.

5. The method of claim 1, further comprising, in response to tracking that the point of regard has moved to the indicator, displaying the supplemental information associated with the respective object of the indicator.

6. The method of claim 1, wherein the one or more criteria are further based on any one or more of:

a property of the supplemental information associated with the object;

a period of time that the point of regard remains in a particular area; and

a distance between the object and the electronic device.

7. The method of claim 6, wherein the property of the supplemental information comprises any one or more of:

a source of the supplemental information;

when the supplemental information was last accessed; and

when the supplemental information was last updated.

8. The method of claim 1, wherein tracking the point of regard comprises eye tracking.

9. A computer readable storage medium for indicating a presence of supplemental information in augmented reality to a user of an electronic device, the electronic device comprising a camera and a display, the computer readable storage medium comprising computer executable instructions for:

capturing a field of view of the camera;

obtaining supplemental information for at least one object in the captured field of view;

displaying the captured field of view on the display;

tracking a point of regard of the user, the point of regard being indicative of an area on the display at which the gaze of the user is focused; and

for each object associated with supplemental information;

displaying, overlaid the captured field of view, a respective indicator if one or more criteria are satisfied, the indicator being indicative that the supplemental information is associated with the object, the one or more criteria being based on at least a proximity between an image of the object in the captured field of view and the point of regard.

10. An electronic device comprising a processor coupled to memory, a camera and a display, the memory storing computer executable instructions for:

capturing a field of view of the camera;

obtaining supplemental information for at least one object in the captured field of view;

displaying the captured field of view on the display;

tracking a point of regard of the user, the point of regard being indicative of an area on the display at which the gaze of the user is focused; and

for each object associated with supplemental information;

displaying, overlaid the captured field of view, a respective indicator if one or more criteria are satisfied, the indicator being indicative that the supplemental information is associated with the object, the one or more criteria being based on at least a proximity between an image of the object in the captured field of view and the point of regard.

11. A method of indicating a presence of supplemental information in augmented reality to a user of a head mounted display, the method comprising:

capturing a field of view of the user through the head mounted display;

obtaining supplemental information for at least one object in the field of view;

tracking a point of regard of the user, the point of regard being indicative of an area in the field of view at which the gaze of the user is focused; and

for each object associated with supplemental information;

displaying, overlaid the field of view, a respective indicator if one or more criteria are satisfied, the indicator being indicative that the supplemental information is associated with the object, the one or more criteria being based on at least a proximity between the object and the point of regard.

12. The method of claim 11, wherein displaying the indicator comprises any one or more of:

displaying an element adjacent to the respective object; and

modifying the display of the respective object.

13. The method of claim 11, wherein the indicator has a visibility based on a distance between the respective object and the point of regard.

14. The method of claim 13, wherein the visibility of the indicator is inversely proportional to the distance.

15. The method of claim 11, further comprising, in response to tracking that the point of regard has moved to the indicator, displaying the supplemental information associated with the respective object of the indicator.

16. The method of claim 11, wherein the one or more criteria are further based on any one or more of:

a property of the supplemental information associated with the object;

a period of time that the point of regard remains in a particular area; and

a distance between the object and the head mounted display.

17. The method of claim 16, wherein the property of the supplemental information comprises any one or more of:

a source of the supplemental information;

when the supplemental information was last accessed; and

when the supplemental information was last updated.

18. The method of claim 11, wherein tracking the point of regard comprises eye tracking.

19. A computer readable storage medium for indicating a presence of supplemental information in augmented reality to a user of a head mounted display, the computer readable storage medium comprising computer executable instructions for:

capturing a field of view of the user through the head mounted display;

obtaining supplemental information for at least one object in the field of view;

tracking a point of regard of the user, the point of regard being indicative of an area in the field of view at which the gaze of the user is focused; and

for each object associated with supplemental information;

displaying, overlaid the field of view, a respective indicator if one or more criteria are satisfied, the indicator being indicative that the supplemental information is associated with the object, the one or more criteria being based on at least a proximity between the object and the point of regard.

20. An electronic device comprising a head mounted display and a processor coupled to memory, the memory storing computer executable instructions for:

capturing a field of view of the user through the head mounted display;

obtaining supplemental information for at least one object in the field of view;

tracking a point of regard of the user, the point of regard being indicative of an area in the field of view at which the gaze of the user is focused; and

for each object associated with supplemental information;

displaying, overlaid the field of view, a respective indicator if one or more criteria are satisfied, the indicator being indicative that the supplemental information is associated with the object, the one or more criteria being based on at least a proximity between the object and the point of regard.