Apparatus and method for controlling an augmented reality device

• US 10,359,841 B2
• Filed: 01/13/2013
• Issued: 07/23/2019
• Est. Priority Date: 01/13/2013
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Liu, Kexi, Kim, Frederick D., Renschler, Martin H.
• Primary Examiner(s): Yang, Ryan R
• Application Number:
  US13/740,256
• Publication Number:
  US 20140198129A1
• Time in Patent Office: 2,382 Days
• Field of Search: 345633
• CPC Class Codes: G02B 2027/0127 comprising devices increasing the depth of field
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

An apparatus, a method, and a computer program product are provided. The apparatus detects an eye gaze on a first region in a real world scene, sets a boundary that surrounds the first region, the boundary excluding at least a second region in the real world scene, performs an object recognition procedure on the first region within the boundary, and refrains from performing an object recognition procedure on the at least the second region.

Claims

1. A method of controlling an optical device, the method comprising:

detecting, via the optical device, an eye gaze of a user on a first region in a real world scene;

setting a boundary that surrounds the first region, the boundary excluding at least a second region in the real world scene;

performing an object recognition procedure on the first region within the boundary for one or more of a plurality of image frames of the real world scene to identify an object within the first region;

refraining from performing an object recognition procedure on the at least the second region;

displaying digital content associated with the object within the eye gaze, wherein the digital content is displayed while the eye gaze remains on the first region; and

displaying a photo editing tool via the optical device while the user gazes at the real world scene via the optical device prior to capturing a photo of at least a portion of the real world scene.

2. The method of claim 1, further comprising:

detecting the eye gaze on a third region in the real world scene; and

setting the boundary to surround the third region while excluding at least a fourth region in the real world scene.

3. The method of claim 1, further comprising:

determining a first count of consecutive times the object has been recognized by the object recognition procedure within the boundary or a second count of a number of times the object has been recognized within the boundary;

highlighting the object when the first count or the second count exceeds a threshold; and

displaying an icon proximate to the object when the first count or the second count exceeds the threshold.

4. The method of claim 3, wherein the threshold is dynamically set based on at least one of a type of object, a history, and a user preference.

5. The method of claim 1, wherein if no object has been recognized in the first region within the boundary, the method further comprises performing at least one of an operation to zoom into a center of the first region within the boundary, an operation to enlarge the boundary, or an operation to enter a power saving state.

6. The method of claim 1, further comprising:

receiving at least one of a voice command and a gesture command; and

adjusting a magnification of the first region within the boundary in response to the at least one of the voice command and the gesture command.

7. The method of claim 1, wherein displaying digital content further includes:

editing, prior to capturing the photo of at least a portion of the real world scene, the first region within the boundary using the photo editing tool based on at least one of a voice command and a gesture command;

capturing a photo of at least the edited first region of the real world scene; and

storing the photo of the edited first region.

8. The method of claim 1, further comprising:

storing at least a first recognized object in response to at least one of a first voice command and a first gesture command;

storing at least a second recognized object in response to at least one of a second voice command and a second gesture command; and

comparing the at least the first recognized object and the second recognized object.

9. The method of claim 1, further comprising:

setting one or more updated boundaries in the one or more of the plurality of image frames based on continued detection of the eye gaze; and

performing the object recognition procedure on one or more regions surrounded by the one or more updated boundaries to identify the object within the one or more regions.

10. The method of claim 1, wherein the photo editing tool includes at least one of a filter tool, a clip art tool, a color saturation tool, a sepia toe conversion tool, and a black and white conversion tool.

11. An apparatus for controlling an optical device, the apparatus comprising:

means for detecting, via the optical device, an eye gaze of a user on a first region in a real world scene;

means for setting a boundary that surrounds the first region, the boundary excluding at least a second region in the real world scene;

means for performing an object recognition procedure on the first region within the boundary for one or more of a plurality of image frames of the real world scene to identify an object within the first region;

means for refraining from performing an object recognition procedure on the at least the second region;

means for displaying digital content associated with the object within the eye gaze, wherein the digital content is displayed while the eye gaze remains on the first region; and

means for displaying a photo editing tool via the optical device while the user gazes at the real world scene via the optical device prior to capturing a photo of at least a portion of the real world scene.

12. The apparatus of claim 11, further comprising:

means for detecting the eye gaze on a third region in the real world scene; and

means for setting the boundary to surround the third region while excluding at least a fourth region in the real world scene.

13. The apparatus of claim 11, further comprising:

means for determining a first count of consecutive times the object has been recognized by the object recognition procedure within the boundary or a second count of a number of times the object has been recognized within the boundary;

means for highlighting the object when the first count or the second count exceeds a threshold; and

means for displaying an icon proximate to the object when the first count or the second count exceeds the threshold.

14. The apparatus of claim 13, wherein the threshold is dynamically set based on at least one of a type of object, a history, and a user preference.

15. The apparatus of claim 11, further comprising means for performing at least one operation, the at least one operation comprising at least one of an operation to zoom into a center of the first region within the boundary, an operation to enlarge the boundary, or an operation to enter a power saving state if no object has been recognized in the first region within the boundary.

16. The apparatus of claim 11, further comprising:

means for receiving at least one of a voice command and a gesture command; and

means for adjusting a magnification of the first region within the boundary in response to the at least one of the voice command and the gesture command.

17. The apparatus of claim 11, wherein the means for displaying digital content is further configured to:

edit, prior to capturing the photo of at least a portion of the real world scene, the first region within the boundary using the photo editing tool based on at least one of a voice command and a gesture command;

capture a photo of at least the edited first region of the real world scene; and

store the photo of the edited first region.

18. The apparatus of claim 11, further comprising:

means for storing at least a first recognized object in response to at least one of a first voice command and a first gesture command;

means for storing at least a second recognized object in response to at least one of a second voice command and a second gesture command; and

means for comparing the at least the first recognized object and the second recognized object.

19. The apparatus of claim 11, further comprising:

means for setting one or more updated boundaries in the one or more of the plurality of image frames based on continued detection of the eye gaze; and

means for performing the object recognition procedure on one or more regions surrounded by the one or more updated boundaries to identify the object within the one or more regions.

20. The apparatus of claim 11, wherein the photo editing tool includes at least one of a filter tool, a clip art tool, a color saturation tool, a sepia toe conversion tool, and a black and white conversion tool.

21. An apparatus for controlling an optical device, the apparatus comprising:

a memory;

at least one processor coupled to the memory and configured to;

detect, via the optical device, an eye gaze of a user on a first region in a real world scene;

set a boundary that surrounds the first region, the boundary excluding at least a second region in the real world scene;

perform an object recognition procedure on the first region within the boundary for one or more of a plurality of image frames of the real world scene to identify an object within the first region;

refrain from performing an object recognition procedure on the at least the second region;

display digital content associated with the object within the eye gaze, wherein the digital content is displayed while the eye gaze remains on the first region; and

display a photo editing tool via the optical device while the user gazes at the real world scene via the optical device prior to capturing a photo of at least a portion of the real world scene.

22. The apparatus of claim 21, wherein the at least one processor is further configured to:

detect the eye gaze on a third region in the real world scene; and

set the boundary to surround the third region while excluding at least a fourth region in the real world scene.

23. The apparatus of claim 21, wherein the at least one processor is further configured to:

determine a first count of consecutive times the object has been recognized by the object recognition procedure within the boundary or a second count of a number of times the object has been recognized within the boundary;

highlight the object when the first count or the second count exceeds a threshold; and

display an icon proximate to the object when the first count or the second count exceeds the threshold.

24. The apparatus of claim 23, wherein the threshold is dynamically set based on at least one of a type of object, a history, and a user preference.

25. The apparatus of claim 21, wherein the at least one processor is further configured to perform at least one of an operation to zoom into a center of the first region within the boundary, an operation to enlarge the boundary, or an operation to enter a power saving state if no object has been recognized in the first region within the boundary.

26. The apparatus of claim 21, wherein the at least one processor is further configured to:

receive at least one of a voice command and a gesture command; and

adjust a magnification of the first region within the boundary in response to the at least one of the voice command and the gesture command.

27. The apparatus of claim 21, wherein as part of the configuration to display the digital content the at least one processor is further configured to:

edit, prior to capturing the photo of at least a portion of the real world scene, the first region within the boundary using the photo editing tool based on at least one of a voice command and a gesture command;

capture a photo of at least the edited first region of the real world scene; and

store the photo of the edited first region.

28. The apparatus of claim 21, wherein the at least one processor is further configured to:

store at least a first recognized object in response to at least one of a first voice command and a first gesture command;

store at least a second recognized object in response to at least one of a second voice command and a second gesture command; and

compare the at least the first recognized object and the second recognized object.

29. The apparatus of claim 21, wherein the at least one processor is further configured to:

set one or more updated boundaries in the one or more of the plurality of image frames based on continued detection of the eye gaze; and

perform the object recognition procedure on one or more regions surrounded by the one or more updated boundaries to identify the object within the one or more regions.

30. The apparatus of claim 21, wherein the photo editing tool includes at least one of a filter tool, a clip art tool, a color saturation tool, a sepia toe conversion tool, and a black and white conversion tool.

31. A non-transitory computer-readable medium storing computer executable code, comprising code for:

detecting, via the optical device, an eye gaze of a user on a first region in a real world scene;

setting a boundary that surrounds the first region, the boundary excluding at least a second region in the real world scene;

performing an object recognition procedure on the first region within the boundary for one or more of a plurality of image frames of the real world scene to identify an object within the first region;

refraining from performing an object recognition procedure on the at least the second region;

displaying digital content associated with the object within the eye gaze, wherein the digital content is displayed while the eye gaze remains on the first region; and

displaying a photo editing tool via the optical device while the user gazes at the real world scene via the optical device prior to capturing a photo of at least a portion of the real world scene.

32. The non-transitory computer-readable medium of claim 31, further comprising code for:

detecting the eye gaze on a third region in the real world scene; and

setting the boundary to surround the third region while excluding at least a fourth region in the real world scene.

33. The non-transitory computer-readable medium of claim 31, further comprising code for:

determining a first count of consecutive times the object has been recognized by the object recognition procedure within the boundary or a second count of a number of times the object has been recognized within the boundary;

highlighting the object when the first count or the second count exceeds a threshold; and

displaying an icon proximate to the object when the first count or the second count exceeds the threshold.

34. The non-transitory computer-readable medium of claim 33, wherein the threshold is dynamically set based on at least one of a type of object, a history, and a user preference.

35. The non-transitory computer-readable medium of claim 31, further comprising code for performing at least one of an operation to zoom into a center of the first region within the boundary, an operation to enlarge the boundary, or an operation to enter a power saving state if no object has been recognized in the first region within the boundary.

36. The non-transitory computer-readable medium of claim 31, further comprising code for:

receiving at least one of a voice command and a gesture command; and

adjusting a magnification of the first region within the boundary in response to the at least one of the voice command and the gesture command.

37. The non-transitory computer-readable medium of claim 31, wherein the code for displaying digital content further includes code for:

editing, prior to capturing the photo of at least a portion of the real world scene, the first region within the boundary using the photo editing tool based on at least one of a voice command and a gesture command;

capturing a photo of at least the edited first region of the real world scene; and

storing the photo of the edited first region.

38. The non-transitory computer-readable medium of claim 31, further comprising code for:

storing at least a first recognized object in response to at least one of a first voice command and a first gesture command;

storing at least a second recognized object in response to at least one of a second voice command and a second gesture command; and

comparing the at least the first recognized object and the second recognized object.

39. The non-transitory computer-readable medium of claim 31, further comprising code for:

setting one or more updated boundaries in the one or more of the plurality of image frames based on continued detection of the eye gaze; and

performing the object recognition procedure on one or more regions surrounded by the one or more updated boundaries to identify the object within the one or more regions.

40. The non-transitory computer-readable medium of claim 31, wherein the photo editing tool includes at least one of a filter tool, a clip art tool, a color saturation tool, a sepia toe conversion tool, and a black and white conversion tool.