APPARATUS AND METHOD FOR CONTROLLING AN AUGMENTED REALITY DEVICE

• US 20140198129A1
• Filed: 01/13/2013
• Published: 07/17/2014
• Est. Priority Date: 01/13/2013
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: KIM, Frederick D., LIU, Kexi, RESCHLER, Martin H.
• Granted Patent: US 10,359,841 B2
• Time in Patent Office: Days
• US Class Current: 345/633
• CPC Class Codes: G02B 2027/0127 comprising devices increasing the depth of field
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

An apparatus, a method, and a computer program product are provided. The apparatus detects an eye gaze on a first region in a real world scene, sets a boundary that surrounds the first region, the boundary excluding at least a second region in the real world scene, performs an object recognition procedure on the first region within the boundary, and refrains from performing an object recognition procedure on the at least the second region.

Claims

1. A method of controlling an optical device configured to display content, the method comprising:

detecting an eye gaze on a first region in a real world scene;

setting a boundary that surrounds the first region, the boundary excluding at least a second region in the real world scene;

performing an object recognition procedure on the first region within the boundary; and

refraining from performing an object recognition procedure on the at least the second region.

2. The method of claim 1, further comprising:

detecting the eye gaze on a third region in the real world scene; and

setting the boundary to surround the third region while excluding at least a fourth region in the real world scene.

3. The method of claim 1, wherein performing the object recognition procedure comprises performing the object recognition procedure on the first region within the boundary for one or more of a plurality of image frames of the real world scene, the method further comprising:

storing, in a memory, a first count of consecutive times an object has been recognized within the boundary or a second count of a number of times an object has been recognized within the boundary.

4. The method of claim 3, further comprising:

determining whether the first count or the second count exceeds a threshold;

highlighting the object when the first count or the second count exceeds the threshold; and

displaying an icon proximate to the object when the first count or the second count exceeds the threshold.

5. The method of claim 1, wherein if no object has been recognized in the first region within the boundary, the method further comprises performing at least one of an operation to zoom into a center of the first region within the boundary, an operation to enlarge the boundary, or an operation to enter a power saving state.

6. The method of claim 1, wherein performing the object recognition procedure comprises performing the object recognition procedure on the first region within the boundary for one or more of a plurality of image frames of the real world scene, the method further comprising:

storing, in a memory, a count of a number of times an object has been recognized within the boundary.

7. The method of claim 6, further comprising:

dynamically setting a threshold based on at least one of a type of object, a history, and a user preference;

determining whether the count exceeds the threshold;

highlighting the object when the count exceeds the threshold; and

displaying an icon proximate to the object when the count exceeds the threshold.

8. The method of claim 1, further comprising:

receiving at least one of a voice command and a gesture command; and

adjusting a magnification of the first region within the boundary in response to the at least one of the voice command and the gesture command.

9. The method of claim 1, further comprising:

displaying at least one photo editing tool;

editing the first region within the boundary using the at least one photo editing tool based on at least one of a voice command and a gesture command; and

storing an image of the edited first region.

10. The method of claim 1, further comprising:

storing at least a first recognized object in response to at least one of a first voice command and a first gesture command;

storing at least a second recognized object in response to at least one of a second voice command and a second gesture command; and

comparing the at least the first recognized object and the second recognized object.

11. An apparatus for controlling an optical device configured to display content, the apparatus comprising:

means for detecting an eye gaze on a first region in a real world scene;

means for setting a boundary that surrounds the first region, the boundary excluding at least a second region in the real world scene;

means for performing an object recognition procedure on the first region within the boundary; and

means for refraining from performing an object recognition procedure on the at least the second region.

12. The apparatus of claim 11, further comprising:

means for detecting the eye gaze on a third region in the real world scene; and

means for setting the boundary to surround the third region while excluding at least a fourth region in the real world scene.

13. The apparatus of claim 11, wherein the means for performing the object recognition procedure is configured to perform the object recognition procedure on the first region within the boundary for one or more of a plurality of image frames of the real world scene, the apparatus further comprising:

means for storing a first count of consecutive times an object has been recognized within the boundary or a second count of a number of times an object has been recognized within the boundary.

14. The apparatus of claim 11, further comprising:

means for determining whether the first count or the second count exceeds a threshold;

means for highlighting the object when the first count or the second count exceeds the threshold; and

means for displaying an icon proximate to the object when the first count or the second count exceeds the threshold.

15. The apparatus of claim 11, further comprising means for performing at least one operation, the at least one operation comprising at least one of an operation to zoom into a center of the first region within the boundary, an operation to enlarge the boundary, or an operation to enter a power saving state if no object has been recognized in the first region within the boundary.

16. The apparatus of claim 11, wherein the means for performing the object recognition procedure is configured to perform the object recognition procedure on the first region within the boundary for one or more of a plurality of image frames of the real world scene, the apparatus further comprising:

means for storing a count of a number of times an object has been recognized within the boundary.

17. The apparatus of claim 16, further comprising:

means for dynamically setting a threshold based on at least one of a type of object, a history, and a user preference;

means for determining whether the count exceeds the threshold;

means for highlighting the object when the count exceeds the threshold; and

means for displaying an icon proximate to the object when the count exceeds the threshold.

18. The apparatus of claim 11, further comprising:

means for receiving at least one of a voice command and a gesture command; and

means for adjusting a magnification of the first region within the boundary in response to the at least one of the voice command and the gesture command.

19. The apparatus of claim 11, further comprising:

means for displaying at least one photo editing tool;

means for editing the first region within the boundary using the at least one photo editing tool based on at least one of a voice command and a gesture command; and

means for storing an image of the edited first region.

20. The apparatus of claim 11, further comprising:

means for storing at least a first recognized object in response to at least one of a first voice command and a first gesture command;

means for storing at least a second recognized object in response to at least one of a second voice command and a second gesture command; and

means for comparing the at least the first recognized object and the second recognized object.

21. An apparatus for controlling an optical device configured to display content, the apparatus comprising:

a processing system configured to;

detect an eye gaze on a first region in a real world scene;

set a boundary that surrounds the first region, the boundary excluding at least a second region in the real world scene;

perform an object recognition procedure on the first region within the boundary; and

refrain from performing an object recognition procedure on the at least the second region.

22. The apparatus of claim 21, wherein the processing system is further configured to:

detect the eye gaze on a third region in the real world scene; and

set the boundary to surround the third region while excluding at least a fourth region in the real world scene.

23. The apparatus of claim 21, wherein the object recognition procedure is performed by performing the object recognition procedure on the first region within the boundary for one or more of a plurality of image frames of the real world scene, the processing system further configured to:

store, in a memory, a first count of consecutive times an object has been recognized within the boundary or a second count of a number of times an object has been recognized within the boundary.

24. The apparatus of claim 23, wherein the processing system is further configured to:

determine whether the first count or the second count exceeds a threshold;

highlight the object when the first count or the second count exceeds the threshold; and

display an icon proximate to the object when the first count or the second count exceeds the threshold.

25. The apparatus of claim 21, wherein the processing system is further configured to perform at least one of an operation to zoom into a center of the first region within the boundary, an operation to enlarge the boundary, or an operation to enter a power saving state if no object has been recognized in the first region within the boundary.

26. The apparatus of claim 21, wherein the object recognition procedure is performed by performing the object recognition procedure on the first region within the boundary for one or more of a plurality of image frames of the real world scene, the processing system further configured to:

store, in a memory, a count of a number of times an object has been recognized within the boundary.

27. The apparatus of claim 26, wherein the processing system is further configured to:

dynamically set a threshold based on at least one of a type of object, a history, and a user preference;

determine whether the count exceeds the threshold;

highlight the object when the count exceeds the threshold; and

display an icon proximate to the object when the count exceeds the threshold.

28. The apparatus of claim 21, wherein the processing system is further configured to:

receive at least one of a voice command and a gesture command; and

adjust a magnification of the first region within the boundary in response to the at least one of the voice command and the gesture command.

29. The apparatus of claim 21, wherein the processing system is further configured to:

display at least one photo editing tool;

edit the first region within the boundary using the at least one photo editing tool based on at least one of a voice command and a gesture command; and

store an image of the edited first region.

30. The apparatus of claim 21, wherein the processing system is further configured to:

store at least a first recognized object in response to at least one of a first voice command and a first gesture command;

store at least a second recognized object in response to at least one of a second voice command and a second gesture command; and

compare the at least the first recognized object and the second recognized object.

31. A computer program product, comprising:

a computer-readable medium comprising code for;

detecting an eye gaze on a first region in a real world scene;

setting a boundary that surrounds the first region, the boundary excluding at least a second region in the real world scene;

performing an object recognition procedure on the first region within the boundary; and

refraining from performing an object recognition procedure on the at least the second region.

32. The computer program product of claim 31, the computer-readable medium further comprising code for:

detecting the eye gaze on a third region in the real world scene; and

setting the boundary to surround the third region while excluding at least a fourth region in the real world scene.

33. The computer program product of claim 31, wherein performing the object recognition procedure comprises performing the object recognition procedure on the first region within the boundary for one or more of a plurality of image frames of the real world scene, the computer-readable medium further comprising code for:

storing, in a memory, a first count of consecutive times an object has been recognized within the boundary or a second count of a number of times an object has been recognized within the boundary.

34. The computer program product of claim 33, the computer-readable medium further comprising code for:

determining whether the first count or the second count exceeds a threshold;

highlighting the object when the first count or the second count exceeds the threshold; and

displaying an icon proximate to the object when the first count or the second count exceeds the threshold.

35. The computer program product of claim 31, the computer-readable medium further comprising code for performing at least one of an operation to zoom into a center of the first region within the boundary, an operation to enlarge the boundary, or an operation to enter a power saving state if no object has been recognized in the first region within the boundary.

36. The computer program product of claim 31, wherein performing the object recognition procedure comprises performing the object recognition procedure on the first region within the boundary for one or more of a plurality of image frames of the real world scene, the computer-readable medium further comprising code for:

storing, in a memory, a count of a number of times an object has been recognized within the boundary.

37. The computer program product of claim 36, the computer-readable medium further comprising code for:

dynamically setting a threshold based on at least one of a type of object, a history, and a user preference;

determining whether the count exceeds the threshold;

highlighting the object when the count exceeds the threshold; and

displaying an icon proximate to the object when the count exceeds the threshold.

38. The computer program product of claim 31, the computer-readable medium further comprising code for:

receiving at least one of a voice command and a gesture command; and

adjusting a magnification of the first region within the boundary in response to the at least one of the voice command and the gesture command.

39. The computer program product of claim 31, the computer-readable medium further comprising code for:

displaying at least one photo editing tool;

editing the first region within the boundary using the at least one photo editing tool based on at least one of a voice command and a gesture command; and

storing an image of the edited first region.

40. The computer program product of claim 31, the computer-readable medium further comprising code for:

storing at least a first recognized object in response to at least one of a first voice command and a first gesture command;

storing at least a second recognized object in response to at least one of a second voice command and a second gesture command; and

comparing the at least the first recognized object and the second recognized object.