User interface for augmented reality enabled devices

• US 9,317,972 B2
• Filed: 12/18/2012
• Issued: 04/19/2016
• Est. Priority Date: 12/18/2012
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Forutanpour, Babak, Ganesh, Shriram, Baker, Daniel S., Yamakawa, Devender A.
• Primary Examiner(s): Wu, Xiao
• Assistant Examiner(s): Akhavannik, Mohammad H
• Application Number:
  US13/718,923
• Publication Number:
  US 20140168262A1
• Time in Patent Office: 1,218 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

Method and apparatus for displaying augmented reality contents are disclosed. The method may include controlling a camera to scan an environment in view of a user, identifying a set of surfaces in the environment for displaying user interface windows according to characteristics of the environment, prioritizing a set of augmented reality contents for display with respect to the set of surfaces in the environment, and displaying the set of augmented reality contents on the set of surfaces in a display. Characteristics of the environment comprise at least aspect ratio of the set of surfaces with respect to the set of augmented reality contents to be displayed, and/or background color of the set of surfaces with respect to the set of augmented reality contents to be displayed.

Claims

1. A method of displaying augmented reality contents, comprising:

receiving, by a camera, a scan of an environment in view of a user;

identifying a set of surfaces in the environment for displaying user interface windows;

assigning a surface priority level to each surface in the set of surfaces, wherein the surface priority level for each surface is selected from a plurality of surface priority levels;

assigning a surface trackability level to each surface in the set of surfaces, wherein the surface trackability level for each surface is selected from a plurality of surface trackability levels;

assigning a content priority level to each content item in a set of augmented reality contents for display with respect to the set of surfaces in the environment, wherein the content priority level for each content item is selected from a plurality of content priority levels; and

displaying each content item in the set of augmented reality contents for display on a surface in the set of surfaces in a display based on the surface priority level, the surface trackability level, and the content priority level.

2. The method of claim 1, wherein the environment comprises real world objects, and wherein the set of surfaces includes surfaces in top, bottom, left, right, and front view of the user;

and wherein the display is a head mounted display.

3. The method of claim 1, wherein assigning the surface priority level to each surface in the set of surfaces comprises:

assigning the surface priority level based on;

areas of the set of surfaces, orientations of the set of surfaces, and locations of the set of surfaces relative to field of view of the user, wherein the plurality of surface priority levels include high, medium and low surface priority levels.

4. The method of claim 1, wherein assigning the content priority level to each content item in a set of augmented reality contents for display with respect to the set of surfaces in the environment comprises:

assigning the content priority level based on at least one of;

aspect ratio of the set of surfaces with respect to the set of augmented reality contents to be displayed, and background color of the set of surfaces with respect to the set of augmented reality contents to be displayed.

5. The method of claim 1, wherein assigning the content priority level to each content item in a set of augmented reality contents for display with respect to the set of surfaces in the environment comprises:

assigning the content priority level based on a history of prior use patterns of the user in the environment.

6. The method of claim 5, wherein assigning the content priority level based on a history of prior use patterns of the user in the environment comprises at least one of:

selecting a first set of augmented reality contents for display in response to a determination of the user being at work;

selecting a second set of augmented reality contents for display in response to a determination of the user being at home;

selecting a third set of augmented reality contents for display in response to a determination of the user being in a business meeting; and

selecting a fourth set of augmented reality contents for display in response to a determination of the user being in a social event.

7. The method of claim 6, wherein

the first set of augmented reality contents includes application programs for communication and documentation the user is accustomed to use at work;

and

the second set of augmented reality contents includes application programs for communication and entertainment the user is accustomed to use at home.

8. The method of claim 7, wherein

the first set of augmented reality contents for communication and documentation includes email, web browser, and office productivity applications;

and

the second set of augmented reality contents for communication and entertainment includes social networking websites, movie and video game applications.

9. The method of claim 1, wherein assigning the content priority level to each content item in a set of augmented reality contents for display with respect to the set of surfaces in the environment comprises:

assigning the content priority level based on at least one of;

scaling factors of the set of augmented reality contents to be displayed, and logical relationships of the set of augmented reality contents to be displayed.

10. The method of claim 9, wherein the logical relationships of the set of augmented reality contents to be displayed comprises:

a document and a corresponding video presentation of the document.

11. The method of claim 9, wherein the logical relationships of the set of augmented reality contents to be displayed comprises:

a document and corresponding presentation slides of the document.

12. The method of claim 1, wherein assigning the content priority level to each content item in a set of augmented reality contents for display with respect to the set of surfaces in the environment comprises:

assigning the content priority level based on predefined user preferences.

13. The method of claim 12, wherein the predefined user preferences comprise at least one of:

importance of an augmented reality content to the user;

amount of the augmented reality content to be displayed; and

viewing orientation of the augmented reality content with respect to the set of surfaces.

14. The method of claim 1, further comprising:

identifying a set of markers in the environment;

associating the set of markers with the set of surfaces; and

tracking the set of surfaces using the set of markers.

15. A computer program product comprising a non-transitory medium storing computer programs for execution by one or more computer systems, the computer program product comprising:

code for receiving, by a camera, a scan of an environment in view of a user;

code for identifying a set of surfaces in the environment for displaying user interface windows;

code for assigning a surface priority level to each surface in the set of surfaces, wherein the surface priority level for each surface is selected from a plurality of surface priority levels;

code for assigning a surface trackability level to each surface in the set of surfaces, wherein the surface trackability level for each surface is selected from a plurality of surface trackability levels;

code for assigning a content priority level to each content item in a set of augmented reality contents for display with respect to the set of surfaces in the environment, wherein the content priority level for each content item is selected from a plurality of content priority levels; and

code for displaying each content item in the set of augmented reality contents for display on a surface in the set of surfaces in a display based on the surface priority level, the surface trackability level, and the content priority level.

16. The computer program product of claim 15, wherein the environment comprises real world objects, and wherein the set of surfaces includes surfaces in top, bottom, left, right, and front view of the user;

and wherein the display is a head mounted display.

17. The computer program product of claim 15, wherein code for assigning the surface priority level to each surface in the set of surfaces comprises:

code for assigning the surface priority level based on;

areas of the set of surfaces, orientations of the set of surfaces, and locations of the set of surfaces relative to field of view of the user, wherein the plurality of surface priority levels include high, medium and low surface priority levels.

18. The computer program product of claim 15, wherein code for assigning the content priority level to each content item in a set of augmented reality contents for display with respect to the set of surfaces in the environment comprises:

code for assigning the content priority level based on at least one of;

aspect ratio of the set of surfaces with respect to the set of augmented reality contents to be displayed, and background color of the set of surfaces with respect to the set of augmented reality contents to be displayed.

19. The computer program product of claim 15, wherein code for assigning the content priority level to each content item in a set of augmented reality contents for display with respect to the set of surfaces in the environment comprises:

code for assigning the content priority level based on a history of prior use patterns of the user in the environment.

20. The computer program product of claim 19, wherein code for assigning the content priority level based on a history of prior use patterns of the user in the environment comprises at least one of:

code for selecting a first set of augmented reality contents for display in response to a determination of the user being at work;

code for selecting a second set of augmented reality contents for display in response to a determination of the user being at home;

code for selecting a third set of augmented reality contents for display in response to a determination of the user being in a business meeting; and

code for selecting a fourth set of augmented reality contents for display in response to a determination of the user being in a social event.

21. The computer program product of claim 20, wherein

the first set of augmented reality contents includes application programs for communication and documentation the user is accustomed to use at work;

and

the second set of augmented reality contents includes application programs for communication and entertainment the user is accustomed to use at home.

22. The computer program product of claim 21, wherein

the first set of augmented reality contents for communication and documentation includes email, web browser, and office productivity applications;

and

the second set of augmented reality contents for communication and entertainment includes social networking websites, movie and video game applications.

23. The computer program product of claim 15, wherein code for assigning the content priority level to each content item in a set of augmented reality contents for display with respect to the set of surfaces in the environment comprises:

code for assigning the content priority level based on at least one of;

scaling factors of the set of augmented reality contents to be displayed, and logical relationships of the set of augmented reality contents to be displayed.

24. The computer program product of claim 15, wherein code for assigning the content priority level to each content item in a set of augmented reality contents for display with respect to the set of surfaces in the environment comprises:

code for assigning the content priority level based on code for determining a set of augmented reality contents for display according to predefined user preferences.

25. The computer program product of claim 24, wherein the predefined user preferences comprise at least one of:

importance of an augmented reality content to the user;

amount of the augmented reality content to be displayed; and

viewing orientation of the augmented reality content with respect to the set of surfaces.

26. The computer program product of claim 15, further comprising:

code for identifying a set of markers in the environment;

code for associating the set of markers with the set of surfaces; and

code for tracking the set of surfaces using the set of markers.

27. An apparatus, comprising:

at least one processor, wherein the at least one processor comprises

logic configured to receive, by a camera, a scan of an environment in view of a user;

logic configured to identify a set of surfaces in the environment for displaying user interface windows;

logic configured to assign a surface priority level to each surface in the set of surfaces, wherein the surface priority level for each surface is selected from a plurality of surface priority levels;

logic configured to assign a surface trackability level to each surface in the set of surfaces, wherein the surface trackability level for each surface is selected from a plurality of surface trackability levels;

logic configured to assign a content priority level to each content item in a set of augmented reality contents for display with respect to the set of surfaces in the environment, wherein the content priority level for each content item is selected from a plurality of content priority levels; and

logic configured to display each content item in the set of augmented reality contents for display on a surface in the set of surfaces in a display based on the surface priority level, the surface trackability level, and the content priority level.

28. The apparatus of claim 27, wherein the environment comprises real world objects, and wherein the set of surfaces includes surfaces in top, bottom, left, right, and front view of the user;

and wherein the display is a head mounted display.

29. The apparatus of claim 27, wherein logic configured to assign the surface priority level to each surface in the set of surfaces comprises:

logic configured to assign the surface priority level based on;

areas of the set of surfaces, orientations of the set of surfaces, and locations of the set of surfaces relative to field of view of the user, wherein the plurality of surface priority levels include high, medium and low surface priority levels.

30. The apparatus of claim 27, wherein logic configured to assign the content priority level to each content item in a set of augmented reality contents for display with respect to the set of surfaces in the environment comprise:

logic configured to assign the content priority level based on at least one of;

aspect ratio of the set of surfaces with respect to the set of augmented reality contents to be displayed, and background color of the set of surfaces with respect to the set of augmented reality contents to be displayed.

31. The apparatus of claim 27, wherein logic configured to assign the content priority level to each content item in a set of augmented reality contents for display with respect to the set of surfaces in the environment comprises:

logic configured to assign the content priority level based on a history of prior use patterns of the user in the environment.

32. The apparatus of claim 31, wherein logic configured to assign the content priority level based on a history of prior use patterns of the user in the environment comprises at least one of:

logic configured to select a first set of augmented reality contents for display in response to a determination of the user being at work;

logic configured to select a second set of augmented reality contents for display in response to a determination of the user being at home;

logic configured to select a third set of augmented reality contents for display in response to a determination of the user being in a business meeting; and

logic configured to select a fourth set of augmented reality contents for display in response to a determination of the user being in a social event.

33. The apparatus of claim 32, wherein

the first set of augmented reality contents includes application programs for communication and documentation the user is accustomed to use at work;

and

the second set of augmented reality contents includes application programs for communication and entertainment the user is accustomed to use at home.

34. The apparatus of claim 33, wherein

the first set of augmented reality contents for communication and documentation includes email, web browser, and office productivity applications;

and

the second set of augmented reality contents for communication and entertainment includes social networking websites, movie and video game applications.

35. The apparatus of claim 27, wherein logic configured to assign the content priority level to each content item in a set of augmented reality contents for display with respect to the set of surfaces in the environment comprises:

logic configured to assign the content priority level based on at least one of;

scaling factors of the set of augmented reality contents to be displayed, and logical relationships of the set of augmented reality contents to be displayed.

36. The apparatus of claim 27, wherein logic configured to assign the content priority level to each content item in a set of augmented reality contents for display with respect to the set of surfaces in the environment comprises:

logic configured to assign the content priority level based on predefined user preferences.

37. The apparatus of claim 36, wherein the predefined user preferences comprise at least one of:

importance of an augmented reality content to the user;

amount of the augmented reality content to be displayed; and

viewing orientation of the augmented reality content with respect to the set of surfaces.

38. The apparatus of claim 27, further comprising:

logic configured to identify a set of markers in the environment;

logic configured to associate the set of markers with the set of surfaces; and

logic configured to track the set of surfaces using the set of markers.

39. A system for displaying augmented reality contents, comprising:

means for receiving, by a camera, a scan of an environment in view of a user;

means for identifying a set of surfaces in the environment for displaying user interface windows;

means for prioritizing a set of augmented reality contents for display with respect to the set of surfaces in the environment by assigning a surface priority level to each surface in the set of surfaces, wherein the surface priority level for each surface is selected from a plurality of surface priority levels, assigning a surface trackability level to each surface in the set of surfaces, wherein the surface trackability level for each surface is selected from a plurality of surface trackability levels, and assigning a content priority level to each content item in a set of augmented reality contents for display with respect to the set of surfaces in the environment, wherein the content priority level for each content item is selected from a plurality of content priority levels; and

means for displaying each content item in the set of augmented reality contents for display on a surface in the set of surfaces in a display based on the surface priority level, the surface trackability level, and the content priority level.

40. The system of claim 39, wherein means for prioritizing a set of augmented reality contents for display with respect to the set of surfaces in the environment comprises:

means for assigning the surface priority level based on;

areas of the set of surfaces, orientations of the set of surfaces, and locations of the set of surfaces relative to field of view of the user, wherein the plurality of surface priority levels include high, medium and low surface priority levels.

41. The system of claim 39, wherein means for prioritizing a set of augmented reality contents for display with respect to the set of surfaces in the environment comprises:

means for assigning the content priority level based on at least one of;

aspect ratio of the set of surfaces with respect to the set of augmented reality contents to be displayed, and background color of the set of surfaces with respect to the set of augmented reality contents to be displayed.

42. The system of claim 39, wherein means for prioritizing a set of augmented reality contents for display with respect to the set of surfaces in the environment comprises:

means for prioritizing the set of augmented reality contents for display with respect to the set of surfaces in the environment according to a history of prior use patterns of the user in the environment.

43. The system of claim 39, wherein means for prioritizing a set of augmented reality contents for display with respect to the set of surfaces in the environment comprises:

means for assigning the content priority level based on at least one of;

scaling factors of the set of augmented reality contents to be displayed, and logical relationships of the set of augmented reality contents to be displayed.

44. The system of claim 39, wherein means for prioritizing a set of augmented reality contents for display with respect to the set of surfaces in the environment comprises:

means for prioritizing the set of augmented reality contents for display with respect to the set of surfaces in the environment according to predefined user preferences.