Three-dimensional user interface for head-mountable display

• US 10,416,835 B2
• Filed: 06/22/2015
• Issued: 09/17/2019
• Est. Priority Date: 06/22/2015
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd. (Samsung Group)
• Original Assignee: Samsung Electronics Co. Ltd. (Samsung Group)
• Inventors: Chu, Alexander, Kim, Sophie
• Primary Examiner(s): Repsher, III, John T
• Application Number:
  US14/746,388
• Publication Number:
  US 20160370970A1
• Time in Patent Office: 1,548 Days
• Field of Search: 715836
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object

Abstract

A method, apparatus, and computer-readable medium for a user interface (UI) for a head-mountable display (HMD). The method includes generating three-dimensional content for display by the HMD. The method also includes identifying three-dimensional coordinates for UI elements of the UI that are associated with the three-dimensional content. The three-dimensional coordinates identified within an angular range for the UI that includes a viewable region of a user while wearing the HMD. Additionally, the method includes, in response to a user input, displaying the UI elements at the identified three-dimensional coordinates over the three-dimensional content. The method may further include, after displaying the UI elements, moving the display of the UI elements on the HMD in a direction corresponding to a movement direction of the HMD in response to detecting movement of the HMD corresponding to movement of a head of the user wearing the HMD.

Claims

1. A method for displaying a user interface (UI) for a head-mountable display (HMD), the method comprising:

generating three-dimensional content for display by the HMD;

identifying three-dimensional coordinates for an array of UI elements of the UI that are associated with the three-dimensional content, the three-dimensional coordinates identified within an angular range for the UI that includes a viewable frustum region of the HMD, wherein generating the three-dimensional content for display by the HMD comprises converting two-dimensional UI content displayable by a display of a mobile communication device into the three-dimensional content for display by the HMD in at least the viewable frustum region, wherein the array of UI elements comprises two-dimensional UI elements that are one of;

menu items or interface elements configured to control or manipulate the three-dimensional content; and

in response to receiving an input, displaying the array of UI elements at the identified three-dimensional coordinates as three-dimensional objects in front of the three-dimensional content; and

in response to detecting movement of the HMD corresponding to a head movement, adjusting the array of UI elements and moving the display of the array of UI elements on the HMD in a direction corresponding to a direction of the head movement of the HMD such that the display of the array of UI elements provides a 3D visual effect wherein a location of the array of UI elements, displayed as three-dimensional objects in front of the three-dimensional content, remain constant with respect to the displayed three-dimensional content as the displayed three-dimensional content moves in accordance with the head movement.

2. The method of claim 1, wherein the angular range for the UI is a plane that is wider than the viewable frustum region of the HMD, wherein a portion of the angular range is within the viewable frustum region of the HMD and a portion of the angular range is outside the viewable frustum region of the HMD;

and

wherein one or more UI elements of the array of UI elements are displayed on the HMD outside of the viewable frustum region of the HMD such that the one or more UI elements displayed outside the viewable frustum region of the HMD can be viewed when a yaw movement of the HMD is sensed by a movement sensor in the HMD.

3. The method of claim 1, wherein the displayed array of UI elements are a first array of UI elements corresponding to a first level in a nested set of arrays, the method further comprising displaying a second array of UI elements corresponding to a second level in the nested set of arrays in response to detecting an input corresponding to the second level.

4. The method of claim 3, wherein:

the first array of UI elements is a parent array of UI elements and the second array of UI elements is a child array of UI elements, and

displaying the second array of UI elements comprises displaying the child array of UI elements in a location on the HMD where the parent array of UI elements was displayed and moving the display of the parent array of UI elements to another location on the HMD in response to detecting the input corresponding to the second level.

5. The method of claim 1, wherein:

the HMD is a display for a mobile communication device,

the viewable frustum region of the HMD is smaller than the display for the mobile communication device, and

generating the three-dimensional content for display by the HMD comprises converting two-dimensional content displayable by the display for the mobile communication device into the three-dimensional content for display on the HMD in the viewable frustum region of the HMD.

15. The method of claim 1, wherein one or more UI elements of the array of UI elements are generated outside the viewable frustum region of the HMD, further comprising:

in response to detecting a movement of the HMD, relocating the array of UI elements such that the one or more of the UI elements generated outside the viewable frustum region of the HMD are displayed in the viewable frustum region of the HMD.

6. An apparatus for displaying a user interface (UI), the apparatus comprising:

a head-mountable display (HMD); and

at least one processor configured to;

generate three-dimensional content for display by the HMD;

identify three-dimensional coordinates for an array of UI elements of the UI that are associated with the three-dimensional content, the three-dimensional coordinates identified within an angular range for the UI that includes a viewable frustum region of the HMD,

wherein when the at least one processor generates the three-dimensional content, the at least one processor is further configured to convert two-dimensional UI content displayable by a display of a mobile communication device into the three-dimensional content for display by the HMD in at least the viewable frustum region, and wherein the array of UI elements comprises two-dimensional UI elements that are one of;

menu items or interface elements configured to control or manipulate the three-dimensional content;

in response to receipt of an input, cause the HMD to display the array of UI elements at the identified three-dimensional coordinates as three-dimensional objects in front of the three-dimensional content; and

in response to movement detection of the HMD that corresponds to a head movement, adjust and move the display of the array of UI elements on the HMD in a direction corresponding to a direction of the head movement of the HMD such that the display of the array of UI elements provides a 3D visual effect wherein a location of the array of UI elements, displayed as three-dimensional objects in front of the three-dimensional content, remain constant with respect to the displayed three-dimensional content as the displayed three-dimensional content moves in accordance with the head movement.

7. The apparatus of claim 6, wherein the angular range for the UI is a plane that is wider than the viewable frustum region of the HMD, wherein a portion of the angular range is within the viewable frustum region of the HMD and a portion of the angular range is outside the viewable frustum region of the HMD;

and wherein one or more UI elements of the array of UI elements are displayed on the HMD outside of the viewable frustum region of the HMD such that the one or more UI elements displayed outside the viewable frustum region of the HMD can be viewed when a yaw movement of the HMD is sensed by a movement sensor in the HMD.

8. The apparatus of claim 6, wherein:

the displayed array of UI elements are a first array of UI elements corresponding to a first level in a nested set of arrays, and

the at least one processor is further configured to cause the HMD to display a second array of UI elements corresponding to a second level in the nested set of arrays in response to detecting an input corresponding to the second level.

9. The apparatus of claim 8, wherein:

the first array of UI elements is a parent array of UI elements and the second array of UI elements is a child array of UI elements, and

the at least one processor is configured to cause the HMD to display the child array of UI elements in a location on the HMD where the parent array of UI elements was displayed and move the display of the parent array of UI elements to another location on the HMD in response to detecting the input corresponding to the second level.

10. The apparatus of claim 6, wherein:

the HMD is a display for a mobile communication device,

the viewable frustum region of the HMD is smaller than the display for the mobile communication device, and

the at least one processor is configured to convert two dimensional content displayable by the display for the mobile communication device into the three-dimensional content for display on the HMD in the viewable frustum region of the HMD.

11. A non-transitory computer-readable medium comprising program code for generating a display of a user interface (UI) for a head-mountable display (HMD) that, when executed by at least one processor, causes an electronic device to:

generate three-dimensional content for display by the HMD;

identify three-dimensional coordinates for an array of UI elements of the UI that are associated with the three-dimensional content, the three-dimensional coordinates identified within an angular range for the UI that includes a viewable frustum region of the HMD,

wherein when the electronic device is caused to generate the three-dimensional content for display by the HMD, the non-transitory computer-readable medium further comprises program code that, when executed by the at least one processor, causes the electronic device to convert two-dimensional UI content displayable by a display of a mobile communication device into the three-dimensional content for display by the HMD in at least the viewable frustum region, and wherein the array of UI elements comprises two-dimensional UI elements that are one of;

menu items or interface elements configured to control or manipulate the three-dimensional content; and

in response to receipt of an input, cause the HMD to display the array of UI elements at the identified three-dimensional coordinates as three-dimensional objects in front of the three-dimensional content; and

in response to detecting movement of the HMD corresponding to a head movement, adjust and move the display of the array of UI elements on the HMD in a direction corresponding to a direction of the head movement of the HMD such that the display of the array of UI elements provides a 3D visual effect wherein a location of the array of UI elements, displayed as three-dimensional objects in front of the three-dimensional content, remain constant with respect to the displayed three-dimensional content as the three-dimensional content moves in accordance with the head movement.

12. The non-transitory computer-readable medium of claim 11, wherein the angular range for the UI is a plane that is wider than the viewable frustum region of the HMD, wherein a portion of the angular range is within the viewable frustum region of the HMD and a portion of the angular range is outside the viewable frustum region of the HMD;

and

wherein one or more UI elements of the array of UI elements are displayed on the HMD outside of the viewable frustum region of the HMD such that one or more of the UI elements displayed outside the viewable frustum region of the HMD can be viewed when a yaw movement of the HMD is sensed by a movement sensor in the HMD.

13. The non-transitory computer-readable medium of claim 11, wherein:

the displayed array of UI elements are a first array of UI elements corresponding to a first level in a nested set of arrays, and

the computer-readable medium further comprises program code that that, when executed by the at least one processor, causes the HMD to display a second array of UI elements corresponding to a second level in the nested set of arrays in response to detection of an input corresponding to the second level.

14. The non-transitory computer-readable medium of claim 13, wherein:

the first array of UI elements is a parent array of UI elements and the second array of UI elements is a child array of UI elements, and

the program code that, when executed by the at least one processor, causes the HMD to display the child array of UI elements in a location on the HMD where the parent array of UI elements was displayed and move the display of the parent array of UI elements to another location on the HMD in response to the detection of the input corresponding to the second level.