Apparatus and method of providing user interface on head mounted display and head mounted display thereof

• US 9,378,592 B2
• Filed: 09/16/2013
• Issued: 06/28/2016
• Est. Priority Date: 09/14/2012
• Status: Active Grant

• Current Assignee: LG Electronics, Inc. (LG Corporation)
• Original Assignee: LG Electronics, Inc. (LG Corporation)
• Inventors: Klee, Doyoung, Kim, Jihwan, Chun, Sinae, Cho, Eunhyung
• Primary Examiner(s): Tung, Kee M
• Assistant Examiner(s): Wilson, Nicholas R
• Application Number:
  US14/028,153
• Publication Number:
  US 20140078176A1
• Time in Patent Office: 1,016 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/147 using display panels
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

An apparatus and method of providing a user interface (UI) on head mounted display and the head mounted display (HMD) thereof are discussed. The apparatus includes a sensor unit configured to detect whether an object exits in the proximity of the HMD and if the object is detected, sense a distance between the object and the HMD, and a processor configured to apply a physical User Interface (UI) mode if the detected object is within a predetermined distance from the HMD. The physical UI mode provides a display for a virtual object. The processor is configured to adjust a display distance of the virtual object, when the virtual object is a 3-dimensional (3D) object and the 3D object includes a touch recognition surface, in order for the touch recognition surface to coincide with a surface of the object, and display the virtual object based on the adjusted display distance.

Claims

1. A User Interface (UI) apparatus for a Head Mounted Display (HMD) comprising:

a sensor unit configured to detect whether an object exists in the proximity of the HMD and if the object is detected, sense a distance between the object and the HMD; and

a processor configured to apply a physical User Interface (UI) mode if the detected object is within a first predetermined manipulable distance from the HMD and apply a non-physical User Interface (UI) mode if the object is not detected or is not within the first predetermined manipulate distance from the HMD,

wherein the processor receives a user command inputted to the UI of the HMD in the physical UI mode by physically contacting the object and in the non-physical UI mode by not physically contacting the object,

wherein the physical UI mode provides a display for a virtual object, and

wherein the processor is further configured to;

adjust a display distance of the virtual object, when the virtual object is a 3-dimensional (3D) object and the virtual object includes a touch recognition surface, in order for the touch recognition surface to coincide with a surface of the object, and

render the virtual object for display based on the adjusted display distance.

2. The apparatus of claim 1, wherein the processor adjusts the display distance of the virtual object in order for at least one touch recognition surface selected by a predetermined method based on selection information to coincide with the surface of the object when the virtual object includes a plurality of touch recognition surfaces, and renders the virtual object for display based on the adjusted display distance.

3. The apparatus of claim 2, wherein the selection information includes at least one of information on display area, information on use frequency, and information on distance between a front side and a rear side of the virtual object.

4. The apparatus of claim 1, wherein the sensor unit is further configured to detect a user gesture on the virtual object, and wherein the processor is further configured to:

render the virtual object for display according to a predetermined display distance when the virtual object includes a plurality of touch recognition surfaces,

when the user gesture is detected, acquire a distance between an input object detected by the gesture and the plurality of touch recognition surfaces,

determine whether the acquired distance is less than or equal to a second predetermined distance,

adjust a display distance of the virtual object in order for the touch recognition surface determined to be less or equal to the second predetermined distance to coincide with the surface of the object, and

render the virtual object for display based on the adjusted display distance.

5. The apparatus of claim 1, wherein the processor renders the virtual object for display according to a third predetermined distance when the virtual object is a 3D object and the virtual object includes a touch recognition surface.

6. The apparatus of claim 1, wherein the non-physical UI mode operates in a status in which the non-physical UI does not have a physical contact with the object.

7. The apparatus of claim 6, wherein the non-physical UI mode comprises at least one of a voice recognition User Interface (UI) or a gesture User Interface (UI).

8. The apparatus of claim 1, wherein the processor selects the physical UI mode if the detected object is within the first predetermined manipulable distance and the detected object is within a view angle.

9. The apparatus of claim 8, wherein the processor changes to the non-physical UI mode if the detected object is within the view angle but is not user-interfaceable.

10. The apparatus of claim 1, wherein the processor selects the non-physical UI mode if the detected object is within the first predetermined manipulable distance from the HMD but not within the view angle.

11. The apparatus of claim 10, wherein the processor changes to the physical UI mode if the detected object is not within the view angle but returns within the view angle within a predetermined period of time.

12. The apparatus of claim 10, wherein the processor stays in the non-physical UI mode if the detected object is not within the view angle and does not return within the view angle within the predetermined period of time.

13. The apparatus of claim 1, wherein the processor selects the non-physical UI mode if the detected object is within the first predetermined manipulable distance from the HMD but not within the view angle, and the detected object does not return within the view angle within the predetermined period of time.

14. The apparatus of claim 1, wherein the processor applies a display function of a new digital device as the physical UI mode if the detected object is within the first predetermined manipulable distance from the HMD but not within the view angle, and the new digital device is detected within the first predetermined manipulable distance within a certain period of time.

15. The apparatus of claim 1, wherein the processor applies the display of the new digital device as the physical UI mode if the object is not detected or is not within the first predetermined manipulable distance from the HMD, and the new digital device is detected within the first predetermined manipulable distance within the certain period of time.

16. The apparatus of claim 1, wherein the processor applies the display of the new digital device as the physical UI mode if the object is not detected or is not within the first predetermined manipulable distance from the HMD, and the new digital device is detected within the first predetermined manipulable distance within the certain period of time.

17. A method providing a User Interface (UI) for a Head Mounted Display (HMD), the method comprising:

detecting whether an object exists in the proximity of the HMD;

sensing a distance between the detected object and the HMD;

applying a physical User Interface (UI) mode if the object is within a first predetermined manipulable distance from the HMD, wherein the physical UI mode provides a display for a virtual object and applying a non-physical User Interface (UI) mode if the object is not detected or is not within the first predetermined manipulate distance from the HMD,

wherein a user command is inputted to the UI of the HMD in the physical UI mode by physically contacting the object and in the non-physical UI mode by not physically contacting the object;

adjusting a display distance of the virtual object, when the virtual object is a 3-dimensional (3D) object and the virtual object includes a touch recognition surface, in order for the touch recognition surface to coincide with a surface of the object; and

rendering the virtual object for display based on the adjusted display distance.

18. The method of claim 17, wherein the virtual object includes a plurality of touch recognition surfaces, and wherein the step of adjusting the display distance includes adjusting a display distance of the virtual object in order for at least one of touch recognition surface selected using a predetermined method based on selection information to coincide with a surface of the object.