MOBILE TERMINAL AND METHOD OF CONTROLLING OPERATION THEREOF

• US 20120302289A1
• Filed: 05/09/2012
• Published: 11/29/2012
• Est. Priority Date: 05/27/2011
• Status: Active Grant

• Current Assignee: LG Electronics, Inc. (LG Corporation)
• Original Assignee: LG Electronics, Inc. (LG Corporation)
• Inventors: KANG, Heejoon
• Granted Patent: US 9,256,283 B2
• Time in Patent Office: Days
• US Class Current: 455/557
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: H04M 1/72412 using two-way short-range wireless interfaces

Abstract

A mobile terminal and a method of controlling an operation thereof are disclosed. In the method of controlling the operation of the mobile terminal, the mobile terminal is communicatively with a head mounted display for displaying an augmented reality (AR) screen obtained by combining a virtual image with a real environment. If image data acquired through a camera is analyzed and a predetermined gesture input is detected, the head mounted display is controlled such that a display change corresponding to the detected gesture input is displayed on the AR screen. In a state of displaying the AR screen through the head mounted display, various operations related to the mobile terminal can be controlled according to the predetermined gesture input.

Claims

1. A method of controlling an operation of a mobile terminal, the method comprising:

communicatively connecting the mobile terminal with a head mounted display for displaying an augmented reality (AR) screen obtained by combining a virtual image with a real environment;

analyzing image data acquired through a camera and detecting a predetermined gesture input; and

controlling the head mounted display such that a display change corresponding to the detected gesture input is displayed on the AR screen.

2. The method according to claim 1, wherein the camera is disposed on the head mounted display.

3. The method according to claim 1, wherein the display change is controlled to be displayed at an eye position of the user, which is detected by performing eye tracking.

4. The method according to claim 1, wherein the controlling includes controlling the head mounted display such that a closed curve is displayed on the AR screen if a gesture input of drawing a closed curve in a real space is detected.

5. The method according to claim 4, further comprising displaying items belonging to a specific group in the closed curve according to a predetermined user input.

6. The method according to claim 5, wherein the predetermined user input includes an input of touching a display of the mobile terminal and performing a dragging operation to the closed curve.

7. The method according to claim 4, further comprising displaying a menu for setting a tag corresponding to the closed curve on the AR screen in response to a gesture input of tapping a boundary of the closed curve as a predetermined number of times.

8. The method according to claim 7, wherein the menu is displayed on a display of the mobile terminal.

9. The method according to claim 4, further comprising enlarging and displaying items displayed in the closed curve at the outside of the closed curve in response to a gesture input of shaking the closed curve.

10. The method according to claim 4, further comprising reducing and displaying the closed curve on the AR screen in response to a gesture input of pushing a boundary of the closed curve inward.

11. The method according to claim 4, further comprising enlarging and displaying the closed curve on the AR screen in response to a gesture input of pulling a boundary of the closed curve outward.

12. The method according to claim 4, further comprising dividing the closed curve into first and second closed curves and displaying the divided closed curves on the AR screen in response to a gesture input of twisting and then pulling the closed curve outward.

13. The method according to claim 12, further comprising combining the first and second closed curves into one closed curve and displaying the combined closed curve on the AR screen in response to a gesture input of moving the first closed curve onto the second closed curve.

14. A mobile terminal comprising:

a wireless communication unit configured to communicatively connect the mobile terminal with a head mounted display for displaying an augmented reality (AR) screen obtained by combining a virtual image with a real environment; and

a controller configured to analyze image data acquired through a camera to detect a predetermined gesture input and control the head mounted display such that a display change corresponding to the detected gesture input is displayed on the AR screen.

15. The mobile terminal according to claim 14, wherein the controller enables to display the display change at an eye position of a user, which is detected by performing eye tracking.

16. The mobile terminal according to claim 15, wherein the head mounted display includes a first camera for acquiring the image data and a second camera for performing the eye tracking of the user.

17. The mobile terminal according to claim 14, wherein the controller enables to display a closed curve on the AR screen if a gesture input of drawing a closed curve in a real space is detected.

18. The mobile terminal according to claim 17, wherein the controller enables to display items belonging to a specific group in the closed curve if a gesture input of touching a display and then performing a dragging operation to the closed curve is detected.

19. The mobile terminal according to claim 17, wherein the controller enables to display a text input window for attaching a tag to the closed curve on a display and sets text input through the text input window as a tag of the closed curve, if a gesture input of tapping a boundary of the closed curve is detected.

20. The mobile terminal according to claim 17, wherein the controller divides the closed curve into first and second closed curves and enables to display the divided closed curves on the AR screen, if a gesture input of twisting and then pushing the closed curve outward is detected.

21. A system comprising:

a head mounted display configured to display an augmented reality (AR) screen obtained by combining a virtual image with a real environment; and

a mobile terminal communicatively connected to the head mounted display and configured to analyze image data acquired through a camera to detect a predetermined gesture input and control the head mounted display such that a display change corresponding to the detected gesture input is displayed on the AR screen.

22. The system according to claim 21, wherein the head mounted display includes a first camera configured to acquire the image data and a second camera configured to perform eye tracking of a user.

23. The system according to claim 21, wherein the mobile terminal controls the head mounted display such that the display change is displayed at an eye position of a user, which is detected by performing eye tracking through the second camera.

24. The system according to claim 21, wherein the mobile terminal controls the head mounted display such that a closed curve is displayed on the AR screen, if a gesture input of drawing a closed curve in a real space is detected.