Mobile terminal and method of controlling operation thereof

• US 9,256,283 B2
• Filed: 05/09/2012
• Issued: 02/09/2016
• Est. Priority Date: 05/27/2011
• Status: Active Grant

• Current Assignee: LG Electronics, Inc. (LG Corporation)
• Original Assignee: LG Electronics, Inc. (LG Corporation)
• Inventors: Kang, Heejoon
• Primary Examiner(s): Ajibade Akonai, Olumide T
• Assistant Examiner(s): COSME, NATASHA W
• Application Number:
  US13/467,899
• Publication Number:
  US 20120302289A1
• Time in Patent Office: 1,371 Days
• Field of Search: 455/41.2, 455/41.3, 455418-420, 455/550.1, 455/556.1, 455/575.1, 455/575.2, 455/575.6, 715/863, 345/6, 345/8, 345/158, 345/419, 345/632, 345/633, 348/47, 348/53
• US Class Current: 1/1
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: H04M 1/72412 using two-way short-range wireless interfaces

Abstract

A mobile terminal and a method of controlling an operation thereof are disclosed. In the method of controlling the operation of the mobile terminal, the mobile terminal is communicatively with a head mounted display for displaying an augmented reality (AR) screen obtained by combining a virtual image with a real environment. If image data acquired through a camera is analyzed and a predetermined gesture input is detected, the head mounted display is controlled such that a display change corresponding to the detected gesture input is displayed on the AR screen. In a state of displaying the AR screen through the head mounted display, various operations related to the mobile terminal can be controlled according to the predetermined gesture input.

Claims

1. A method of controlling an operation of a mobile terminal, the method comprising:

communicatively connecting the mobile terminal with a head mounted display for displaying an augmented reality (AR) screen;

receiving first data from the head mounted display, the received first data includes gesture input data acquired by a first camera at the head mounted display and image data of an image corresponding to an environment acquired by the first camera;

receiving second data from the head mounted display, the received second data includes image data acquired by a second camera at the head mounted display;

analyzing the first and second data;

determining a predetermined gesture based on the analyzed first data and determining an eye position of a user based on the analyzed second data; and

controlling the head mounted display to display an indicator corresponding to a shape of the determined gesture on the AR screen, wherein the indicator is displayed at the determined eye position of the user, and wherein the controlling of the head mounted display includes controlling the head mounted display to display a closed curve region on the AR screen when the predetermined gesture is determined to correspond to a closed curve gesture.

2. The method according to claim 1, wherein the first camera is disposed on a front surface of the head mounted display, and the second camera is disposed on a rear surface of the head mounted display.

3. The method according to claim 1, further comprising displaying items belonging to a specific group in the closed curve region according to a predetermined user input.

4. The method according to claim 3, wherein the predetermined user input includes an input of touching a display of the mobile terminal and performing a dragging operation to the closed curve region.

5. The method according to claim 1, further comprising displaying a menu for setting a tag corresponding to the closed curve region on the AR screen in response to a gesture input of tapping a boundary of the closed curve region a predetermined number of times.

6. The method according to claim 5, wherein the menu is displayed on a display of the mobile terminal.

7. The method according to claim 1, further comprising enlarging and displaying items displayed in the closed curve region at the outside of the closed curve in response to a gesture input of shaking the closed curve region.

8. The method according to claim 1, further comprising reducing and displaying the closed curve region on the AR screen in response to a gesture input of pushing a boundary of the closed curve region inward.

9. The method according to claim 1, further comprising enlarging and displaying the closed curve region on the AR screen in response to a gesture input of pulling a boundary of the closed curve region outward.

10. The method according to claim 1, further comprising dividing the closed curve region into first and second closed curve regions and displaying the divided closed curve regions on the AR screen in response to a gesture input of twisting and then pulling the closed curve region outward.

11. The method according to claim 10, further comprising combining the first and second closed curve regions into one closed curve region and displaying the combined closed curve region on the AR screen in response to a gesture input of moving the first closed curve region onto the second closed curve region.

12. The method according to claim 1, wherein the AR screen combines a virtual image and the image corresponding to the environment.

13. The method according to claim 1, wherein the closed curve region is in a shape of one of a circle or an ellipse.

14. A mobile terminal comprising:

a wireless communication unit configured to communicatively connect the mobile terminal with a head mounted display for displaying an augmented reality (AR) screen; and

a controller configured to;

receive first data from the head mounted display, the received first data includes gesture input data acquired by a first camera at the head mounted display and image data of an image corresponding to an environment acquired by the first camera;

receive second data from the head mounted display, the received second data includes image data acquired by a second camera at the head mounted display;

analyze the first and second data;

determine a predetermined gesture based on the analyzed first data and determine an eye position of a user based on the analyzed second data; and

control the head mounted display to display an indicator corresponding to a shape of the determined gesture on the AR screen, wherein the indicator is displayed at the determined eye position of the user, and wherein the controller enables to display a closed curve region on the AR screen when the predetermined gesture is determined to correspond to a closed curve gesture.

15. The mobile terminal according to claim 14, wherein the second camera is configured to perform eye tracking of the user.

16. The mobile terminal according to claim 14, wherein the controller enables to display items belonging to a specific group in the closed curve region when a gesture input of touching a display and then performing a dragging operation to the closed curve region is detected.

17. The mobile terminal according to claim 14, wherein the controller divides the closed curve region into first and second closed curve regions and enables to display the divided closed curve regions on the AR screen, when a gesture input of twisting and then pushing the closed curve region outward is detected.

18. The mobile terminal according to claim 14, wherein the AR screen combines a virtual image and the image corresponding to the environment.

19. The mobile terminal according to claim 14, wherein the closed curve region is in a shape of one of a circle or an ellipse.

20. A mobile terminal comprising:

a wireless communication unit configured to communicatively connect the mobile terminal with a head mounted display for displaying an augmented reality (AR) screen; and

a controller configured to receive data from the head mounted display, the received data includes gesture input data acquired by a camera at the head mounted display and image data of an image corresponding to an environment acquired by the camera, the controller to analyze the data and determine a predetermined gesture based on the analyzed data, and to control the head mounted display on the AR screen based on the determined gesture,

wherein the controller enables to display a closed curve region on the AR screen when the predetermined gesture is determined to correspond to a closed curve gesture, and

wherein the controller enables to display a text input window for attaching a tag to the closed curve region on a display and sets text input through the text input window as a tag of the closed curve region, when a gesture input of tapping a boundary of the closed curve region is detected.

21. A system comprising:

a head mounted display configured to display an augmented reality (AR) screen; and

a mobile terminal communicatively connected to the head mounted display and configured to;

receive first data from the head mounted display, the received first data includes gesture input data acquired by a first camera at the head mounted display and image data of an image corresponding to an environment acquired by the first camera;

receive second data from the head mounted display, the received second data includes image data acquired by a second camera at the head mounted display;

analyze the first and second data;

determine a predetermined gesture based on the analyzed first data, and determine an eye position of a user based on the analyzed second data; and

control the head mounted display to display an indicator corresponding to a shape of the determined gesture on the AR screen, wherein the indicator is displayed at the determined eye position of the user, and wherein the mobile terminal controls the head mounted display to display a closed curve region on the AR screen when the predetermined gesture is determined to correspond to a closed curve gesture.

22. The system according to claim 21, wherein the second camera is configured to perform eye tracking of a user.

23. The system according to claim 21, wherein the AR screen combines a virtual image and the image corresponding to the environment.

24. The system according to claim 21, wherein the closed curve region is in a shape of one of a circle or an ellipse.