Mobile terminal and controlling method thereof

• US 10,495,878 B2
• Filed: 11/19/2015
• Issued: 12/03/2019
• Est. Priority Date: 04/28/2015
• Status: Active Grant

• Current Assignee: LG Electronics, Inc. (LG Corporation)
• Original Assignee: LG Electronics, Inc. (LG Corporation)
• Inventors: Chi, Jumin, Hong, Sukhyun
• Primary Examiner(s): Haley, Joseph R
• Application Number:
  US15/570,199
• Publication Number:
  US 20180136465A1
• Time in Patent Office: 1,475 Days
• Field of Search: None
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/0416 Control or interface arrangements specially adapted for digitisers

Abstract

Disclosed are a mobile terminal and a controlling method thereof. The mobile terminal includes a camera for sequentially acquiring a first image for augmented reality (AR), which include at least one object, a display unit for outputting the first image, and a controller for, if an input for selecting a specific object among the at least one object included in the first image is received, storing information related to the selected specific object. If a virtual reality (VR) mode corresponding to the first image is executed, the controller forms and outputs a second image corresponding to the VR mode, based on the stored information. If update information related to the selected specific object is received from an external server, the controller controls a mark change corresponding to the received update information to be marked on the second image.

Claims

1. A mobile terminal comprising:

a camera configured to sequentially acquire a first image for augmented reality (AR), which include at least one object;

a display unit configured to output the first image; and

a controller configured to, if an input for selecting a specific object among the at least one object included in the first image is received, store information related to the selected specific object,

wherein, if a virtual reality (VR) mode corresponding to the first image is executed, the controller forms and outputs a second image corresponding to the VR mode, based on the stored information, and, if update information related to the selected specific object is received from an external server, the controller controls a mark change corresponding to the received update information to be marked on the second image, and

wherein, if a first input is received, the controller executes an AR mode for sequentially acquiring the first image by activating a camera provided outside a main body of the mobile terminal, and, if a second input is received, the controller terminates the AR mode and executes the VR mode corresponding to the first image.

2. The mobile terminal of claim 1, further comprising a tracking unit configured to track a user'"'"'s head or user'"'"'s eyes based on image information acquired through a camera provided inside a main body of the mobile terminal and direction information acquired through a gyro sensor provided in the mobile terminal,

wherein the controller displays a corresponding image object by detecting a specific object to which the user'"'"'s head or eyes are fixed in the first image through the tracking unit, and, if a third input is received in the state in which the image object is displayed, the controller selects the detected specific object.

3. The mobile terminal of claim 2, wherein, if the detected specific object is selected as the third input is received, the controller controls a mark change corresponding to the selection to be marked on the first image.

4. The mobile terminal of claim 3, wherein the main body of the mobile terminal is formed to be wearable on the user'"'"'s head or coupled to a wearable mechanism formed to be wearable on the user'"'"'s head, and

wherein, if a predetermined voice command or wink is sensed in a state in which an input is applied to an input unit provided in the main body of the mobile terminal or the wearable mechanism, the controller recognizes that the third input for selecting the specific object has been received.

5. The mobile terminal of claim 3, wherein, if the detected specific object is selected in plurality and if a fourth input is received, the controller executes a screen mode for comparing the plurality of selected specific objects, and gathers and outputs, on the first image, images corresponding to the plurality of selected specific objects.

6. The mobile terminal of claim 1, wherein the information related to the selected specific object includes at least one of a position at which the selected specific object is located, a user'"'"'s visiting time, and corresponding product information.

7. The mobile terminal of claim 1, wherein, if the VR mode is executed, the controller arranges image objects marked on the second image, based on at least one of position information of the selected specific object and user'"'"'s movement information in the first image.

8. The mobile terminal of claim 7, wherein, if the VR mode is executed, the controller activates a camera provided inside the main body of the mobile terminal to track movement of the user'"'"'s pupils.

9. The mobile terminal of claim 8, wherein, if the VR mode is executed, the controller marks a visually distinguished indicator around an image object which is to enter a new space among the marked image objects.

10. The mobile terminal of claim 9, wherein, in the VR mode, if a predetermined movement of the user'"'"'s pupils is sensed in a state in which the user'"'"'s head or eyes are fixed to any one of the image objects, around which the indicator is marked, the controller controls a mark change for notifying that the user has entered a new virtual space to be marked on the second image.

11. The mobile terminal of claim 7, wherein, if the VR mode is executed, the controller senses a user'"'"'s hand gesture by activating the camera provided outside the main body of the mobile terminal, and

wherein, if any one of the image objects marked on the second image is pointed using a virtual hand image moving along the sensed hand gesture, the controller provides a preview image of a virtual space corresponding to the pointed image object.

12. The mobile terminal of claim 1, wherein, if a predetermined input signal is received in the VR mode, the controller gathers virtual images corresponding to the selected specific object and marks the gathered virtual images on the second image, and

wherein a notification icon for notifying, in real time, update information related to the selected specific object is marked together with the virtual images.

13. The mobile terminal of claim 12, wherein, in the VR mode, if the user'"'"'s head or eyes are fixed to any one of the virtual images and if a predetermined movement of the user'"'"'s pupils is sensed, the controller controls the user to enter a virtual space in which a specific object corresponding to the one virtual image is located, and executes a process for purchasing a product corresponding to the specific object.

14. The mobile terminal of claim 13, wherein, if the process for purchasing the product is executed, the controller performs user authentication, based on at least one of a user'"'"'s iris scan, a user'"'"'s eyeball radius, a distance between the user'"'"'s pupils, and a kappa angle, using the camera provided inside the main body of the mobile terminal.

15. The mobile terminal of claim 13, wherein, if the process for purchasing the product is executed, the controller displays option information related to purchase among update information related to a specific object corresponding to the one virtual image, and controls the displayed option information to be selected based on movement of the user'"'"'s pupils.

16. The mobile terminal of claim 12, wherein, in the VR mode, if the movement of the user'"'"'s pupils faces a predetermined direction in a state in which the user enters a virtual space in which a specific object corresponding to the one virtual image is located, the controller controls the user to move, along the sensed direction, to a virtual space in which an object corresponding to another AR image is located.

17. The mobile terminal of claim 12, wherein, in the VR mode, when an event related to another virtual image occurs in a state in which the user enters a virtual space in which a specific object corresponding to the one virtual image is located, the controller controls a virtual icon for notifying that the event has occurred to be popped up.

18. The mobile terminal of claim 17, wherein, if the user'"'"'s head or eyes are fixed to the virtual icon and if a predetermined movement of the user'"'"'s pupils is sensed, the controller controls a current image to move to a virtual space in which a specific object corresponding to the virtual icon is located.

19. The mobile terminal of claim 12, wherein, in the VR mode, if the user enters a virtual space in which a specific object corresponding to the one virtual image is located, the controller displays a thumbnail image corresponding to a virtual space which the user previously visited to be overlapped on a current image.