MOBILE TERMINAL AND CONTROLLING METHOD THEREOF

• US 20120050324A1
• Filed: 06/17/2011
• Published: 03/01/2012
• Est. Priority Date: 08/24/2010
• Status: Active Grant

• Current Assignee: LG Electronics, Inc. (LG Corporation)
• Original Assignee: LG Electronics, Inc. (LG Corporation)
• Inventors: Lee, Jiyeon, JEONG, Byoungzoo, Kim, Hyoungsoo, Jee, Hyunho
• Granted Patent: US 9,069,760 B2
• Time in Patent Office: Days
• US Class Current: 345/633
• CPC Class Codes: G06F 16/00 Information retrieval; Database structures therefor; File system structures therefor
• CPC Class Codes: G06F 16/9537 Spatial or temporal dependent retrieval, e.g. spatiotemporal queries
• CPC Class Codes: G09G 2340/145 related to small screens
• CPC Class Codes: H04N 1/21 Intermediate information storage H04N1/387, H04N1/41 take precedence ; for control between transmitter and receiver or between image input and image output device H04N1/32358; indexing, editing G11B27/00

Abstract

A mobile terminal and controlling method thereof are disclosed by which a user can be informed of an event associated with augmented reality based on an occurrence of the event using various haptic effects that differ from each other.

Claims

1. A mobile terminal, comprising:

a camera configured for capturing an image for an augmented reality (AR) function of the mobile terminal;

a position location module configured for determining a current position of the mobile terminal;

at least one haptic module configured for generating a plurality of haptic effects;

a display unit configured for displaying the captured image as a preview image; and

a controller configured for;

controlling the display unit to display at least one object included in the preview image and at least one object information item related to the at least one object based on the current position of the mobile terminal; and

controlling the at least one haptic module to generate at least one haptic effect indicating an occurrence of an event associated with the AR function.

2. The mobile terminal of claim 1, wherein the controller is further configured for:

recognizing a selection of a specific object of the at least one object included in the preview image or a selection of a specific object information item of the at least one object information item displayed in the preview image as the event;

recognizing an existence of at least one hidden object or at least one hidden object information item in the preview image behind the selected specific object or the selected specific object information item; and

controlling the at least one haptic module to generate at least one haptic effect indicating the existence of the hidden object or the hidden object information item.

3. The mobile terminal of claim 1, wherein the controller is further configured for:

receiving a drag input for shifting a position of a selected object of the at least one object included in the preview image or a selected object information item of the at least one object information item displayed in the preview image in a specific direction on the display unit;

recognizing the shifting of the position of the selected object or the selected object information item as the event;

recognizing an existence of at least one hidden object or at least one hidden object information item in the preview image behind the dragged selected object or the dragged selected object information item; and

controlling the at least one haptic module to generate at least one haptic effect indicating the existence of the hidden object or the hidden object information item based on a distance of the drag input.

4. The mobile terminal of claim 1, wherein the controller is further configured for:

recognizing a selection of an object of the at least one object included in the preview image as the event;

determining a distance between the mobile terminal and the selected object; and

generating at least one haptic effect indicating the determined distance.

5. The mobile terminal of claim 4, wherein the controller is further configured for generating different haptic effects based on the determined distance between the mobile terminal and the selected object changing.

6. The mobile terminal of claim 4, wherein the controller is further configured for changing a magnitude of the haptic effect when the determined distance between the mobile terminal and the selected object decreases.

7. The mobile terminal of claim 1, wherein the controller is further configured for:

controlling the display unit to display at least one new object or at least one new object information item in the preview image; and

controlling the at least one haptic module to generate at least one haptic effect indicating the display of the new object or the new object information item.

8. The mobile terminal of claim 7, wherein:

a plurality of new objects are displayed or a plurality of new object information items are displayed; and

the controller is further configured for;

determining a distance between each of the plurality of new objects or each of the plurality of new object information items; and

generating at least one haptic effect indicating the determined distance between each of the plurality of new objects or each of the plurality of new object information items.

9. The mobile terminal of claim 1, wherein the controller is further configured for:

recognizing a change of a photographing view point of the camera;

controlling the display unit to change the preview image based on the change of the photographing view point;

recognizing the change of the preview image as the event; and

controlling the at least one haptic module to generate at least one haptic effect indicating the change of the preview image.

10. The mobile terminal of claim 9, wherein the controller is further configured for:

determining if the at least one object or the at least one object information item included in the preview image changes based on the change of the preview image; and

generating at least one haptic effect indicating that the at least one object or the at least one object information has changed.

11. The mobile terminal of claim 1, further comprising an audio output module configured for outputting sound signals, wherein the controller is further configured for:

controlling the audio output module to output sound signals associated with the AR function;

recognizing the output of the sound signals as the event; and

generating the at least one haptic effect indicating the output of the sound signals.

12. The mobile terminal of claim 1, wherein the controller is further configured for:

receiving input of a command to search for a specific object or a specific object information item;

recognizing the input of the command as the event;

controlling a search for the specific object or the specific object information item corresponding to the command; and

controlling the display unit to display the specific object or the specific object information item when the specific object or the specific object information item is found.

13. The mobile terminal of claim 12, wherein the controller is further configured for controlling the at least one haptic module to generate at least one haptic effect when the specific object or the specific object information item exists in the preview image.

14. The mobile terminal of claim 13, wherein:

the at least one haptic effect related to the specific object or the specific object information item existing in the preview image is a vibration; and

the controller is further configured for controlling the display unit to display a representation of the specific object or the specific object information item as shaking when the at least one haptic effect related to the specific object or the specific object information item existing in the preview image is generated.

15. The mobile terminal of claim 12, wherein the controller is further configured for controlling the at least one haptic module to generate at least one haptic effect indicating a direction in which the specific object or the specific object information item is situated in relation to the mobile terminal when the specific object or the specific object information item does not exist in the preview image.

16. The mobile terminal of claim 15, wherein:

the mobile terminal comprises a first haptic module and a second haptic module; and

the controller is further configured to control the first haptic module to generate a first haptic effect and to control the second haptic module to generate a second haptic effect to indicate the direction in which the specific object or the specific object information item is situated in relation to the mobile terminal by adjusting a magnitude of the first haptic effect and the second haptic effect.

17. The mobile terminal of claim 15, wherein the controller is further configured to control the display unit to display information indicating the direction in which the specific object or the specific object information item is situated in relation to the mobile terminal.

18. The mobile terminal of claim 1, wherein the controller is further configured for:

receiving input of a request for directions leading from a current location of the mobile terminal to a destination of a specific object of the displayed at least one object or a specific object information item of the displayed at least one object information item;

controlling the display unit to display a path leading from the current location of the mobile terminal to the specific object or the specific object information item;

recognizing the display of the path as the event; and

controlling the haptic module to generate at least one haptic effect indicating the display of the path.

19. The mobile terminal of claim 18, wherein the controller is further configured for:

recognizing when the mobile terminal deviates from the path; and

controlling the haptic module to generate at least one haptic effect indicating the deviation of the mobile terminal from the path.

20. A method of controlling a mobile terminal, the method comprising:

capturing an image for an augmented reality (AR) function via a camera of the mobile terminal;

displaying the captured image as a preview image on a display unit of the mobile terminal;

determining a current position of the mobile terminal via a position location module of the mobile terminal;

displaying at least one object included in the preview image and at least one object information item related to at least one object based on the current position of the mobile terminal;

detecting an occurrence of an event associated with the AR function via a controller; and

generating at least one haptic effect, via a haptic module, indicating the occurrence of the event.

21. The method of claim 20, further comprising:

recognizing, via the controller, a selection of a specific object of the at least one object included in the preview image or a selection of a specific object information item of the at least one object information item displayed in the preview image as the event;

recognizing, via the controller, an existence of at least one hidden object or at least one hidden object information item in the preview image behind the selected specific object or the selected specific object information item; and

controlling, via the controller, the at least one haptic module to generate at least one haptic effect indicating the existence of the hidden object or the hidden object information item.

22. The method of claim 20, further comprising:

receiving, via the controller, a drag input for shifting a position of a selected object of the at least one object included in the preview image or a selected object information item of the at least one object information item displayed in the preview image in a specific direction on the display unit;

recognizing, via the controller, the shifting of the position of the selected object or the selected object information item as the event;

recognizing, via the controller, an existence of at least one hidden object or at least one hidden object information item in the preview image behind the dragged selected object or the dragged selected object information item; and

controlling, via the controller, the at least one haptic module to generate at least one haptic effect indicating the existence of the hidden object or the hidden object information item based on a distance of the drag input.

23. The method of claim 20, further comprising:

recognizing, via the controller, a selection of an object of the at least one object included in the preview image as the event;

determining a distance between the mobile terminal and the selected object via the controller; and

controlling the at least one haptic module, via the controller, to generate at least one haptic effect indicating the determined distance.

24. The method of claim 23, further comprising controlling the at least one haptic module, via the controller, to generate different haptic effects based on the determined distance between the mobile terminal and the selected object changing.

25. The method of claim 23, further comprising controlling the at least one haptic module, via the controller, to change a magnitude of the haptic effect when the determined distance between the mobile terminal and the selected object decreases.

26. The method of claim 20, further comprising:

controlling the display unit, via the controller, to display at least one new object or at least one new object information item in the preview image; and

controlling the at least one haptic module, via the controller, to generate at least one haptic effect indicating the display of the new object or the new object information item.

27. The method of claim 26, wherein a plurality of new objects are displayed or a plurality of new object information items are displayed, the method further comprising:

determining a distance between each of the plurality of new objects or each of the plurality of new object information items via the controller; and

controlling the at least one haptic module, via the controller, to generate at least one haptic effect indicating the determined distance between each of the plurality of new objects or each of the plurality of new object information items.

28. The method of claim 20, further comprising:

recognizing a change of a photographing view point of the camera via the controller;

controlling the display unit, via the controller, to change the preview image based on the change of the photographing view point;

recognizing the change of the preview image as the event via the controller; and

controlling the at least one haptic module, via the controller, to generate at least one haptic effect indicating the change of the preview image.

29. The method of claim 28, further comprising:

determining, via the controller, if the at least one object or the at least one object information item included in the preview image changes based on the change of the preview image; and

controlling the at least one haptic module, via the controller, to generate at least one haptic effect indicating that the at least one object or the at least one object information has changed.

30. The method of claim 20, further comprising:

controlling, via the controller, an audio output module to output sound signals associated with the AR function;

recognizing the output of the sound signals as the event via the controller; and

controlling the at least one haptic module, via the controller, to generate the at least one haptic effect indicating the output of the sound signals.

31. The method of claim 20, further comprising:

receiving, via the controller, input of a command to search for a specific object or a specific object information item;

recognizing the input of the command as the event via the controller;

controlling a search for the specific object or the specific object information item corresponding to the command via the controller; and

controlling the display unit, via the controller, to display the specific object or the specific object information item when the specific object or the specific object information item is found.

32. The method of claim 31, further comprising controlling the at least one haptic module, via the controller, to generate at least one haptic effect when the specific object or the specific object information item exists in the preview image.

33. The method of claim 32, wherein the at least one haptic effect related to the specific object or the specific object information item existing in the preview image is a vibration, the method further comprising:

controlling the display unit, via the controller, to display a representation of the specific object or the specific object information item as shaking when the at least one haptic effect related to the specific object or the specific object information item existing in the preview image is generated.

34. The method of claim 31, further comprising controlling the at least one haptic module, via the controller, to generate at least one haptic effect indicating a direction in which the specific object or the specific object information item is situated in relation to the mobile terminal when the specific object or the specific object information item does not exist in the preview image.

35. The method of claim 34, wherein the mobile terminal comprises a first haptic module and a second haptic module, the method further comprising:

controlling the first haptic module, via the controller, to generate a first haptic effect;

controlling the second haptic module, via the controller, to generate a second haptic effect; and

adjusting a magnitude of the first haptic effect and the second haptic effect, via the controller, to indicate the direction in which the specific object or the specific object information item is situated in relation to the mobile terminal.

36. The method of claim 34, further comprising controlling the display unit, via the controller, to display information indicating the direction in which the specific object or the specific object information item is situated in relation to the mobile terminal.

37. The method of claim 20, further comprising:

receiving, via the controller, input of a request for directions leading from a current location of the mobile terminal to a destination of a specific object of the displayed at least one object or a specific object information item of the displayed at least one object information item;

controlling the display unit, via the controller, to display a path leading from the current location of the mobile terminal to the specific object or the specific object information item;

recognizing the display of the path as the event via the controller; and

controlling the haptic module, via the controller, to generate at least one haptic effect indicating the display of the path.

38. The method of claim 37, further comprising:

recognizing when the mobile terminal deviates from the path via the controller; and

controlling the haptic module, via the controller, to generate at least one haptic effect indicating the deviation of the mobile terminal from the path.