Mobile terminal and controlling method thereof

• US 9,069,760 B2
• Filed: 06/17/2011
• Issued: 06/30/2015
• Est. Priority Date: 08/24/2010
• Status: Active Grant

• Current Assignee: LG Electronics, Inc. (LG Corporation)
• Original Assignee: LG Electronics, Inc. (LG Corporation)
• Inventors: Jeong, Byoungzoo, Lee, Jiyeon, Kim, Hyoungsoo, Jee, Hyunho
• Primary Examiner(s): Tung, Kee M
• Assistant Examiner(s): Du, Haixia
• Application Number:
  US13/163,579
• Publication Number:
  US 20120050324A1
• Time in Patent Office: 1,474 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/00 Information retrieval; Database structures therefor; File system structures therefor
• CPC Class Codes: G06F 16/9537 Spatial or temporal dependent retrieval, e.g. spatiotemporal queries
• CPC Class Codes: G09G 2340/145 related to small screens
• CPC Class Codes: H04N 1/21 Intermediate information storage H04N1/387, H04N1/41 take precedence ; for control between transmitter and receiver or between image input and image output device H04N1/32358; indexing, editing G11B27/00

Abstract

A mobile terminal and controlling method thereof are disclosed by which a user can be informed of an event associated with augmented reality based on an occurrence of the event using various haptic effects that differ from each other.

Claims

1. A mobile terminal, comprising:

a camera configured to receive in real-time a preview image for an augmented reality (AR) function of the mobile terminal;

at least one haptic module configured to generate a plurality of haptic effects;

a display unit; and

a controller configured to;

obtain a current position of the mobile terminal;

control the display unit to display in real-time the preview image,

wherein the preview image includes a plurality of objects;

control the display unit to display a plurality of object information items in conjunction with the preview image, wherein each item of the plurality of object information items is descriptive of a characteristic of an associated object of the plurality of objects;

selecting a first object of the plurality of objects in response to a user input received at the mobile terminal;

control the at least one haptic module to generate at least one haptic effect that is indicative of distance between the current position of the mobile terminal and a real position of the selected first object;

control the display unit to display a search window in conjunction with displaying of at least a portion of the preview image, the search window for receiving user input of a search term;

cause the at least one haptic module to generate a second haptic effect when the search term corresponds to a second object of the plurality of objects included in the preview image or when the search term corresponds to a first item of the plurality of object information items; and

control the display unit to display the second object, or the first item, as shaking when the second haptic effect is generated.

2. The mobile terminal of claim 1, wherein the controller is further configured to:

control the at least one haptic module to generate at least one haptic effect indicating an existence of a hidden object or a hidden object information item that is virtually located behind the selected first object.

3. The mobile terminal of claim 1, wherein the controller is further configured to:

control the display unit to shift a position of the selected first object in response to an input;

control the at least one haptic module to generate at least one haptic effect indicating an existence of a hidden object or a hidden object information item that is virtually located behind the positioned-shifted selected first object.

4. The mobile terminal of claim 1, wherein the controller is further configured to control the at least one haptic module to change the at least one haptic effect related to the distance based on a change in the distance.

5. The mobile terminal of claim 1, wherein the controller is further configured to control the at least one haptic module to change a magnitude of the at least one haptic effect when the distance decreases.

6. The mobile terminal of claim 1, wherein the controller is further configured to:

control the display unit to display at least one new object or at least one new object information item in the preview image; and

control the at least one haptic module to generate at least one haptic effect indicating the displaying of the at least one new object or the at least one new object information item.

7. The mobile terminal of claim 6, wherein:

the controller is further configured to;

control the display unit to display a plurality of new objects or a plurality of new object information items; and

control the at least one haptic module to generate at least one haptic effect indicating distance between each of the plurality of new objects or between each of the plurality of new object information items.

8. The mobile terminal of claim 1, wherein the controller is further configured to:

control the display unit to change the preview image based on a change of a photographing view point of the camera; and

control the at least one haptic module to generate at least one haptic effect indicating the change of the preview image based on the change of the view point.

9. The mobile terminal of claim 1, further comprising an audio output module configured to output sound signals, wherein the controller is further configured to:

control the audio output module to output sound signals associated with the AR function; and

control the at least one haptic module to generate at least one haptic effect indicating the output of the sound signals.

10. The mobile terminal of claim 1, wherein:

the at least one haptic effect is a vibration; and

the controller is further configured to control the display unit to display a representation of the selected first object as shaking when the at least one haptic effect is generated.

11. The mobile terminal of claim 1, wherein the controller is further configured to control the at least one haptic module to generate at least one haptic effect indicating a direction in which a third object is virtually situated in relation to the mobile terminal when the third object does not exist in the preview image.

12. The mobile terminal of claim 11, wherein:

the at least one haptic module comprises a first haptic module and a second haptic module; and

the controller is further configured to control the first haptic module to generate a first haptic effect and to control the second haptic module to generate a third haptic effect to indicate the direction in which the third object is virtually situated in relation to the mobile terminal by adjusting a magnitude of the first haptic effect and the third haptic effect.

13. The mobile terminal of claim 11, wherein the controller is further configured to control the display unit to display information indicating the direction in which the third object is virtually situated in relation to the mobile terminal.

14. The mobile terminal of claim 1, wherein the controller is further configured to:

control the display unit to display a path between the current position of the mobile terminal to the real position of the first object; and

control the haptic module to generate at least one haptic effect indicating the displaying of the path.

15. The mobile terminal of claim 14, wherein the controller is further configured to:

control the haptic module to generate at least one haptic effect indicating deviation of the mobile terminal from the path.

16. The mobile terminal of claim 1, wherein the controller is further configured to:

determine a number of a plurality of new objects or a number of a plurality of new object information items in the preview image when the plurality of new objects or the plurality of new object information items are displayed in the preview image; and

control the at least one haptic module to generate a number of different haptic effects in order equal to the number of the plurality of new objects or the number of the plurality of new object information items, the different haptic effects having different haptic patterns corresponding to each of the plurality of new objects or each of the plurality of new object information items.

17. A method for controlling a mobile terminal, the method comprising:

receiving in real-time, via a camera, a preview image for an augmented reality (AR) function;

obtaining a current position of the mobile terminal;

displaying in real-time the preview image, wherein the preview image includes a plurality of objects;

displaying a plurality of object information items in conjunction with the preview image, wherein each item of the plurality of object information items is descriptive of a characteristic of an associated object of the plurality of objects;

selecting a first object of the plurality of objects in response to a user input received at the mobile terminal;

generating at least one haptic effect that is indicative of distance between the current position of the mobile terminal and a real position of the selected first object;

displaying a search window in conjunction with displaying of at least a portion of the preview image, the search window for receiving user input of a search term; and

generating a second haptic effect when the search term corresponds to a second object of the plurality of objects included in the preview image or when the search term corresponds to a first item of the plurality of object information items; and

displaying the second object, or the first item, as shaking when the second haptic effect is generated.

18. The method of claim 17, further comprising:

generating at least one haptic effect indicating an existence of a hidden object or a hidden object information item that is virtually located behind the selected first object.

19. The method of claim 17, further comprising:

shifting a position of the selected first object in response to an input;

generating at least one haptic effect indicating an existence of a hidden object or a hidden object information item that is virtually located behind the positioned-shifted selected first object.

20. The method of claim 17, further comprising:

changing the at least one haptic effect related to the distance based on a change in the distance.

21. The method of claim 17, further comprising:

changing a magnitude of the at least one haptic effect when the distance decreases.

22. The method of claim 17, further comprising:

displaying at least one new object or at least one new object information item in the preview image; and

generating at least one haptic effect indicating the displaying of the at least one new object or the at least one new object information item.

23. The method of claim 22, further comprising:

displaying a plurality of new objects or a plurality of new object information items; and

generating at least one haptic effect indicating distance between each of the plurality of new objects or between each of the plurality of new object information items.

24. The method of claim 17 further comprising:

changing the preview image based on a change of a photographing view point of the camera; and

generating at least one haptic effect indicating the change of the preview image based on the change of the photographing view point.

25. The method of claim 17, further comprising:

outputting sound signals associated with the AR function; and

generating at least one haptic effect indicating the outputting of the sound signals.

26. The method of claim 17, wherein the at least one haptic effect is a vibration, the method further comprising:

displaying a representation of the selected first object as shaking when the at least one haptic effect is generated.

27. The method of claim 17, further comprising:

generating at least one haptic effect indicating a direction in which a third object is virtually situated in relation to the mobile terminal when the third object does not exist in the preview image.

28. The method of claim 27, further comprising:

controlling a first haptic module to generate a first haptic effect; and

controlling a second haptic module to generate a second haptic effect to indicate the direction in which the third object is virtually situated in relation to the mobile terminal by adjusting a magnitude of the first haptic effect and the second haptic effect generated by the second haptic module.

29. The method of claim 27, further comprising:

displaying information indicating the direction in which the third object is virtually situated in relation to the mobile terminal.

30. The method of claim 17, further comprising:

displaying a path between the current position of the mobile terminal to the real position of the first object; and

generating at least one haptic effect indicating the displaying of the path.

31. The method of claim 30, further comprising:

generating at least one haptic effect indicating deviation of the mobile terminal from the path.

32. The method of claim 27, further comprising:

determining a number of a plurality of new objects or a number of a plurality of new object information items in the preview image when the plurality of new objects or the plurality of new object information items are displayed in the preview image; and

generating a number of different haptic effects in order equal to the number of the plurality of new objects or the number of the plurality of new object information items, the different haptic effects having different haptic patterns corresponding to each of the plurality of new objects or each of the plurality of new object information items.