Mobile terminal for controlling various operations using a stereoscopic 3D pointer on a stereoscopic 3D image and control method thereof

• US 9,513,710 B2
• Filed: 09/01/2011
• Issued: 12/06/2016
• Est. Priority Date: 09/15/2010
• Status: Active Grant

• Current Assignee: LG Electronics, Inc. (LG Corporation)
• Original Assignee: LG Electronics, Inc. (LG Corporation)
• Inventors: Lee, Jinsool, Kim, Dongok, Kim, Taeyun, Seen, Seungmin
• Primary Examiner(s): To, Jennifer
• Assistant Examiner(s): Burwell, Joseph R
• Application Number:
  US13/223,880
• Publication Number:
  US 20120062558A1
• Time in Patent Office: 1,923 Days
• Field of Search: 715/700
• US Class Current: 1/1
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/04812 Interaction techniques based on cursor appearance or behaviour, e.g. being affected by the presence of displayed objects
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object

Abstract

A mobile terminal and a method for controlling the operation of the mobile terminal are provided. In the method, a stereoscopic 3D image that utilizes binocular disparity between a left-eye image and a right-eye image is displayed on a display and a stereoscopic 3D pointer is displayed in the stereoscopic 3D image when a user'"'"'s hand is detected according to analysis of visual data acquired by a camera. The position of the stereoscopic 3D pointer is moved according to movement of the hand when the movement of the hand is detected according to analysis of the visual data. According to this method, the operation of the mobile terminal can be controlled by the user'"'"'s hand movement or gesture.

Claims

1. A method of a mobile terminal, the method comprising:

displaying a perceived three dimensional (3D) image on a display;

receiving first visual data of an indicator from at least one camera;

displaying a 3D pointer in the displayed 3D image based on the received first visual data of the indicator;

receiving second visual data of the indicator from the at least one camera;

changing a position of the displayed 3D pointer on the display based on the received second visual data;

displaying depth information corresponding to a depth of the 3D pointer in the 3D image while the 3D pointer is moved, and not displaying the depth information in the 3D image while the 3D pointer is stopped;

outputting a vibration pattern corresponding to a new layer each time the 3D pointer reaches the new layer; and

performing an operation associated with an object indicated by the 3D pointer, wherein the performing of the operation includes performing the operation in response to a predetermined gesture when the mobile terminal is a first operation mode, and performing the operation in response to a determination that the indicator has not moved for a predetermined time when the mobile terminal is a second operation mode,

wherein the 3D image is divided into a plurality of layers located at different depths and one or more icons are arranged at each layer, and

wherein an icon located in the same layer as the 3D pointer is displayed with a different color or shape than other icons located at other layers such that a user of the mobile terminal can easily identify the icon located in the same layer as the 3D pointer.

2. The method of claim 1, wherein the indicator is a hand.

3. The method of claim 1, wherein changing the position of the displayed 3D pointer includes changing the position of the displayed 3D pointer based on movement of the indicator.

4. The method of claim 1, further comprising:

detecting a gesture based on visual data received from the camera, wherein performing the operation associated with the object indicated by the 3D pointer includes performing the operation based on the detected gesture.

5. The method of claim 1, further comprising changing a 3D depth of the displayed 3D pointer based on movement of the indicator.

6. The method of claim 1, wherein the determination that the indicator has not moved for the predetermined time is based on visual data received from the camera.

7. The method of claim 1, further comprising outputting a feedback signal in response to a change of a depth of the 3D pointer.

8. The method of claim 1, further comprising displaying a menu for setting a function corresponding to a specific gesture of the indicator.

9. The method of claim 1, wherein the color or shape of the 3D pointer is changed when the 3D pointer moves to a specific layer set by a user based on the received second visual data without further interaction of the indicator.

10. The method of claim 1, further comprising:

displaying a guide icon for initially setting a size of the 3D pointer on the 3D image, and

in response to a movement of the indicator, moving the 3D pointer into the guide icon.

11. The method of claim 1, further comprising:

providing a menu for adjusting a transparency of the 3D pointer.

12. The method of claim 1, wherein the depth information is displayed as a number or a bar graph.

13. A mobile terminal comprising:

a display configured to display a three-dimensional (3D) image;

at least one camera configured to acquire visual data; and

a controller configured to;

receive, from the camera, first visual data of an indicator,

display a 3D pointer in the 3D image based on the received first visual data of the indicator,

receive second visual data of the indicator from the camera,

change a position of the displayed 3D pointer based on the received second visual data,

display depth information corresponding to a depth of the 3D pointer in the 3D image while the 3D pointer is moved, and not display the depth information in the 3D image while the 3D pointer is stopped,

output a vibration pattern corresponding to a new layer each time the 3D pointer reaches the new layer, and

perform an operation associated with an object indicated by the 3D pointer,

wherein the 3D image is divided into a plurality of layers located at different depths and one or more icons are arranged at each layer,

wherein an icon located in the same layer as the 3D pointer is displayed with a different color or shape than other icons located at other layers such that a user of the mobile terminal can easily identify the icon located in the same layer as the 3D pointer, and

wherein the controller is further configured to;

in response to a predetermined gesture when the mobile terminal is a first operation mode, perform the operation associated with the object indicated by the 3D pointer, and

in response to a determination that the indicator has not moved for a predetermined time when the mobile terminal is a second operation mode, perform the operation associated with the object indicated by the 3D pointer.

14. The mobile terminal of claim 13, wherein the indicator is a hand.

15. The mobile terminal of claim 13, wherein the controller changes the position of the displayed 3D pointer based on movement of the indicator.

16. The mobile terminal of claim 13, wherein the controller detects a gesture based on visual data received from the camera, and the controller performs the operation associated with the object indicated by the 3D pointer based on the detected gesture.

17. The mobile terminal of claim 13, wherein the controller determines that the indicator has not moved for a predetermined time based on visual data received from the camera.

18. The mobile terminal of claim 13, wherein the controller generates visual effects to the 3D pointer as the 3D pointer is closer to a specific object.

19. The mobile terminal of claim 13, further comprising a haptic module to generate tactile effects, wherein the controller controls the haptic module to generate tactile effects based on a change of a depth of the 3D pointer.

20. A display method of a mobile terminal, the method comprising:

displaying a three dimensional (3D) image on a display;

displaying a 3D pointer on the 3D image;

receiving data of a user'"'"'s hand from a camera;

changing the displayed 3D pointer based on the received data of the user'"'"'s hand;

displaying depth information corresponding to a depth of the 3D pointer in the 3D image while the 3D pointer is moved, and not displaying the depth information in the 3D image while the 3D pointer is stopped;

outputting a vibration pattern corresponding to a new layer each time the 3D pointer reaches the new layer; and

performing an operation associated with an object indicated by the 3D pointer, wherein the performing of the operation includes performing the operation in response to a predetermined gesture when the mobile terminal is a first operation mode, and performing the operation in response to a determination that the hand has not moved for a predetermined time when the mobile terminal is a second operation mode,

wherein the 3D image is divided into a plurality of layers located at different depths and one or more icons are arranged at each layer, and

wherein an icon located in the same layer as the 3D pointer is displayed with a different color or shape than other icons located at other layers such that a user of the mobile terminal can easily identify the icon located in the same layer as the 3D pointer.

21. The method of claim 20, wherein changing the displayed 3D pointer includes changing a position of the displayed 3D pointer based on movement of the hand.

22. The method of claim 20, further comprising:

detecting a gesture based on data received from the camera, wherein performing the operation associated with the object indicated by the 3D pointer includes performing the operation based on the detected gesture.

23. The method of claim 20, further comprising changing a 3D depth of the displayed 3D pointer based on movement of the user'"'"'s hand.

24. The method of claim 20, wherein the determination that the hand has not moved for the predetermined time is based on data received from the camera.

25. The method of claim 20, further comprising selecting an object on the 3D image using the displayed 3D pointer.