Method for providing a user interface using three-dimensional gestures and an apparatus using the same

• US 8,612,894 B2
• Filed: 10/13/2009
• Issued: 12/17/2013
• Est. Priority Date: 10/13/2008
• Status: Active Grant

• Current Assignee: LG Electronics, Inc. (LG Corporation)
• Original Assignee: LG Electronics, Inc. (LG Corporation)
• Inventors: Kim, Hyun Ju
• Primary Examiner(s): Abdul-Ali, Omar
• Assistant Examiner(s): Jiang, Haimei
• Application Number:
  US12/577,978
• Publication Number:
  US 20100095206A1
• Time in Patent Office: 1,526 Days
• Field of Search: 715/863
• US Class Current: 715/863
• CPC Class Codes: G06F 2203/04101 2.5D-digitiser, i.e. digitiser detecting the X/Y position of the input means, finger or stylus, also when it does not touch, but is proximate to the digitiser's interaction surface and also measures the distance of the input means within a short range in the Z direction, possibly with a separate measurement setup
• CPC Class Codes: G06F 2203/04108 Touchless 2D- digitiser, i.e. digitiser detecting the X/Y position of the input means, finger or stylus, also when it does not touch, but is proximate to the digitiser's interaction surface without distance measurement in the Z direction
• CPC Class Codes: G06F 2203/04808 Several contacts: gestures triggering a specific function, e.g. scrolling, zooming, right-click, when the user establishes several contacts with the surface simultaneously; e.g. using several fingers or a combination of fingers and pen
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/043 using propagating acoustic waves
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text

Abstract

Provided is a method capable of making various modifications to widgets, graphic objects, or images, which are displayed on a display device, according to motions of a plurality of input units such as finger or stylus pen, with the use of a three-dimensional multi-sensor configured to detect the motions of the input units in a space, without touching the display device.

Claims

1. A method for controlling a user interface of a mobile device, the method comprising:

controlling display, on a user interface of a mobile device, of a main menu including widgets that each have a depth within a three-dimensional menu space and that each control execution of a specific application on the mobile device;

selecting, from among the widgets in the main menu and in a manner that does not cause execution of a specific application on the mobile device, a widget;

at a time when the widget is selected, detecting, by a controller, a user'"'"'s gesture related to modifying a depth of the selected widget within the three-dimensional menu space of the main menu; and

based on detecting the user'"'"'s gesture related to modifying the depth of the selected widget, modifying, on the user interface of the mobile device, the depth of the selected widget within the three-dimensional menu space of the main menu such that, when the mobile device displays the main menu, the selected widget is displayed within the three-dimensional menu space of the main menu at the modified depth, which is different than depths of other widgets within the three-dimensional menu space of the main menu.

2. The method according to claim 1:

wherein detecting a user'"'"'s gesture related to modifying a depth of the selected widget comprises detecting a gesture of changing a distance between a user input unit and the mobile device in a space over the selected widget; and

wherein modifying, on the user interface of the mobile device, the depth of the selected widget comprises modifying, on the user interface of the mobile device, the depth of the selected widget based on detecting the gesture of changing the distance between the user input unit and the mobile device in the space over the selected widget.

3. The method according to claim 2:

wherein detecting a gesture of changing a distance between a user input unit and the mobile device in a space over the selected widget comprises detecting a gesture of approaching at least one user input unit to the mobile device or departing the at least one user input unit from the mobile device; and

wherein modifying, on the user interface of the mobile device, the depth of the selected widget comprises modifying, on the user interface of the mobile device, the depth of the selected widget based on detecting the gesture of approaching at least one user input unit to the mobile device or departing the at least one user input unit from the mobile device.

4. The method according to claim 1, further comprising:

detecting a user'"'"'s gesture of rotating at least one user input unit in a space over the selected widget; and

rotating the selected widget according to the user'"'"'s gesture of rotating.

5. The method according to claim 1:

wherein controlling display, on a user interface of a mobile device, of a main menu including widgets that each have a depth within a three-dimensional menu space comprises controlling display, on a user interface of a mobile device, of a main menu including widgets that have different sizes; and

wherein modifying, on the user interface of the mobile device, the depth of the selected widget comprises modifying, on the user interface of the mobile device, a size of the selected widget such that, when the mobile device displays the three-dimensional menu space of the main menu, the selected widget is displayed within the three-dimensional menu space of the main menu at the modified size, which is different than sizes of other widgets within the three-dimensional menu space of the main menu.

6. The method according to claim 1:

wherein controlling display, on a user interface of a mobile device, of a main menu including widgets that each have a depth within a three-dimensional menu space comprises controlling display, on a user interface of a mobile device, of a main menu including widgets that have different shadows; and

wherein modifying, on the user interface of the mobile device, the depth of the selected widget comprises modifying, on the user interface of the mobile device, a shadow of the selected widget such that, when the mobile device displays the three-dimensional menu space of the main menu, the selected widget is displayed within the three-dimensional menu space of the main menu with the modified shadow, which is different than shadows of other widgets within the three-dimensional menu space of the main menu.

7. The method of claim 1:

wherein controlling display, on a user interface of a mobile device, of a main menu including widgets that each have a depth within a three-dimensional menu space comprises controlling display, on a user interface of a mobile device, of a main menu including widgets using at least three discrete levels of depth within the three-dimensional menu space, each of the at least three discrete levels of depth representing a different predetermined depth within the three-dimensional menu space; and

wherein modifying, on the user interface of the mobile device, the depth of the selected widget comprises changing, on the user interface of the mobile device, in which of the at least three discrete levels of depth the selected widget is classified such that, when the mobile device displays the three-dimensional menu space of the main menu, the selected widget is displayed within the main menu at the changed level of depth, which is different than levels of depth of other widgets within the three-dimensional menu space of the main menu.

15. The method according to claim 1, wherein modifying, on the user interface of the mobile device, the depth of the selected widget further comprises modifying an area on the user interface of the mobile device at which the selected widget is pressed to cause execution of a specific application on the mobile device associated with the selected widget.

17. The method according to claim 1, further comprising:

detecting a user'"'"'s gesture of rotating at least one user input unit in a space over the selected widget;

determining whether a direction of rotation of the at least one user input unit in the space over the selected widget is in;

a first plane perpendicular to the user interface,

a second plane parallel to the user interface, or

a third plane that is tilted with respect to the user interface such that the third plane is neither perpendicular nor parallel to the user interface;

based on determining that the direction of rotation of the at least one user input unit in the space over the selected widget is in the first plane perpendicular to the user interface, rotating the selected widget, according to the user'"'"'s gesture of rotating, about a first virtual axis parallel to the user interface;

based on determining that the direction of rotation of the at least one user input unit in the space over the selected widget is in the second plane parallel to the user interface, rotating the selected widget, according to the user'"'"'s gesture of rotating, about a second virtual axis perpendicular to the user interface; and

based on determining that the direction of rotation of the at last one user input unit in the space over the selected widget is in the third plane that is tilted with respect to the user interface, rotating the selected widget, according to the user'"'"'s gesture of rotating, about a third virtual axis sloped with respect to the user interface.

18. The method according to claim 1, wherein selecting the widget comprises selecting a widget that is defined as a rectangular parallelepiped widget, further comprising:

detecting a user'"'"'s gesture of rotating at least one user input unit in a space over the selected widget; and

in response to detecting the user'"'"'s gesture of rotating at least one user input unit in the space over the selected widget;

rotating the rectangular parallelepiped widget about an axis to change, from a first side of the rectangular parallelepiped widget, to a second side of the rectangular parallelepiped widget, and

modifying, on the user interface of the mobile device, a display of the selected widget such that, when the mobile device displays the three-dimensional menu space of the main menu, the selected widget is displayed within the three-dimensional menu space of the main menu at the second side of the rectangular parallelepiped widget, rather than the first side of the rectangular parallelepiped widget.

8. A device comprising:

a controller; and

a memory coupled to the controller having stored thereon instructions which, when executed by the controller, causes the controller to perform operations comprising;

controlling display, on a user interface of a mobile device, of a main menu including widgets that each have a depth within a three-dimensional menu space and that each control execution of a specific application on the mobile device;

selecting, from among the widgets in the main menu and in a manner that does not cause execution of a specific application on the mobile device, a widget;

at a time when the widget is selected, detecting a user'"'"'s gesture related to modifying a depth of the selected widget within the three-dimensional menu space of the main menu; and

based on detecting the user'"'"'s gesture related to modifying the depth of the selected widget, modifying, on the user interface of the mobile device, the depth of the selected widget within the three-dimensional menu space of the main menu such that, when the mobile device displays the main menu, the selected widget is displayed within the three-dimensional menu space of the main menu at the modified depth, which is different than depths of other widgets within the three-dimensional menu space of the main menu.

9. The device according to claim 8:

wherein detecting a user'"'"'s gesture related to modifying a depth of the selected widget comprises detecting a gesture of changing a distance between a user input unit and the mobile device in a space over the selected widget; and

wherein modifying, on the user interface of the mobile device, the depth of the selected widget comprises modifying, on the user interface of the mobile device, the depth of the selected widget based on detecting the gesture of changing the distance between the user input unit and the mobile device in the space over the selected widget.

10. The device according to claim 9:

wherein detecting a gesture of changing a distance between a user input unit and the mobile device in a space over the selected widget comprises detecting a gesture of approaching at least one user input unit to the mobile device or departing the at least one user input unit from the mobile device; and

wherein modifying, on the user interface of the mobile device, the depth of the selected widget comprises modifying, on the user interface of the mobile device, the depth of the selected widget based on detecting the gesture of approaching at least one user input unit to the mobile device or departing the at least one user input unit from the mobile device.

11. The device according to claim 8, wherein the operations further comprise:

detecting a user'"'"'s gesture of rotating at least one user input unit in a space over the selected widget; and

rotating the selected widget according to the user'"'"'s gesture of rotating.

12. The device according to claim 8:

wherein controlling display, on a user interface of a mobile device, of a main menu including widgets that each have a depth within a three-dimensional menu space comprises controlling display, on a user interface of a mobile device, of a main menu including widgets that have different sizes; and

wherein modifying, on the user interface of the mobile device, the depth of the selected widget comprises modifying, on the user interface of the mobile device, a size of the selected widget such that, when the mobile device displays the three-dimensional menu space of the main menu, the selected widget is displayed within the three-dimensional menu space of the main menu at the modified size, which is different than sizes of other widgets within the three-dimensional menu space of the main menu.

13. The device according to claim 8:

wherein controlling display, on a user interface of a mobile device, of a main menu including widgets that each have a depth within a three-dimensional menu space comprises controlling display, on a user interface of a mobile device, of a main menu including widgets that have different shadows; and

wherein modifying, on the user interface of the mobile device, the depth of the selected widget comprises modifying, on the user interface of the mobile device, a shadow of the selected widget such that, when the mobile device displays the three-dimensional menu space of the main menu, the selected widget is displayed within the three-dimensional menu space of the main menu with the modified shadow, which is different than shadows of other widgets within the three-dimensional menu space of the main menu.

14. The device according to claim 8:

wherein controlling display, on a user interface of a mobile device, of a main menu including widgets that each have a depth within a three-dimensional menu space comprises controlling display, on a user interface of a mobile device, of a main menu including widgets using at least three discrete levels of depth within the three-dimensional menu space, each of the at least three discrete levels of depth representing a different predetermined depth within the three-dimensional menu space; and

wherein modifying, on the user interface of the mobile device, the depth of the selected widget comprises changing, on the user interface of the mobile device, in which of the at least three discrete levels of depth the selected widget is classified such that, when the mobile device displays the three-dimensional menu space of the main menu, the selected widget is displayed within the main menu at the changed level of depth, which is different than levels of depth of other widgets within the three-dimensional menu space of the main menu.

16. The device according to claim 8, wherein modifying, on the user interface of the mobile device, the depth of the selected widget further comprises modifying an area on the user interface of the mobile device at which the selected widget is pressed to cause execution of a specific application on the mobile device associated with the selected widget.

19. The device according to claim 8, further comprising:

detecting a user'"'"'s gesture of rotating at least one user input unit in a space over the selected widget;

determining whether a direction of rotation of the at least one user input unit in the space over the selected widget is in;

a first plane perpendicular to the user interface,

a second plane parallel to the user interface, or

a third plane that is tilted with respect to the user interface such that the third plane is neither perpendicular nor parallel to the user interface;

based on determining that the direction of rotation of the at least one user input unit in the space over the selected widget is in the first plane perpendicular to the user interface, rotating the selected widget, according to the user'"'"'s gesture of rotating, about a first virtual axis parallel to the user interface;

based on determining that the direction of rotation of the at least one user input unit in the space over the selected widget is in the second plane parallel to the user interface, rotating the selected widget, according to the user'"'"'s gesture of rotating, about a second virtual axis perpendicular to the user interface; and

based on determining that the direction of rotation of the at last one user input unit in the space over the selected widget is in the third plane that is tilted with respect to the user interface, rotating the selected widget, according to the user'"'"'s gesture of rotating, about a third virtual axis sloped with respect to the user interface.

20. The device according to claim 8, wherein selecting the widget comprises selecting a widget that is defined as a rectangular parallelepiped widget, further comprising:

detecting a user'"'"'s gesture of rotating at least one user input unit in a space over the selected widget; and

in response to detecting the user'"'"'s gesture of rotating at least one user input unit in the space over the selected widget;

rotating the rectangular parallelepiped widget about an axis to change, from a first side of the rectangular parallelepiped widget, to a second side of the rectangular parallelepiped widget, and

modifying, on the user interface of the mobile device, a display of the selected widget such that, when the mobile device displays the three-dimensional menu space of the main menu, the selected widget is displayed within the three-dimensional menu space of the main menu at the second side of the rectangular parallelepiped widget, rather than the first side of the rectangular parallelepiped widget.