Gesture controls for multi-screen user interface

• US 9,046,992 B2
• Filed: 11/17/2010
• Issued: 06/02/2015
• Est. Priority Date: 10/01/2010
• Status: Active Grant

• Current Assignee: Z124 LLC (Flexis)
• Original Assignee: Z124 LLC (Flexis)
• Inventors: Sirpal, Sanjiv
• Primary Examiner(s): Ng, Amy
• Assistant Examiner(s): Gutierrez, Andres E
• Application Number:
  US12/948,667
• Publication Number:
  US 20120084735A1
• Time in Patent Office: 1,658 Days
• Field of Search: 715/863, 345/157
• US Class Current: 1/1
• CPC Class Codes: G06F 1/1616 with folding flat displays, e.g. laptop computers or notebooks having a clamshell configuration, with body parts pivoting to an open position around an axis parallel to the plane they define in closed position
• CPC Class Codes: G06F 1/1641 the display being formed by a plurality of foldable display components G06F1/1647 takes precedence
• CPC Class Codes: G06F 1/1647 including at least an additional display G06F1/1692 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0412 Digitisers structurally integrated in a display
• CPC Class Codes: G06F 3/0416 Control or interface arrangements specially adapted for digitisers
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06F 3/04817 using icons graphical or visual programming using iconic symbols G06F8/34
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/0483 Interaction with page-structured environments, e.g. book metaphor
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06F 3/04847 Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
• CPC Class Codes: G06F 3/0486 Drag-and-drop
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 3/04886 by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus
• CPC Class Codes: G06F 3/1423 controlling a plurality of local displays, e.g. CRT and flat panel display

Abstract

Method and apparatus for controlling a computing device using gesture inputs. The computing device may be a handheld computing device with multiple displays. The displays may be capable of displaying a graphical user interface (GUI). The GUI may be a multi screen GUI or a single screen GUI such that receipt of gesture inputs may result in the movement of a GUI from one display to another display or may result in maximization of a multi screen GUI across multiple displays.

Claims

1. A method for controlling a plurality of displays of a handheld computing device, comprising:

providing a hand held device having two displays, wherein the displays are physically attached but movable with respect to each other, wherein when the hand held device is folded, the two displays face in substantially opposite directions and, when the hand held device is open, the displays face in a substantially same direction, wherein the first display includes a first gesture sensor and a first touch sensitive display within a boundary of a first bezel, and wherein the second display includes a second gesture sensor and a second touch sensitive display within a boundary of a second bezel, wherein the first and second gesture sensors accept touch input but do not display information on the first or second displays;

executing an application on said handheld computing device such that a screen of said application is displayed in the first touch sensitive display;

receiving a first gesture input at the first or second gesture sensor of said handheld computing device, wherein the first and second gesture sensors are physically separate from the first and second touch sensitive displays;

modifying the first and second touch sensitive displays in response to the first gesture input such that the screen is displayed in at least a second touch sensitive display of the plurality of touch sensitive displays, wherein if the first gesture input were received in the first or second touch sensitive display, a different response would occur, and wherein at least one of the following is true;

(a) the first gesture input is directional, the second touch sensitive display is located in a direction with respect to the first touch sensitive display corresponding to the direction of the first gesture, the screen occupies a single touch sensitive display and said modifying includes moving the screen from the first touch sensitive display to the second touch sensitive display, and the modifying includes revealing an underlying screen in the first touch sensitive display and obscuring another screen in the second touch sensitive display,

(b) the first gesture input is directional, the second touch sensitive display is located in a direction with respect to the first touch sensitive display corresponding to the direction of the first gesture, the application is expandable to occupy the first and second touch sensitive displays and the modifying includes expanding the application such that one or more screens corresponding to the application occupy both the first touch sensitive display and the second touch sensitive display, and the expanding includes scaling the application to occupy the first and second touch sensitive display, or

(c) the modifying includes constrained movement of the screen linearly between the first touch sensitive display and the second touch sensitive display.

2. The method according to claim 1, wherein (a) is true.

3. The method according to claim 1, wherein (b) is true.

4. The method according to claim 1, wherein (c) is true.

5. The method according to claim 4, wherein the handheld computing device is in operative communication with an external display, and the method further comprises relocating the screen from the first touch sensitive display to the external display.

6. The method according to claim 4, wherein once the screen has been relocated to the external display, the screen is unconstrainably moveable about the external display.

7. A method for controlling a plurality of displays of a handheld computing device, comprising:

executing an application on a handheld computing device such that one or more displays of said application is displayed in a first touch sensitive display and a second touch sensitive display of said handheld computing device, wherein the hand held computing device has two touch sensitive displays including the first and second touch sensitive displays, wherein the touch sensitive displays are physically attached but movable with respect to each other, wherein when the hand held device is folded, the two touch sensitive displays face in substantially opposite directions and, when the hand held device is open, the touch sensitive displays face in a substantially same direction, wherein the first touch sensitive display includes a first gesture sensor and a touch sensitive display, within a boundary of a first bezel, and wherein the second touch sensitive display includes a second gesture sensor and a touch sensitive display, within a boundary of a second bezel, wherein the first and second gesture sensors accept touch input but do not display information on the first or second touch sensitive displays;

receiving a gesture input at the first or second gesture sensor of said handheld computing device, wherein the first and second gesture sensors are physically separate from the first and second touch sensitive displays; and

modifying the first and second touch sensitive displays in response to the gesture input such that the application is displayed in the first touch sensitive display of the handheld computing device and not in the second touch sensitive display, wherein if the first gesture input were received in the first or second touch sensitive display, a different response would occur, wherein the gesture display is directional and the first touch sensitive display corresponds to the direction of the gesture, and wherein at least one of the following is true;

(a) said modifying includes hiding a node screen displayed in the second touch sensitive display prior to the modifying, or

(b) said modifying includes reducing the visible area of the application.

8. The method according to claim 7, wherein (a) is true.

9. The method according to claim 7, wherein (b) is true.

10. A non-transitory computer readable medium that causes a processor to execute a method for controlling a plurality of displays of a handheld computing device, comprising:

providing a hand held computing device with a unitary display having two portions, wherein the two portions are logically separate, wherein a first portion is a first touch sensitive display and is associated with a first gesture sensor, and wherein a second portion is a second touch sensitive display and is associated with a second gesture sensor;

executing an application on said handheld computing device such that a screen of said application is displayed in the first touch sensitive display;

receiving a first gesture input at the first gesture sensor or the second gesture sensor of said handheld computing device, wherein the first and second gesture sensors are physically separate from the first and second touch sensitive displays but located on the display within a boundary of a bezel defining the unitary display, and wherein the first and second gesture sensors accept touch input but do not display information on the first or second touch sensitive displays;

in response to the first gesture input, modifying the first and second touch sensitive displays such that the screen is displayed in at least the second touch sensitive display, wherein if the first gesture input were received in the first touch sensitive display, a different response would occur.

11. The non-transitory computer readable medium according to claim 10, wherein the first gesture input is directional, the second touch sensitive display is located in the direction of the first gesture, wherein the screen occupies a single display, wherein the modifying includes moving the screen from the first touch sensitive display to the second touch sensitive display, and wherein the modifying includes revealing a second screen in the first touch sensitive display and obscuring a third screen in the second touch sensitive display.

12. The non-transitory computer readable medium according to claim 10, wherein the first gesture input is directional, wherein the second touch sensitive display is located in a direction of the first gesture, wherein the application is expandable to occupy multiple displays, wherein the modifying includes expanding the application such that the screen occupies both the first touch sensitive display and the second touch sensitive display, wherein the expanding includes scaling the screen to occupy the first and second touch sensitive displays, and wherein the modifying includes obscuring a second screen in the second touch sensitive display.

13. The non-transitory computer readable medium according to claim 10, wherein the modifying includes constrained movement of the screen linearly between the first touch sensitive display and the second touch sensitive display.