Gesture controls for multi-screen hierarchical applications

• US 10,613,706 B2
• Filed: 11/19/2015
• Issued: 04/07/2020
• Est. Priority Date: 10/01/2010
• Status: Active Grant

• Current Assignee: Multifold International Incorporated Pte. Ltd. (Transpacific IP Group Limited)
• Original Assignee: Z124 LLC (Flexis)
• Inventors: Gimpl, Martin, Sirpal, Sanjiv
• Primary Examiner(s): Kim, Sang H
• Application Number:
  US14/945,844
• Publication Number:
  US 20160179367A1
• Time in Patent Office: 1,601 Days
• CPC Class Codes: G06F 1/1616 with folding flat displays, e.g. laptop computers or notebooks having a clamshell configuration, with body parts pivoting to an open position around an axis parallel to the plane they define in closed position
• CPC Class Codes: G06F 1/1641 the display being formed by a plurality of foldable display components G06F1/1647 takes precedence
• CPC Class Codes: G06F 1/1647 including at least an additional display G06F1/1692 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0412 Digitisers structurally integrated in a display
• CPC Class Codes: G06F 3/0416 Control or interface arrangements specially adapted for digitisers
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06F 3/04817 using icons graphical or visual programming using iconic symbols G06F8/34
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/0483 Interaction with page-structured environments, e.g. book metaphor
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06F 3/04847 Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
• CPC Class Codes: G06F 3/0486 Drag-and-drop
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 3/04886 by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus
• CPC Class Codes: G06F 3/1423 controlling a plurality of local displays, e.g. CRT and flat panel display

Abstract

Control of a computing device using gesture inputs. The computing device may be a handheld computing device with a plurality of displays. The displays may be capable of displaying a graphical user interface (GUI). The plurality of displays may be modified in response to receipt of a gesture input such that a hierarchical application having related GUI screens are modified in response to the gesture input. The modification may include changing the hierarchical application from being displayed in a single screen mode to being displayed in a multi screen mode or vice versa.

Claims

1. A method for controlling a plurality of displays of a handheld computing device, comprising:

providing the handheld device including at least a first display and a second display, wherein each display includes a touch screen display and an off-display touch sensor, and wherein the off-display touch sensor is physically separate from the touch screen display;

executing a hierarchical application on the handheld computing device, wherein the hierarchical application comprises a plurality of hierarchically related screens, the plurality of hierarchically related screens comprising at least a root screen providing a top level view of a sequence of the hierarchical application and at least one node screen providing a view of a part of the sequence of the hierarchical application;

displaying a first screen of the plurality of hierarchically related screens of the hierarchical application on a first touch screen display associated with the first display;

displaying a second screen of the plurality of hierarchically related screens of the hierarchical application on a second touch screen display associated with the second display, wherein the second screen is a child screen of the first screen;

receiving a first gesture input at the handheld computing device in a second off-display touch sensor associated with the second display, wherein the first gesture input is a directional input to move the second screen to the first display, wherein the second off-display touch sensor is physically separate from the plurality of displays, and wherein the first gesture input has a first direction;

in response to the first gesture input displaying the first screen on the first touch screen display and moving the second screen to the first touch screen display, wherein the second screen is not displayed on the first and second touch screen displays and the hierarchical application is changed from a multi-screen mode by the first gesture input to a single screen mode and the second screen is displayed based on a hierarchical relationship between the first and second screens in the plurality of hierarchically related screens;

receiving a second gesture input at the handheld computing device in the second touch screen display associated with the second display, wherein the second gesture input is directional and in a different direction from a direction of the first gesture input; and

in response to receiving the second gesture input, moving the first screen to display on the second touch screen display and displaying the second screen on the first touch screen display, wherein the hierarchical application is changed from a single screen mode to a multi-screen mode.

2. The method as recited in claim 1, wherein the first gesture input is directional from the second touch screen display to the first touch screen display and the movement of the second screen corresponds to the first direction of the first gesture input, and wherein, during the movement of the second screen, displaying a visual indicia of the movement of the second screen to indicate the movement is occurring, wherein the visual indicia cannot receive an input from a user.

3. The method as recited in claim 2, wherein the first gesture input comprises a drag gesture, and wherein the second screen appears to slide toward the first touch screen display.

4. The method as recited in claim 1, wherein, in response to the first gesture input, the second screen is hidden behind the first screen and a second desktop is displayed on the second touch screen display.

5. The method as recited in claim 1, wherein the first screen comprises a node screen and the second screen comprises a leaf screen such that the hierarchical application is a sequence of hierarchically related screens.

6. The method as recited in claim 1, wherein, in response to the second gesture input, a first desktop is displayed on the first touch screen display.

7. The method as recited in claim 1, wherein in response to the first gesture input and moving the second screen to the first touch screen display, displaying a screen of another hierarchical application executing on the handheld computing device on the second touch screen display.

8. A handheld computing device, comprising:

a processor operable to execute a hierarchical application comprising;

a sequence of a plurality of hierarchically related screens, the plurality of hierarchically related screens comprising at least a root screen providing a top level view of a sequence of the hierarchical application and at least one node screen providing a view of a part of the sequence of the hierarchical application;

a first display comprising;

a first touch screen display in operative communication with the processor, the first touch screen display operable to display a first screen of said hierarchically related screens;

a first off-display touch sensor in operative communication with the processor, the first off-display touch sensor operable to receive a first gesture input, wherein the first gesture input is a directional input to move the first screen to a second display, wherein the first gesture input has a first direction, and wherein the first off-display touch sensor is physically separate from the first touch screen display;

the second display physically hinged to the first display comprising;

a second touch screen display in operative communication with the processor and operable to display a second screen of said hierarchically related screens;

a second off-display touch sensor in operative communication with the processor and operable to receive a second gesture input;

wherein the processor is operable to;

in response to the first gesture input, display the first screen on the second touch screen display and move the second screen to the first touch screen display, wherein the second screen is not displayed on the first and second touch screen displays and the hierarchical application is changed from a multi-screen mode by the first gesture input to a single screen mode and the second screen is displayed based on a hierarchical relationship between the first and second screens in the plurality of hierarchically related screens;

receive a second gesture input received in the second off-display touch sensor, wherein the second gesture is directional and has a direction different from a direction of the first gesture; and

in response to the second gesture input, move the first screen to display on the second touch screen display and display the second screen on the first touch screen display, wherein the hierarchical application is changed from a single screen mode to a multi-screen mode.

9. The device as recited in claim 8, wherein the handheld computing device comprises a smart phone.

10. The device as recited in claim 9, wherein the first display and second display are positioned with respect to each other between an open and closed position.

11. The device as recited in claim 10, wherein when in the open position, the first display and the second display are visible from the vantage point of a user, and wherein when in the closed position, the first and second displays are exposed on an exterior of the device and only one of the first display and the second display is visible from the vantage point of a user.

12. The device as recited in claim 8, wherein, in response to the first gesture input, the hierarchical application is changed from a multi-screen mode to a single screen mode and the second screen is hidden by the first screen, and wherein the first gesture input is not received by the second off-display touch sensor.

13. The device as recited in claim 8, wherein in response to the first gesture input, the first screen is moved over the second screen on the second touch screen display and the hierarchical application is changed from a multi-screen mode to a single screen mode.