Method and system for performing drag and drop operations on a device via user gestures

• US 8,527,892 B2
• Filed: 09/28/2011
• Issued: 09/03/2013
• Est. Priority Date: 10/01/2010
• Status: Active Grant

• Current Assignee: Multifold International Incorporated Pte. Ltd. (Transpacific IP Group Limited)
• Original Assignee: Z124 LLC (Flexis)
• Inventors: Sirpal, Sanjiv, Reeves, Paul, de Paz, Alexander, Ficklin, Jared, Burton, Denise, Wygonik, Gregg
• Primary Examiner(s): PHANTANA ANGKOOL, DAVID
• Application Number:
  US13/247,949
• Publication Number:
  US 20120084694A1
• Time in Patent Office: 706 Days
• Field of Search: 715/769, 715/702, 715/709
• US Class Current: 715/769
• CPC Class Codes: G06F 1/16 Constructional details or arrangements
• CPC Class Codes: G06F 1/1616 with folding flat displays, e.g. laptop computers or notebooks having a clamshell configuration, with body parts pivoting to an open position around an axis parallel to the plane they define in closed position
• CPC Class Codes: G06F 1/1632 External expansion units, e.g. docking stations
• CPC Class Codes: G06F 1/1641 the display being formed by a plurality of foldable display components G06F1/1647 takes precedence
• CPC Class Codes: G06F 1/1647 including at least an additional display G06F1/1692 takes precedence
• CPC Class Codes: G06F 1/1677 for detecting open or closed state or particular intermediate positions assumed by movable parts of the enclosure, e.g. detection of display lid position with respect to main body in a laptop, detection of opening of the cover of battery compartment
• CPC Class Codes: G06F 1/1694 the I/O peripheral being a single or a set of motion sensors for pointer control or gesture input obtained by sensing movements of the portable computer
• CPC Class Codes: G06F 2200/1614 Image rotation following screen orientation, e.g. switching from landscape to portrait mode
• CPC Class Codes: G06F 2200/1637 Sensing arrangement for detection of housing movement or orientation, e.g. for controlling scrolling or cursor movement on the display of an handheld computer
• CPC Class Codes: G06F 2203/04803 Split screen, i.e. subdividing the display area or the window area into separate subareas
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/041 Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means
• CPC Class Codes: G06F 3/0412 Digitisers structurally integrated in a display
• CPC Class Codes: G06F 3/0416 Control or interface arrangements specially adapted for digitisers
• CPC Class Codes: G06F 3/044 by capacitive means
• CPC Class Codes: G06F 3/048 Interaction techniques based on graphical user interfaces [GUI]
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06F 3/04812 Interaction techniques based on cursor appearance or behaviour, e.g. being affected by the presence of displayed objects
• CPC Class Codes: G06F 3/04817 using icons graphical or visual programming using iconic symbols G06F8/34
• CPC Class Codes: G06F 3/0482 : Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/0483 : Interaction with page-structured environments, e.g. book metaphor
• CPC Class Codes: G06F 3/0484 : for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
• CPC Class Codes: G06F 3/04842 : Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04845 : for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06F 3/04847 : Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
• CPC Class Codes: G06F 3/0486 : Drag-and-drop
• CPC Class Codes: G06F 3/0488 : using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06F 3/04883 : for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 3/04886 : by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus
• CPC Class Codes: G06F 3/1423 : controlling a plurality of local displays, e.g. CRT and flat panel display
• CPC Class Codes: G06F 3/1446 : display composed of modules, e.g. video walls
• CPC Class Codes: G06T 3/02 : Affine transformations for image registration G06T3/147; for image mosaicing G06T3/4038
• CPC Class Codes: G06T 3/40 : Scaling of whole images or parts thereof, e.g. expanding or contracting
• CPC Class Codes: G09G 2356/00 : Detection of the display position w.r.t. other display screens
• CPC Class Codes: G09G 5/14 : Display of multiple viewports
• CPC Class Codes: H04L 12/1827 : Network arrangements for conference optimisation or adaptation

Abstract

A multi-screen user device and methods for performing a drag and drop operation using finger gestures are disclosed. A first finger gesture is used to select a display area from which data is to be copied. Subsequently, a drag finger gesture is used to identify where the data that is to be pasted. The drag may extend across a non-display boundary between a first and second display screen of the multi-screen device.

Claims

1. A method for performing a drag and drop operation using user finger gesture input to a device having a plurality of touch sensitive displays, comprising:

receiving an input of a first finger gesture to a first of the touch sensitive displays;

determining a source area of the first touch sensitive display where the first finger gesture input was detected;

identifying data within the source area to be copied;

receiving an input of a finger drag gesture for identifying a target area of a second of the touch sensitive displays into which the data from the source data is to be copied, wherein the finger drag gesture extends across a boundary between the first touch sensitive display and the second touch sensitive display, wherein the first and second touch sensitive displays are foldable relative to one another along the boundary;

wherein the target area corresponds to a location on the second touch sensitive display where the drag gesture is last detected before it ceases to be detected;

changing a display of the target area for identifying the target area to a user as able to receive the data from the source area; and

copying the data into the target area.

2. The method of claim 1, wherein at least one of the first and second touch sensitive displays includes a capacitive touch sensitive area for receiving the finger gesture input.

3. The method of claim 1, further including a step of outputting on the first touch sensitive display a display change for identifying the source area to the user.

4. The method of claim 3, further including a step of selecting the target area identified by the location on the display where the drag gesture is last detected before it ceases to be detected.

5. The method of claim 4, wherein the step of selecting includes selecting a display area that accepts a clipboard input, wherein the data from the source area is copied to the clipboard, and copied from the clipboard to the target area.

6. The method of claim 1, further including selecting the source area by identifying a location on the first touch sensitive display for the first finger gesture input, and changing a display presentation of the source area.

7. The method of claim 1, wherein the first finger gesture includes pressing and holding two fingers against the display for a predetermined time for identifying a display location of the source area.

8. The method of claim 7, wherein the finger drag gesture includes dragging the finger gesture substantially continuously from the display location to the location on the second touch sensitive display on the touch sensitive display device.

9. The method of claim 1, further including identifying to the user one or more display areas as the user inputs a portion of the finger drag gesture over each of the one or more display areas, wherein each of the one or more display areas are determined to be acceptable for pasting the data therein.

10. A non-transitory computer readable medium for performing a drag and drop operation using user finger gesture inputs to first and second display screens of a device, wherein the device includes a folding mechanism attached to each of the first and second display screens for providing the first and seconds display screens in a folded configuration wherein the first and second display screens face in opposite directions, and in an unfolded configuration wherein the first and second display screens face in a substantially same direction, comprising:

machine instructions for performing the following steps;

determining that the first and second display screens are in the unfolded configuration;

receiving an input of a first finger gesture to the first display screen;

determining a source area of the first display screen where the first finger gesture was detected;

identifying data within the source area to be copied;

receiving an input of a finger drag gesture for identifying a target area of the second display screen into which the data from the source data is to be copied, wherein the finger drag gesture extends across a boundary between the first display screen and the second display screen, wherein the first and second display screens are foldable relative to one another along the boundary;

wherein the target area corresponds to a location on the second display screen where the drag gesture is last detected before it ceases to be detected;

changing a display of the target area for identifying the target area to a user as able to receive the data from the source area; and

copying the data into the target area.

11. The non-transitory computer readable medium of claim 10, wherein the source area and the target area each correspond to a displayed portion of a different application window;

wherein each of the different application windows corresponds to a different software application installed on the device.

12. The non-transitory computer readable medium of claim 10, wherein each of the software applications is operable on the device independently of each of the other of the software applications.

13. The non-transitory computer readable medium of claim 10, wherein the step of copying includes copying the data from the source area to a clipboard, and then automatically inputting corresponding clipboard data to the target area.

14. The non-transitory computer readable medium of claim 10, further including highlighting the source area prior to a performance of the finger drag gesture.

15. A hand-held device for performing a drag and drop operation using user gestures as inputs, comprising:

first and second display screens of the device, wherein the device includes a folding mechanism attached to each of the first and second display screens for providing the first and second display screens in a folded configuration wherein the first and second display screens face in opposite directions, and in an unfolded configuration wherein the first and second display screens face in a substantially same direction;

a display manager for determining whether the first and second display screens are in an unfolded configuration or a folded configuration;

a gesture interpreter for interpreting an input of a first gesture to the first display screen for identifying a first area, and subsequently interpreting an input of a second gesture for selecting a second area, wherein the first area includes the second area;

a window manager for receiving;

(i) first data from the gesture interpreter, the first data includes location data indicative of a location on the first display screen of the first gesture input, and (ii) second data from the gesture interpreter, the second data for identifying the second area;

wherein the window manager uses the location data for determining the first area, and changing a display presentation of the first area for identifying the first area to a user;

wherein, after the change in display presentation of the first area, the gesture interpreter receives the second gesture input, and the window manager uses the second data to change a display presentation of the second area for determining the second area to the user as a source area and identifying data within the source area for the drag and drop operation;

wherein, when the window manager receives input from the gesture interpreter of a drag gesture being input for performing a drag for the source area, a corresponding image to be dragged is generated;

wherein the window manager uses a display screen location from the drag gesture for identifying a target area of the second display screen into which data from the source area is to be copied, wherein the target area corresponds to a location of the second display screen where the drag gesture is last detected by the gesture interpreter before the drag gesture ceases to be detected by the gesture interpreter;

wherein the data from the source area is copied into the target area.

16. The hand-device of claim 15, wherein the source area is included in a display of a first application, and the target area is included in a display of a different second application.

17. The hand-device of claim 15, wherein the first gesture includes pressing and holding a finger against the first display screen for a predetermined time for identifying a display location for the first area.

18. The hand-device of claim 17, wherein the drag gesture includes dragging the first gesture on each of the first and second display screens substantially continuously across a boundary therebetween.

19. The hand-device of claim 15 further including an application support service for inputting the data from the source area into a clipboard, and then automatically inputting corresponding clipboard data to the target area.

20. The hand-device of claim 15, wherein the window manager highlights the source area prior to a performance of the drag gesture.