Multi-input gesture control for a display screen

• US 8,769,444 B2
• Filed: 11/05/2010
• Issued: 07/01/2014
• Est. Priority Date: 11/05/2010
• Status: Active Grant

• Current Assignee: SAP SE
• Original Assignee: SAP AG (SAP SE)
• Inventors: Werner, Horst
• Primary Examiner(s): TITCOMB, WILLIAM D
• Application Number:
  US12/940,616
• Publication Number:
  US 20120113015A1
• Time in Patent Office: 1,334 Days
• Field of Search: 345/173, 715/702
• US Class Current: 715/863
• CPC Class Codes: G06F 2203/04808 Several contacts: gestures triggering a specific function, e.g. scrolling, zooming, right-click, when the user establishes several contacts with the surface simultaneously; e.g. using several fingers or a combination of fingers and pen
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures

Abstract

Improved multi-input gesture control for a display screen is enabled by using a first screen input to determine control operations that result from a second screen input that is followed by a gesture. A gesture that corresponds to a path traced on a display screen may define a new screen input or manipulate existing screen objects depending on some specific feature of the path or the entirety of the path.

Claims

1. A method of multi-input gesture control for a display screen, the method comprising:

receiving a first screen input that identifies a first location on the display screen;

providing an image marker at the first location in response to the first screen input, the image marker persisting on the display screen for a response-time period after the first screen input;

receiving a second screen input that identifies a second location on the display screen during the response-time period after the first screen input;

changing to an alternative input mode based on a position of the second location relative to the image marker, the alternative input mode being selected from a first input mode for drawing a new shape on the display screen if the second location lies inside the image marker and a second input mode for resizing an existing shape on the display screen if the second location lies outside the image marker; and

receiving a gesture as a screen input in the alternative input mode, the gesture including a motion along a path on the display screen starting from the second location, the path determining an outline of the new shape in the first input mode, and the path determining a scale factor for resizing the existing shape in the second input mode.

2. The method of claim 1, wherein

the image marker surrounds the first location as a single two-dimensional shape,

a persistence of the image marker is displayed by changing a color or intensity of the image marker over the response-time period, and

a disappearance of the of the image marker corresponds to an end of the response-time period.

3. The method of claim 1, wherein the path of the gesture in the first input mode defines an input to an application program that generates the new shape on the display screen based on one or more spatial attributes of the path.

4. The method of claim 1, wherein the gesture in the first input mode defines the new shape by completing a closed curve that corresponds to the outline of the new shape.

5. The method of claim 1, wherein the path starting from the second location defines the new shape in the first input mode by reaching a path-ending location that satisfies a completion condition for the new shape.

6. The method of claim 1, wherein the scale factor in the second input mode is based on one or more line lengths measured from the image marker to the path.

7. The method of claim 1, wherein the first and second screen inputs and the gesture each result from touching the display screen.

8. The method of claim 1, wherein the first and second screen inputs and the gesture each result from an application of a pointing device whose movements are echoed on the display screen by a pointer.

9. The method of claim 1, wherein the alternative input mode enables the motion along the path starting from the second location in an arbitrary direction.

10. The method of claim 1, wherein the scale factor in the second input mode is calculated as a ratio of two line lengths measured from the image marker to the path.

11. The method of claim 1, wherein the outline of the new shape is unconstrained by the first input mode.

12. The method of claim 1, wherein the existing shape is resized in the second input mode by changing a size of the new shape in proportion to one or more ratios of line lengths measured from the second image marker to the second path.

13. A non-transitory computer-readable medium embodying instructions which, when executed by a computer, cause the computer to perform operations comprising:

receiving a first screen input that identifies a first location on the display screen;

providing an image marker at the first location in response to the first screen input, the image marker persisting on the display screen for a response-time period after the first screen input;

receiving a second screen input that identifies a second location on the display screen during the response-time period after the first screen input;

changing to an alternative input mode based on a position of the second location relative to the image marker, the alternative input mode being selected from a first input mode for drawing a new shape on the display screen if the second location lies inside the image marker and a second input mode for resizing an existing shape on the display screen if the second location lies outside the image marker; and

receiving a gesture as a screen input in the alternative input mode, the gesture including a motion along a path on the display screen starting from the second location, the path determining an outline of the new shape in the first input mode, and the path determining a scale factor for resizing the existing shape in the second input mode.

14. The computer-readable medium of claim 13, wherein

the image marker surrounds the first location as a single two-dimensional shape,

a persistence of the image marker is displayed by changing a color or intensity of the image marker over the response-time period, and

a disappearance of the of the image marker corresponds to an end of the response-time period.

15. The computer-readable medium of claim 13, wherein the path of the gesture in the first input mode defines an input to an application program that generates the new shape on the display screen based on one or more spatial attributes of the path.

16. The computer-readable medium of claim 13, wherein the gesture in the first input mode defines the new shape by completing a closed curve that corresponds to the outline of the new shape.

17. An apparatus including a display screen, the apparatus comprising:

a first location-receiving module to receive a first screen input that identifies a first location on the display screen;

a marker module to provide an image marker at the first location in response to the first screen input, the image marker persisting on the display screen for a response-time period after the first screen input;

a second location-receiving module to receive a second screen input that identifies a second location on the display screen during the response-time period after the first screen input;

a mode-changing module to change to an alternative input mode for the display screen based on a position of the second location relative to the image marker, the alternative input mode being selected from a first input mode for drawing a new shape on the display screen if the second location lies inside the image marker and a second input mode for resizing an existing shape on the display screen if the second location lies outside the image marker;

a gesture-receiving module to receive a gesture as a screen input in the alternative input mode, the gesture including a motion along a path on the display screen starting from the second location, the path determining an outline of the new shape in the first input mode, and the path determining a scale factor for resizing the existing shape in the second input mode; and

a storage module to persistently store display screen data that identifies the first location, the image marker, the second location, and the gesture.

18. The apparatus of claim 17, wherein

the image marker surrounds the first location as a single two-dimensional shape,

a persistence of the image marker is displayed by changing a color or intensity of the image marker over the response-time period, and

a disappearance of the of the image marker corresponds to an end of the response-time period.

19. The apparatus of claim 17, wherein the path of the gesture in the first input mode defines an input to an application program that generates the new shape on the display screen based on one or more spatial attributes of the path.

20. The apparatus of claim 17, wherein the gesture in the first input mode defines the new shape by completing a closed curve that corresponds to the outline of the new shape.

21. The apparatus of claim 17, wherein the path starting from the second location defines the new shape in the first input mode by reaching a path-ending location that satisfies a completion condition for the new shape.

22. The apparatus of claim 17, wherein the scale factor in the second input mode is based on one or more line lengths measured from the image marker to the path.

23. The apparatus of claim 17, wherein the first and second screen inputs and the gesture each result from touching the display screen.

24. The apparatus of claim 17, wherein the first and second screen inputs and the gesture each result from an application of a computer pointing device whose movements are echoed on the display screen by a pointer.