Multi-stroke smart ink gesture language

• US 10,429,954 B2
• Filed: 05/31/2017
• Issued: 10/01/2019
• Est. Priority Date: 05/31/2017
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Markiewicz, Jan-Kristian, Disano, Robert Joseph, Klein, Christian
• Primary Examiner(s): Rosario, Nelson M
• Application Number:
  US15/609,980
• Publication Number:
  US 20180348896A1
• Time in Patent Office: 853 Days
• CPC Class Codes: G06F 3/03545 Pens or stylus
• CPC Class Codes: G06F 3/04812 Interaction techniques based on cursor appearance or behaviour, e.g. being affected by the presence of displayed objects
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04847 Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06V 30/347 Sampling; Contour coding; Stroke extraction
• CPC Class Codes: G06V 30/36 Matching; Classification

Abstract

This document describes techniques for, and systems that enable, a multi-stroke smart ink gesture language. The described techniques enable a digital ink user interface that allows a display, that is capable of receiving a digital ink stroke from a passive stylus or an active pen, to receive the digital ink stroke as either a content input or an action input. The digital ink stroke may be determined to be an action input based on proximity to, intersection with, or chronological immediacy to a prior-received digital ink stroke. When multiple digital ink strokes are determined to represent a multi-stroke gesture associated with an action input, the action input is provided to the application, which can then perform the action associated with the action input. The multi-stroke gesture allows users to input both content and actions using the passive stylus or active pen and natural digital inking strokes.

Claims

1. A system, comprising:

an operating system (OS);

an application; and

an action input module implemented at least partially in hardware and configured to;

provide, via the OS or the application, a digital ink user interface that is associated with the application and that can receive a digital ink stroke made through a stylus, the digital ink stroke;

received through a display that is capable of receiving input from the stylus; and

received over the digital ink user interface, the digital ink user interface configured to receive the digital ink stroke and present, in real time, a visual representation of the digital ink stroke, the application configured to interpret the digital ink stroke as content input;

determine, based on a detected proximity to a prior-received digital ink stroke and a chronological immediacy time duration associated with a receipt of the digital ink stroke relative to the prior-received digital ink stroke currently presented in or over the digital ink user interface, that the digital ink stroke and the prior-received digital ink stroke represent a multi-stroke gesture associated with an action input; and

provide the action input to the application effective to cause the application to receive and react to the action input.

2. The system of claim 1, wherein the visual representation of the digital ink stroke is a free-form line matching a free-form movement of an active point of the stylus.

3. The system of claim 1, wherein:

the digital ink user interface is presented by the OS, working in conjunction with the application, and the OS;

presents the visual representation over the digital ink user interface;

determines that the digital ink stroke and the prior-received digital ink stroke represent the multi-stroke gesture associated with the action input; and

provides the action input to the application.

4. The system of claim 1, wherein the detected proximity is based on detection of an intersection of the digital ink stroke with a visual representation of the prior-received digital ink stroke.

5. The system of claim 1, wherein determining that the digital ink stroke and the prior-received digital ink stroke represent the multi-stroke gesture is based on the chronological immediacy time duration indicating a threshold amount of time that has elapsed between the receipt of the digital ink stroke and a receipt of the prior-received digital ink stroke.

6. The system of claim 1, wherein determining that the digital ink stroke and the prior-received digital ink stroke represent the multi-stroke gesture is further based on one or more of:

a second proximity to, a second intersection with, and a second chronological immediacy time duration of a second prior-received digital ink stroke, and wherein the second prior-received digital ink stroke is received prior to the prior-received digital ink stroke.

7. The system of claim 1, wherein determining that the digital ink stroke and the prior-received digital ink stroke represent the multi-stroke gesture further comprises comparing an initial gesture determination with a set of action inputs associated with the application.

8. The system of claim 1, wherein a determination that the digital ink stroke and the prior-received digital ink stroke represent the multi-stroke gesture further comprises detecting that the digital ink stroke is received in a transient digital ink.

9. The system of claim 8, wherein the action input module is further configured to dismiss the transient digital ink, from the digital ink user interface, based on a recognition of the action input.

10. The system of claim 1, wherein the action input module is further configured to detect a mode of execution for the application, and wherein the action input is selected based a determination that the multi-stroke gesture is received in the detected mode of execution for the application.

11. A method, comprising:

receiving a digital ink stroke made through a stylus, the digital ink stroke;

received through a display that is capable of receiving input from the stylus; and

received over a digital ink user interface associated with an application, the digital ink user interface configured to receive the digital ink stroke and present, in real time, a visual representation of the digital ink stroke, the application configured to interpret the digital ink stroke as content input;

determining, based on a detected proximity to a prior-received digital ink stroke and a chronological immediacy time duration associated with a receipt of the digital ink stroke relative to the prior-received digital ink stroke currently presented in or over the digital ink user interface, that the digital ink stroke and the prior-received digital ink stroke represent a multi-stroke gesture associated with an action input; and

providing the action input to the application effective to cause the application to receive and react to the action input.

12. The method of claim 11, wherein the digital ink user interface is provided by an operating system working in conjunction with the application, the operating system performing the method and presenting the visual representation over the digital ink user interface.

13. The method of claim 11, wherein the action input is determined to be:

to define a selection, then define an action to be performed on the selection;

to define a property, then adjust the property in either one or two dimensions;

or to define a command, then execute the command with an analog qualifier.

14. The method of claim 11, wherein determining that the digital ink stroke and the prior-received digital ink stroke represent the multi-stroke gesture further comprises comparing an initial gesture determination with a set of action inputs associated with the application.

15. The method of claim 11, wherein a determination that the digital ink stroke and the prior-received digital ink stroke represent the multi-stroke gesture further comprises detecting that the digital ink stroke is received in a transient digital ink, and wherein the method further comprising:

dismissing the transient digital ink, from the digital ink user interface, based on a recognition of the action input.

16. The method of claim 11, further comprising:

detecting a mode of execution for the application, and wherein the action input is selected based a determination that the multi-stroke gesture is received in the detected mode of execution for the application.

17. A method, comprising:

receiving a first digital ink stroke made through a stylus, the first digital ink stroke;

received through a display that is capable of receiving input from the stylus; and

received over a digital ink user interface associated with an application;

presenting, over the digital ink user interface and in real time, a first visual representation of the first digital ink stroke;

receiving, while the first visual representation of the first digital ink stroke is visually maintained, a second digital ink stroke made through the stylus and received over the digital ink user interface;

presenting, over the digital ink user interface and in real time, a second visual representation of the second digital ink stroke while visually maintaining the first visual representation;

determining, based on a detected proximity of the first digital ink stroke and the second digital ink stroke and a chronological immediacy time duration associated with a receipt of the second digital ink stroke relative to the first digital ink stroke, that the first digital ink stroke and the second digital ink stroke represents a multi-stroke gesture associated with an action input; and

providing the action input to the application effective to cause the application to receive and react to the action input.

18. The method of claim 17, wherein the application includes a command user interface, and wherein the action input is selected based on a location of the first digital ink stroke;

and the method further comprising;

determining that the first digital ink stroke, based on the location, selects a visual control or image presented within the command user interface.

19. The method of claim 17, wherein a determination that the first digital ink stroke and the second digital ink stroke represent the multi-stroke gesture further comprises detecting that the first digital ink stroke is received in a transient digital ink, and wherein the method further comprising:

dismissing the transient digital ink, from the digital ink user interface, based on a recognition of the action input.

20. The method of claim 17, further comprising:

detecting a mode of execution for the application, and wherein the action input is selected based a determination that the multi-stroke gesture is received in the detected mode of execution for the application.