Modifying captured stroke information into an actionable form

• US 10,387,034 B2
• Filed: 09/03/2015
• Issued: 08/20/2019
• Est. Priority Date: 09/03/2015
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Lee, Nicole, Markiewicz, Jan-Kristian, Williams, Sarah Graham
• Primary Examiner(s): Bashore, William L
• Assistant Examiner(s): Distefano, Gregory A
• Application Number:
  US14/844,006
• Publication Number:
  US 20170068445A1
• Time in Patent Office: 1,447 Days
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 3/04886 by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus
• CPC Class Codes: G06F 40/171 by use of digital ink
• CPC Class Codes: G06V 30/1423 the instrument generating sequences of position coordinates corresponding to handwriting preprocessing or recognising digital ink G06V30/32
• CPC Class Codes: G06V 30/32 Digital ink
• CPC Class Codes: G06V 30/373 using a special pattern or subpattern alphabet
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands

Abstract

A computer-implemented technique is described herein that receives captured stroke information when a user enters a handwritten note using an input capture device. The technique then analyzes the captured stroke information to produce output analysis information. Based on the output analysis information, the technique modifies the captured stroke information into an actionable form that contains one or more actionable content items, while otherwise preserving the original form of the captured stroke information. The technique then presents the modified stroke information on a canvas display device. The user may subsequently activate one or more actionable content items in the modified stroke information to perform various supplemental tasks that pertain to the handwritten note. In one case, for example, the technique can recognize the presence of entity items and/or list items in the note and then reproduce them in an actionable form.

Claims

1. A computer-readable storage medium storing computer-readable instructions which, when executed by one or more processing devices, cause the one or more processing devices to perform acts comprising:

receiving captured strokes from an input capture device, the input capture device producing the captured strokes in response to freehand ink strokes of handwritten words made on an input surface associated with the input capture device;

transforming the captured strokes into recognized text;

processing the recognized text to distinguish one or more entity words that identify a particular entity and one or more other words that do not identify the particular entity;

selectively transforming the particular entity into an actionable form;

visually modifying first captured strokes that convey the one or more entity words to be visually distinguishable from second captured strokes that convey the one or more other words while preserving a handwritten appearance of the one or more entity words;

displaying the visually modified first captured strokes and the second captured strokes concurrently on a canvas display device;

receiving an activation of the particular entity via the canvas display device, to provide activation information associated with the particular entity; and

forwarding the activation information to at least one action-taking component.

2. The computer-readable storage medium of claim 1, the acts further comprising:

associating a hit testing region on the canvas display device with the particular entity, the hit testing region being proximate to the visually modified first captured strokes that convey the one or more entity words while preserving the handwritten appearance.

3. The computer-readable storage medium of claim 1, wherein the visually modified first captured strokes that convey the one or more entity words are at least one of bold or underlined while preserving the handwritten appearance, and the second captured strokes that convey the one or more other words are not bold and are not underlined.

4. A system comprising:

a touch-sensitive screen;

a hardware processor; and

a computer-readable storage medium storing machine-readable instructions which, when executed by the hardware processor, cause the hardware processor to;

receive handwritten strokes entered via the touch-sensitive screen;

transform the handwritten strokes into recognized text;

automatically identify one or more entity words in the recognized text that identify a particular entity, the recognized text including one or more other words that do not identify the particular entity;

visually modify first handwritten strokes that convey the one or more entity words to be visually distinguishable from second handwritten strokes that convey the one or more other words, the one or more entity words retaining a handwritten appearance while being visually modified to indicate that the particular entity is actionable;

display the visually modified first handwritten strokes concurrently with the second handwritten strokes on the touch-sensitive screen;

receive an activation, via the touch-sensitive screen, of the particular entity to provide activation information associated with the particular entity; and

cause at least one action to be taken in response to activation of the particular entity.

5. The system of claim 4, wherein the machine-readable instructions, when executed by the hardware processor, cause the hardware processor to:

identify multiple candidate entities that share at least one of the one or more entity words;

access user information associated with a user that enters the handwritten strokes; and

select the particular entity from the multiple candidate entities based at least on the user information.

6. The system of claim 5, wherein the multiple candidate entities include multiple different people that share a first name, the particular entity comprises a particular person selected from the multiple different people that share the first name, and the at least one action comprises retrieving contact information for the particular person and displaying the contact information on the touch-sensitive screen.

7. The system of claim 4, wherein the machine-readable instructions, when executed by the hardware processor, cause the hardware processor to:

while maintaining the handwritten appearance of the one or more entity words and the one or more other words, bold and underline the one or more entity words to indicate that the particular entity is actionable while not bolding and not underlining the one or more other words.

8. The system of claim 7, wherein the machine-readable instructions, when executed by the hardware processor, cause the hardware processor to:

cause the at least one action by providing the activation information to an action-taking component.

9. The system of claim 4, wherein the machine-readable instructions, when executed by the hardware processor, cause the hardware processor to:

associate a hit testing region on the touch-sensitive screen with the particular entity, the hit testing region being proximate to the visually modified first handwritten strokes,

wherein the activation is received via a touch input directed to the hit testing region.

10. The system of claim 4, wherein the machine-readable instructions, when executed by the hardware processor, cause the hardware processor to:

evaluate the handwritten strokes to automatically detect that the handwritten strokes convey a list of items; and

responsive to automatically detecting that the handwritten strokes convey a list of items, visually modify the list of items while retaining a handwritten appearance of the list of items.

11. The system of claim 10, wherein the machine-readable instructions, when executed by the hardware processor, cause the hardware processor to:

responsive to automatically detecting that the handwritten strokes convey the list of items, align the items of the list with respect to a reference edge while retaining the handwritten appearance of the list of items.

12. The system of claim 10, wherein the machine-readable instructions, when executed by the hardware processor, cause the hardware processor to:

responsive to automatically detecting that the handwritten strokes convey the list of items, automatically equalize spacing between the items of the list while retaining the handwritten appearance of the list of items.

13. The system of claim 10, wherein the machine-readable instructions, when executed by the hardware processor, cause the hardware processor to:

responsive to automatically detecting that the handwritten strokes convey the list of items, automatically present checkboxes adjacent to each item on the list.

14. The system of claim 13, wherein the machine-readable instructions, when executed by the hardware processor, cause the hardware processor to:

upon selection of an individual checkbox for an individual item, add a mark to the individual checkbox.

15. The system of claim 10, wherein the machine-readable instructions, when executed by the hardware processor, cause the hardware processor to:

receive an input on the touch-sensitive screen selecting an individual item on the list and dragging the individual item to a new location in the list; and

in response to the input, update the list with the individual item in the new location while retaining the handwritten appearance of the list.

16. The system of claim 10, wherein the machine-readable instructions, when executed by the hardware processor, cause the hardware processor to:

automatically detect that the handwritten strokes include the list of items based at least on;

a spatial arrangement of the items of the list;

a presence of symbols preceding the items of the list; and

semantic characteristics of the items of the list.

17. The system of claim 10, wherein the machine-readable instructions, when executed by the hardware processor, cause the hardware processor to:

automatically detect that the handwritten strokes include a list of items based at least on a series of nouns prefaced by dashes and terminated by line breaks in the handwritten strokes.

18. The system of claim 10, wherein the machine-readable instructions, when executed by the hardware processor, cause the hardware processor to:

generate a modified list by transforming the items of the list into modified list-member items that are capable of being collapsed and expanded, based at least on a hierarchy associated with the list; and

collapse or expand a representation of the hierarchy in response to a zoom command.

19. A method implemented by one or more computing devices, the method comprising:

receiving handwritten strokes from an input capture device, the input capture device producing the handwritten strokes in response to handwriting on an input surface associated with the input capture device;

displaying the handwritten strokes on a display;

transforming the handwritten strokes into recognized text;

automatically identifying an entity in the recognized text using linguistic analysis, the entity being conveyed in first handwritten strokes by one or more entity words and the recognized text comprising one or more other words conveyed by second handwritten strokes;

visually modifying the display of the first handwritten strokes that convey the one or more entity words to be visually distinguishable from the second handwritten strokes that convey the one or more other words, the one or more entity words retaining a handwritten appearance while being visually modified to indicate that the entity is actionable;

receiving an activation of the entity; and

performing an action in response to activation of the entity.

20. The method of claim 19, further comprising:

detecting the activation of the entity when a touch input is detected proximate the one or more entity words.