Modifying Captured Stroke Information into an Actionable Form

• US 20170068445A1
• Filed: 09/03/2015
• Published: 03/09/2017
• Est. Priority Date: 09/03/2015
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Lee, Nicole, Markiewicz, Jan-Kristian, Williams, Sarah Graham
• Granted Patent: US 10,387,034 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 3/04886 by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus
• CPC Class Codes: G06F 40/171 by use of digital ink
• CPC Class Codes: G06V 30/1423 the instrument generating sequences of position coordinates corresponding to handwriting preprocessing or recognising digital ink G06V30/32
• CPC Class Codes: G06V 30/32 Digital ink
• CPC Class Codes: G06V 30/373 using a special pattern or subpattern alphabet
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands

Abstract

A computer-implemented technique is described herein that receives captured stroke information when a user enters a handwritten note using an input capture device. The technique then analyzes the captured stroke information to produce output analysis information. Based on the output analysis information, the technique modifies the captured stroke information into an actionable form that contains one or more actionable content items, while otherwise preserving the original form of the captured stroke information. The technique then presents the modified stroke information on a canvas display device. The user may subsequently activate one or more actionable content items in the modified stroke information to perform various supplemental tasks that pertain to the handwritten note. In one case, for example, the technique can recognize the presence of entity items and/or list items in the note and then reproduce them in an actionable form.

Claims

1. A computer-readable storage medium for storing computer-readable instructions, the computer-readable instructions, when executed by one or more processing devices, performing a method that comprises:

receiving captured stroke information from an input capture device, the input capture device producing the captured stroke information in response to capturing freehand ink strokes made by a user on an input surface associated with the input capture device;

transforming the captured stroke information into recognized stroke information;

identifying zero, one, or more of-interest content items in the recognized stroke information, depending on a number of of-interest content items in the recognized stroke information, to produce output analysis information;

generating a modified content item for each of-interest content item, produced by at least selectively transforming each of-interest content item into an actionable form, while otherwise leaving an original handwritten appearance of each of-interest content item intact;

displaying the captured stroke information, together with each modified content item, on a canvas display device;

receiving an activation, by the user, of a modified content item that is displayed on the canvas display device, to provide activation information associated with an activated modified content item; and

forwarding the activation information to at least one action-taking component.

2. The computer-readable storage medium of claim 1, wherein one of-interest content item is an entity item that is associated with an entity.

3. The computer-readable storage medium of claim 1, wherein one of-interest content item is a list item that is formed by a collection of list-member items.

4. A system, implemented by one or more computing devices, for transforming captured stroke information into an actionable form, comprising:

an ink analysis component configured to;

receive captured stroke information from an input capture device, the input capture device producing the captured stroke information upon capturing freehand ink strokes made by a user on an input surface associated with the input capture device;

transform the captured stroke information into recognized stroke information; and

identify zero, one, or more of-interest content items in the recognized stroke information, depending on a number of of-interest content items in the recognized stroke information, to produce output analysis information; and

a canvas presentation component configured to;

generate a modified content item for each of-interest content item that is identified, produced by at least selectively transforming each of-interest content item into an actionable form, while otherwise leaving an original handwritten appearance of each of-interest content item intact; and

display the captured stroke information, together with each modified content item, on a canvas display device,

the input capture device further being configured to receive an activation, by the user, of a modified content item that is presented on the canvas display device, to provide activation information associated with an activated modified content item, and

the system being configured to forward the activation information to at least one action-taking component.

5. The system of claim 4, wherein the ink analysis component is configured to provide the output analysis information by making reference to user information associated with the user.

6. The system of claim 4,

wherein one of-interest content item in the recognized stroke information is an entity item that is associated with an entity, and

wherein the ink analysis component is configured to identify the entity item using an entity identification component that performs entity analysis, to produce an identified entity item.

7. The system of claim 6, wherein the canvas presentation component is configured to generate a modified entity item for the identified entity item, wherein the modified entity item appears, when displayed on the canvas display device, as highlighted with respect to other parts of the captured stroke information, while otherwise leaving an original handwritten appearance of the modified entity item intact.

8. The system of claim 7, further including an action-taking component that is configured to provide further information regarding the entity in response to activation, by the user, of the modified entity item within the captured stroke information.

9. The system of claim 7, further including an action-taking component that is configured to perform an action that is directed to the entity in response to activation, by the user, of the modified entity item within the captured stroke information.

10. The system of claim 4,

wherein one of-interest content item is a list item that includes a collection of list-member items, and

wherein the ink analysis component is configured to identify the list item using a list identification component.

11. The system of claim 10,

wherein at least one list-member item in the list item expresses an intent to perform an action using two or more words, and

wherein the ink analysis component is further configured to perform intent analysis on said at least one list-member item, using an intent identification component, to determine the intent.

12. The system of claim 10, wherein the ink analysis component is further configured to perform linguistic analysis on a title and/or a body of the list item to generate recognized list meaning information.

13. The system of claim 12, further including an action-taking component that is configured to, in cooperation with the canvas presentation component, remind the user of the list item when a context is encountered that is associated with the recognized list meaning information of the list item.

14. The system of claim 10,

wherein the list item that is identified by the list identification component corresponds to an identified list item having a collection of identified list-member items, and

wherein the canvas presentation component is configured to generate a modified list item of the identified list item by aligning the collection of identified list-member items, associated with the identified list item, with respect to at least one reference edge, while otherwise leaving an original handwritten appearance of the modified list item intact.

15. The system of claim 10,

wherein the list item that is identified by the list identification component corresponds to an identified list item having a collection of identified list-member items,

wherein the canvas presentation component is configured to generate a modified list item of the identified list item by associating a command feature with each identified list-member item in the identified list item, to produce a modified list-member item, and

each command feature being configured to change a completion status of a corresponding modified list-member item when acted on by the user.

16. The system of claim 14,

wherein the canvas presentation component is further configured to provide a visual indication that an activated modified list-member item has been completed when the user activates a command feature that is associated with the activated modified list-member item.

17. The system of claim 16, wherein providing the visual indication entails changing a visual appearance of the activated modified list-member item and/or adding a mark in proximity to the activated modified list-member item and/or moving the activated modified list-member item to a new position in a collection of modified list-member items.

18. The system of claim 10,

wherein the list item that is identified by the list identification component corresponds to an identified list item having a collection of identified list-member items,

wherein the identified list item organizes the identified list-member items in the identified list item into at least two levels of a hierarchy, having different respective levels of indentation associated therewith,

wherein the canvas presentation component is configured to generate a modified list item of the identified list item by transforming the identified list-member items into modified list-member items that are capable of being collapsed and expanded, and

wherein the canvas presentation component is further configured to collapse or expand a representation of the hierarchy in response to a zoom command made by the user.

19. A method, implemented by one or more computing devices, for transforming captured stroke information into an actionable form, comprising:

receiving captured stroke information from an input capture device, the input capture device producing the captured stroke information in response to capturing freehand ink strokes made by a user on an input surface associated with the input capture device;

transforming the captured stroke information into recognized text information using at least handwriting-to-text analysis;

identifying zero, one, or more entity items in the recognized text information using linguistic analysis, depending on a number of entity items in the input recognized text information, to produce zero, one, or more identified entity items;

generating a modified entity item for each identified entity item, produced by transforming each entity item into an actionable form;

displaying the captured stroke information, together with each modified entity item, on a canvas display device;

receiving an activation, by the user, of an activated modified entity item, to provide activation information;

forwarding the activation information to at least one action-taking component; and

performing an action by the action-taking component based on the activation information.

20. The method of claim 19, wherein said generating of a modified entity item further comprises visually modifying the identified entity item, such that the modified entity item appears, when displayed on the canvas display device, as highlighted with respect to other parts of the captured stroke information.