Interacting with an assistant component based on captured stroke information

• US 10,210,383 B2
• Filed: 09/03/2015
• Issued: 02/19/2019
• Est. Priority Date: 09/03/2015
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Markiewicz, Jan-Kristian, Auer, Nathaniel E. B., Clark, Lee Dicks, Take, Katsumi, Lee, Nicole
• Primary Examiner(s): Tapp, Amelia L
• Application Number:
  US14/844,009
• Publication Number:
  US 20170068854A1
• Time in Patent Office: 1,265 Days
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 40/166 Editing, e.g. inserting or deleting
• CPC Class Codes: G06F 40/171 by use of digital ink
• CPC Class Codes: G06F 40/279 Recognition of textual entities
• CPC Class Codes: G06V 30/1423 the instrument generating sequences of position coordinates corresponding to handwriting preprocessing or recognising digital ink G06V30/32
• CPC Class Codes: G06V 30/32 Digital ink
• CPC Class Codes: G06V 30/347 Sampling; Contour coding; Stroke extraction
• CPC Class Codes: G06V 30/36 Matching; Classification
• CPC Class Codes: G06V 30/373 using a special pattern or subpattern alphabet
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands

Abstract

A computer-implemented technique is described herein that receives captured stroke information when a user enters handwritten notes using an input capture device. The technique then automatically performs analysis on the captured stroke information to produce output analysis information. Based on the output analysis information, the technique uses an assistant component to identify a response to the captured stroke information and/or to identify an action to be performed. The technique then presents the response, together with the original captured stroke information. In addition, or alternatively, the technique performs the action. In one case, the response is a text-based response; that text-based response may be presented in a freeform handwriting style to give the user the impression that a virtual assistant is responding to the user'"'"'s own note. In another case, the response engages the user in an interactive exercise of any type.

Claims

1. A system comprising:

one or more processing devices; and

a computer-readable storage medium storing computer-readable instructions which, when executed by the one or more processing devices, cause the one or more processing devices to;

receive captured stroke information from an input capture device, the input capture device producing the captured stroke information in response to capturing freehand ink strokes made by a user on an input surface associated with the input capture device;

transform the captured stroke information into recognized stroke information;

automatically perform analysis on the recognized stroke information to identify an ambiguous named entity expressed by a portion of the captured stroke information, the ambiguous named entity being identified without guidance from the user specifying a subset of the captured stroke information on which to perform the analysis;

display the captured stroke information with the portion of the captured stroke information that expresses the ambiguous named entity modified to distinguish the portion from another portion of the captured stroke information that does not express the ambiguous named entity;

request clarification from the user, in a freeform handwriting style, to identify a specific named entity that the user intends to identify; and

based at least on the specific named entity, perform an action for the user or provide a response to the user.

2. The system of claim 1, wherein the computer-readable instructions, when executed by the one or more processing devices, cause the one or more processing devices to:

determine that the ambiguous named entity referenced in the portion of the captured stroke information matches at least two candidate entities;

display the at least two candidate entities to the user; and

receive a user selection of the specific named entity from the displayed at least two candidate entities.

3. The system of claim 2, wherein the ambiguous named entity specifies a first name and at least two candidate entities comprise two different people that share the first name.

4. The system of claim 1, wherein the computer-readable instructions, when executed by the one or more processing devices, cause the one or more processing devices to:

perform analysis of user information associated with the user to identify an intent of the user with respect to the specific named entity.

5. The system of claim 4, wherein the computer-readable instructions, when executed by the one or more processing devices, cause the one or more processing devices to:

select a dialog from among a set of dialogs to conduct with the user based at least on the identified intent of the user; and

perform the dialog, the dialog including the request for clarification.

6. The system of claim 5, the dialog further including a request for a time to meet with the specific named entity.

7. The system of claim 6, wherein the computer-readable instructions, when executed by the one or more processing devices, cause the one or more processing devices to

based at least on the ambiguous named entity, identify at least two candidate named entities that the user potentially intends to identify;

display the at least two candidate named entities with separate hit-testing regions associated therewith; and

identify the specific named entity responsive to a user selection of a specific hit-testing region associated with the specific named entity.

8. The system of claim 7, wherein the computer-readable instructions, when executed by the one or more processing devices, cause the one or more processing devices to:

schedule a meeting with the specific named entity at a time specified by the user in response to the request for the time to meet with the specific named entity.

9. A system, implemented by one or more computing devices, for engaging a user in a dialog based at least on captured stroke information, the system comprising:

an ink analysis component configured to;

receive captured stroke information from an input capture device, the input capture device producing the captured stroke information in response to capturing freehand ink strokes made by a user on an input surface associated with the input capture device;

transform the captured stroke information into recognized stroke information; and

automatically perform analysis on the recognized stroke information to generate output analysis information;

an assistant component configured to;

identify, without being prompted to do so by the user, an appropriate dialog with which to engage the user, based at least on the output analysis information, the dialog encompassing one or more responses; and

formulate a response to the captured stroke information based at least on the dialog and the output analysis information; and

a canvas presentation component configured to display the captured stroke information on a canvas display device, and to also present the response in a manner that simulates writing of the response over a span of time by a human.

10. The system of claim 9, wherein the ink analysis component is configured to generate the output analysis information by determining whether the recognized stroke information specifies one or more entity items.

11. The system of claim 9, wherein the ink analysis component is configured to generate the output analysis information by determining at least one intent associated with the recognized stroke information.

12. The system of claim 9, wherein the ink analysis component is configured to provide the output analysis information by making reference to user information associated with the user.

13. The system of claim 9, wherein the dialog that is selected by the assistant component collects information for a purpose of setting up a reminder.

14. The system of claim 13, wherein the response asks the user to specify an unresolved item of reminder information.

15. The system of claim 9, wherein the dialog that is selected by the assistant component has a purpose of conducting an interactive entertainment exercise between the assistant component and the user.

16. The system of claim 15, wherein the interactive entertainment exercise is a game.

17. The system of claim 9, wherein the system is configured to receive new captured input information when the user generates, in reply to the response, new freehand ink strokes.

18. The system of claim 9, wherein the canvas presentation component is configured to express the response in a freeform handwriting style.

19. A method implemented by one or more computing devices, the method comprising:

receiving captured stroke information from an input capture device, the input capture device producing the captured stroke information in response to freehand ink strokes made by a user on an input surface associated with the input capture device;

transforming the captured stroke information into recognized stroke information;

automatically performing analysis on the recognized stroke information to generate output analysis information identifying a named entity expressed by a portion of the captured stroke information, the named entity being identified without guidance from the user identifying the portion of the captured stroke information on which to perform the analysis;

identifying, based at least on the named entity, a response to the captured stroke information and/or an action to be performed;

displaying the captured stroke information on a canvas display device;

modifying the display of the portion of the captured stroke information that identifies the named entity to distinguish the portion from another unmodified portion of the captured stroke information that does not identify the named entity; and

presenting the response to the user and/or performing the action.

20. The method of claim 19, wherein the modifying comprises underlining the portion of the captured stroke information that identifies the named entity.