MANAGING REAL-TIME HANDWRITING RECOGNITION

• US 20180173415A1
• Filed: 02/15/2018
• Published: 06/21/2018
• Est. Priority Date: 06/09/2013
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Xia, Mei-Qun, Dolfing, Jannes G., Dixon, Ryan S., Groethe, Karl M., Misra, Karan, Bellegarda, Jerome R., Meier, Ueli
• Granted Patent: US 10,579,257 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 3/018 Input/output arrangements for oriental characters
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 40/166 Editing, e.g. inserting or deleting
• CPC Class Codes: G06F 40/274 Converting codes to words; Guess-ahead of partial word inputs
• CPC Class Codes: G06V 30/347 Sampling; Contour coding; Stroke extraction
• CPC Class Codes: G06V 30/36 Matching; Classification
• CPC Class Codes: G06V 30/387 using human interaction, e.g. selection of the best displayed recognition candidate

Abstract

Methods, systems, and computer-readable media related to a technique for providing handwriting input functionality on a user device. A handwriting recognition module is trained to have a repertoire comprising multiple non-overlapping scripts and capable of recognizing tens of thousands of characters using a single handwriting recognition model. The handwriting input module provides real-time, stroke-order and stroke-direction independent handwriting recognition for multi-character handwriting input. In particular, real-time, stroke-order and stroke-direction independent handwriting recognition is provided for multi-character, or sentence level Chinese handwriting recognition. User interfaces for providing the handwriting input functionality are also disclosed.

Claims

1. A method, comprising:

at a device with a display and a touch-sensitive surface;

receiving, via the touch-sensitive surface, a handwriting input from a user, the handwriting input comprising a plurality of handwritten strokes provided in a handwriting input area of a handwriting input interface;

recognizing, based on a handwriting recognition model, a plurality of output characters from the handwriting input, the plurality of output characters including at least a first emoji character and at least a first character from a script of a natural human language; and

displaying, on the display, a recognition result comprising the first emoji character and the first character from the script of the natural human language in a candidate display area of the handwriting input interface.

2. The method of claim 1, further comprising:

based on the handwriting recognition model, recognizing at least a first semantic unit from the handwriting input, wherein the first semantic unit comprises a respective character, word or phrase that is capable of conveying a respective semantic meaning in a respective human language;

identifying a second emoji character associated with the first semantic unit recognized from the handwriting input; and

displaying, in the candidate display area of the handwriting input interface, a second recognition result comprising at least the second emoji character identified from the first semantic unit.

3. The method of claim 2, displaying the second recognition result further comprises:

concurrently displaying the second recognition result with a third recognition result comprising at least the first semantic unit.

4. The method of claim 1, further comprising:

receiving a user input selecting the first recognition result displayed in the candidate display area; and

in response to the user input, entering text of the selected first recognition result in a text input area of the handwriting input interface, wherein the text includes at least the first emoji character and the first character from the script of the natural human language.

5. The method of claim 1, wherein the handwriting recognition model has been trained on a multi-script training corpus comprising writing samples corresponding to characters of at least three non-overlapping scripts, and the three non-overlapping scripts includes a set of emoji characters, Chinese characters, and Latin script.

6. The method of claim 1, further comprising:

identifying a second semantic unit corresponding to the first emoji character recognized from the handwriting input;

displaying, in the candidate display area of the handwriting input interface, a fourth recognition result comprising at least the second semantic unit identified from the first emoji character.

7. The method of claim 6, wherein displaying the fourth recognition result further comprises:

concurrently displaying the fourth recognition result with the first recognition result in the candidate display area.

8. A non-transitory computer-readable medium storing one or more programs configured to be executed by one or more processors of an electronic device with a display and a touch-sensitive surface, the one or more programs including instructions for:

receiving, via the touch-sensitive surface, a handwriting input from a user, the handwriting input comprising a plurality of handwritten strokes provided in a handwriting input area of a handwriting input interface;

recognizing, based on a handwriting recognition model, a plurality of output characters from the handwriting input, the plurality of output characters including at least a first emoji character and at least a first character from a script of a natural human language; and

displaying, on the display, a recognition result comprising the first emoji character and the first character from the script of the natural human language in a candidate display area of the handwriting input interface.

9. The non-transitory computer-readable medium of claim 8, further comprising:

based on the handwriting recognition model, recognizing at least a first semantic unit from the handwriting input, wherein the first semantic unit comprises a respective character, word or phrase that is capable of conveying a respective semantic meaning in a respective human language;

identifying a second emoji character associated with the first semantic unit recognized from the handwriting input; and

displaying, in the candidate display area of the handwriting input interface, a second recognition result comprising at least the second emoji character identified from the first semantic unit.

10. The non-transitory computer-readable medium of claim 9, displaying the second recognition result further comprises:

concurrently displaying, on the display, the second recognition result with a third recognition result comprising at least the first semantic unit.

11. The non-transitory computer-readable medium of claim 8, further comprising:

receiving a user input selecting the first recognition result displayed in the candidate display area; and

in response to the user input, entering text of the selected first recognition result in a text input area of the handwriting input interface, wherein the text includes at least the first emoji character and the first character from the script of the natural human language.

12. The non-transitory computer-readable medium of claim 8, wherein the handwriting recognition model has been trained on a multi-script training corpus comprising writing samples corresponding to characters of at least three non-overlapping scripts, and the three non-overlapping scripts includes a set of emoji characters, Chinese characters, and Latin script.

13. The non-transitory computer-readable medium of claim 8, further comprising:

identifying a second semantic unit corresponding to the first emoji character recognized from the handwriting input;

displaying, in the candidate display area of the handwriting input interface, a fourth recognition result comprising at least the second semantic unit identified from the first emoji character.

14. The non-transitory computer-readable medium of claim 13, wherein displaying the fourth recognition result further comprises:

concurrently displaying the fourth recognition result with the first recognition result in the candidate display area.

15. An electronic device, comprising:

a display;

a touch-sensitive surface;

one or more processors; and

memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for;

receiving, via the touch-sensitive surface, a handwriting input from a user, the handwriting input comprising a plurality of handwritten strokes provided in a handwriting input area of a handwriting input interface;

recognizing, based on a handwriting recognition model, a plurality of output characters from the handwriting input, the plurality of output characters including at least a first emoji character and at least a first character from a script of a natural human language; and

displaying, on the display, a recognition result comprising the first emoji character and the first character from the script of the natural human language in a candidate display area of the handwriting input interface.

16. The electronic device of claim 15, further comprising:

based on the handwriting recognition model, recognizing at least a first semantic unit from the handwriting input, wherein the first semantic unit comprises a respective character, word or phrase that is capable of conveying a respective semantic meaning in a respective human language;

identifying a second emoji character associated with the first semantic unit recognized from the handwriting input; and

displaying, in the candidate display area of the handwriting input interface, a second recognition result comprising at least the second emoji character identified from the first semantic unit.

17. The electronic device of claim 16, displaying the second recognition result further comprises:

concurrently displaying the second recognition result with a third recognition result comprising at least the first semantic unit.

18. The electronic device of claim 15, further comprising:

receiving a user input selecting the first recognition result displayed in the candidate display area; and

in response to the user input, entering text of the selected first recognition result in a text input area of the handwriting input interface, wherein the text includes at least the first emoji character and the first character from the script of the natural human language.

19. The electronic device of claim 15, wherein the handwriting recognition model has been trained on a multi-script training corpus comprising writing samples corresponding to characters of at least three non-overlapping scripts, and the three non-overlapping scripts includes a set of emoji characters, Chinese characters, and Latin script.

20. The electronic device of claim 15, further comprising:

identifying a second semantic unit corresponding to the first emoji character recognized from the handwriting input;

displaying, in the candidate display area of the handwriting input interface, a fourth recognition result comprising at least the second semantic unit identified from the first emoji character.

21. The electronic device of claim 20, wherein displaying the fourth recognition result further comprises:

concurrently displaying the fourth recognition result with the first recognition result in the candidate display area.