Multi-script handwriting recognition using a universal recognizer

• US 9,934,430 B2
• Filed: 09/29/2016
• Issued: 04/03/2018
• Est. Priority Date: 06/09/2013
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Dolfing, Jannes G. A., Groethe, Karl M., Dixon, Ryan S., Bellegarda, Jerome R.
• Primary Examiner(s): Vu, Kim
• Assistant Examiner(s): BLOOM, NATHAN J
• Application Number:
  US15/280,749
• Publication Number:
  US 20170017835A1
• Time in Patent Office: 551 Days
• Field of Search: None
• CPC Class Codes: G06F 18/217 Validation; Performance evaluation; Active pattern learning techniques
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 40/129 Handling non-Latin characters, e.g. kana-to-kanji conversion
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 3/088 Non-supervised learning, e.g. competitive learning
• CPC Class Codes: G06V 10/95 structured as a network, e.g. client-server architectures
• CPC Class Codes: G06V 30/2445 Alphabet recognition, e.g. Latin, Kanji or Katakana
• CPC Class Codes: G06V 30/246 using linguistic properties, e.g. specific for English or German language
• CPC Class Codes: G06V 30/293 of characters other than Kanji, Hiragana or Katakana
• CPC Class Codes: G06V 30/36 Matching; Classification

Abstract

Methods, systems, and computer-readable media related to a technique for providing handwriting input functionality on a user device. A handwriting recognition module is trained to have a repertoire comprising multiple non-overlapping scripts and capable of recognizing tens of thousands of characters using a single handwriting recognition model. The handwriting input module provides real-time, stroke-order and stroke-direction independent handwriting recognition. User interfaces for providing the handwriting input functionality are also disclosed.

Claims

1. A non-transitory computer-readable media having instructions stored thereon, the instructions, when executed by one or more processors, cause the processors to perform operations comprising:

training a multi-script handwriting recognition model based on spatially-derived features of a multi-script training corpus, the multi-script training corpus including respective handwriting samples corresponding to characters of one or more scripts;

receiving a handwriting input from a user which includes characters from a plurality of scripts; and

in response to receiving the handwriting input, providing real-time handwriting recognition output for the handwriting input using the multi-script handwriting recognition model that has been trained on the spatially-derived features of the multi-script training corpus, wherein the real-time handwriting recognition output includes characters from a plurality of scripts.

2. The media of claim 1, wherein the spatially-derived features of the multi-script training corpus are stroke-order independent and stroke-direction independent.

3. The media of claim 1, wherein the training of the multi-script handwriting recognition model is independent of temporal information associated with respective strokes in the handwriting samples.

4. The media of claim 1, wherein the one or more scripts include one or more Chinese characters, emoji characters, and Latin script.

5. The media of claim 1, wherein the one or more scripts include one or more Chinese characters, Arabic script, and Latin script.

6. The media of claim 1, wherein the one or more scripts include one or more non-overlapping scripts defined by the Unicode standard.

7. The media of claim 1, wherein training the multi-script handwriting recognition model further comprises:

providing the handwriting samples of the multi-script training corpus to a single convolutional neural network having a single input plane and a single output plane; and

determining, using the convolutional neural network, the spatially-derived features of the handwriting samples and respective weights for the spatially-derived features for differentiating characters of the one or more scripts represented in the multi-script training corpus.

8. The media of claim 1, wherein the multi-script handwriting recognition model has at least thirty thousand output classes, representing at least thirty thousand characters spanning the one or more scripts.

9. The media of claim 1, wherein providing real-time handwriting recognition for a user'"'"'s handwriting input further comprises:

providing the multi-script handwriting recognition model to a user device, wherein the user device receives a plurality of handwritten strokes from the user, and locally performs handwriting recognition on one or more recognition units identified from the plurality of handwritten strokes based on the received multi-script handwriting recognition model.

10. The media of claim 1, wherein providing real-time handwriting recognition for a user'"'"'s handwriting input further comprises:

continuously revising one or more recognition results for the user'"'"'s handwriting input in response to continued additions to or revisions of the handwriting input by the user; and

in response to each revision of the one or more recognition results, displaying the respective revised one or more recognition results to the user in a candidate display area of the handwriting input user interface.

11. The media of claim 1, comprising instructions, which when executed by the one or more processors, cause the processors to perform operations comprising:

providing the multi-script handwriting recognition model to a plurality of devices having no existing overlap in input languages, wherein the multi-script handwriting recognition model is used on each of the plurality of devices for handwriting recognition of a different input language associated with said each user device.

12. A method of providing multi-script handwriting recognition, comprising:

at a device having one or more processors and memory;

training a multi-script handwriting recognition model based on spatially-derived features of a multi-script training corpus, the multi-script training corpus including respective handwriting samples corresponding to characters of one or more scripts;

receiving a handwriting input from a user which includes characters from a plurality of scripts; and

in response to receiving the handwriting input, providing real-time handwriting recognition output for the handwriting input using the multi-script handwriting recognition model that has been trained on the spatially-derived features of the multi-script training corpus, wherein the real-time handwriting recognition output includes characters from a plurality of scripts.

13. A system, comprising:

one or more processors; and

memory having instructions stored thereon, the instructions, when executed by the one or more processors, cause the processors to perform operations comprising;

training a multi-script handwriting recognition model based on spatially-derived features of a multi-script training corpus, the multi-script training corpus including respective handwriting samples corresponding to characters of one or more scripts;

receiving a handwriting input from a user which includes characters from a plurality of scripts; and

in response to receiving the handwriting input, providing real-time handwriting recognition output for the handwriting input using the multi-script handwriting recognition model that has been trained on the spatially-derived features of the multi-script training corpus, wherein the real-time handwriting recognition output includes characters from a plurality of scripts.

14. A non-transitory computer-readable media having instructions stored thereon, the instructions, when executed by one or more processors, cause the processors to perform operations comprising:

receiving a multi-script handwriting recognition model, the multi-script recognition model having been trained on spatially-derived features of a multi-script training corpus, the multi-script training corpus including respective handwriting samples corresponding to characters of one or more scripts;

receiving a handwriting input from a user, the handwriting input comprising a plurality of handwritten strokes corresponding to a plurality of scripts provided on a touch-sensitive surface coupled to a user device; and

in response to receiving the handwriting input comprising a plurality of handwritten strokes corresponding to a plurality of scripts, providing in real-time one or more handwriting recognition results to the user based on the multi-script handwriting recognition model that has been trained on the spatially-derived features of the multi-script training corpus, wherein the one or more handwriting recognition results includes characters from a plurality of scripts.

15. The media of claim 14, wherein providing real-time handwriting recognition results to the user further comprises:

segmenting the user'"'"'s handwriting input into one or more recognition units, each recognition unit comprising one or more of the handwritten strokes provided by the user;

providing a respective image of each of the one or more recognition units as an input to the multi-script handwriting recognition model; and

for at least one of the one or more recognition units, obtaining from the multi-script handwriting recognition model, at least a first output character from a first script, and at least a second output character from a second script different from the first script.

16. The media of claim 15, wherein providing real-time handwriting recognition results to the user further comprises:

displaying both the first output character and the second output character in a candidate display area of a handwriting input user interface of the user device.

17. The media of claim 15, wherein providing real-time handwriting recognition results to the user further comprises:

selectively displaying one of the first output character and the second output character based on which one of the first or second scripts is a respective script used in a soft keyboard currently installed on the user device.

18. The media of claim 14, wherein providing real-time handwriting recognition for a user'"'"'s handwriting input further comprises:

continuously revising one or more recognition results for the user'"'"'s handwriting input in response to continued additions to or revisions of the handwriting input by the user; and

in response to each revision of the one or more recognition results, displaying the respective revised one or more recognition results to the user in a candidate display area of the handwriting input user interface.

19. The media of claim 14, wherein the one or more scripts include one or more Chinese characters, emoji characters, and Latin script.

20. The media of claim 14, wherein the one or more scripts include one or more Chinese characters, Arabic script, and Latin script.

21. The media of claim 14, wherein the multi-script handwriting recognition model is a single convolutional neural network having a single input plane and a single output plane, and includes spatially-derived features and respective weights for the spatially-derived features for differentiating characters of the one or more scripts represented in the multi-script training corpus.

22. The media of claim 14, wherein the multi-script handwriting recognition model has at least thirty thousand output classes, representing at least thirty thousand characters spanning at least one or more scripts.

23. The media of claim 14, wherein the multi-script handwriting recognition model is configured to recognize characters based on respective input images of one or more recognition units identified in the handwriting input, and wherein respective spatially-derived features used for recognition are independent of respective stroke order, stroke direction, and continuity of strokes in the handwriting input.

24. A method of providing multi-script handwriting recognition, comprising:

at a user device having one or more processors and memory;

receiving a multi-script handwriting recognition model, the multi-script recognition model having been trained on spatially-derived features of a multi-script training corpus, the multi-script training corpus including respective handwriting samples corresponding to characters of one or more scripts;

receiving a handwriting input from a user, the handwriting input comprising a plurality of handwritten strokes corresponding to a plurality of scripts provided on a touch-sensitive surface coupled to a user device; and

in response to receiving the handwriting input comprising a plurality of handwritten strokes corresponding to a plurality of scripts, providing in real-time one or more handwriting recognition results to the user based on the multi-script handwriting recognition model that has been trained on the spatially-derived features of the multi-script training corpus, wherein the one or more handwriting recognition results includes characters from a plurality of scripts.

25. A system, comprising:

one or more processors; and

memory having instructions stored thereon, the instructions, when executed by the one or more processors, cause the processors to perform operations comprising;

receiving a multi-script handwriting recognition model, the multi-script recognition model having been trained on spatially-derived features of a multi-script training corpus, the multi-script training corpus including respective handwriting samples corresponding to characters of one or more scripts;

receiving a handwriting input from a user, the handwriting input comprising a plurality of handwritten strokes corresponding to a plurality of scripts provided on a touch-sensitive surface coupled to a user device; and

in response to receiving the handwriting input comprising a plurality of handwritten strokes corresponding to a plurality of scripts, providing in real-time one or more handwriting recognition results to the user based on the multi-script handwriting recognition model that has been trained on the spatially-derived features of the multi-script training corpus, wherein the one or more handwriting recognition results includes characters from a plurality of scripts.