Managing real-time handwriting recognition

• US 9,898,187 B2
• Filed: 05/29/2014
• Issued: 02/20/2018
• Est. Priority Date: 06/09/2013
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Xia, Mei-Qun, Dolfing, Jannes G., Dixon, Ryan S., Groethe, Karl M., Misra, Karan, Bellegarda, Jerome R., Meier, Ueli
• Primary Examiner(s): Augustine, Nicholas
• Application Number:
  US14/290,935
• Publication Number:
  US 20140365949A1
• Time in Patent Office: 1,363 Days
• CPC Class Codes: G06F 3/018 Input/output arrangements for oriental characters
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 40/166 Editing, e.g. inserting or deleting
• CPC Class Codes: G06F 40/274 Converting codes to words; Guess-ahead of partial word inputs
• CPC Class Codes: G06V 30/347 Sampling; Contour coding; Stroke extraction
• CPC Class Codes: G06V 30/36 Matching; Classification
• CPC Class Codes: G06V 30/387 using human interaction, e.g. selection of the best displayed recognition candidate

Abstract

Methods, systems, and computer-readable media related to a technique for providing handwriting input functionality on a user device. A handwriting recognition module is trained to have a repertoire comprising multiple non-overlapping scripts and capable of recognizing tens of thousands of characters using a single handwriting recognition model. The handwriting input module provides real-time, stroke-order and stroke-direction independent handwriting recognition for multi-character handwriting input. In particular, real-time, stroke-order and stroke-direction independent handwriting recognition is provided for multi-character, or sentence level Chinese handwriting recognition. User interfaces for providing the handwriting input functionality are also disclosed.

Claims

1. A non-transitory computer-readable media comprising instructions stored thereon, the instructions, when executed by one or more processors, cause the processors to perform operations comprising:

receiving a handwriting input from a user, the handwriting input comprising a plurality of handwritten strokes provided in a touch-sensitive surface coupled to the device;

rendering, in real-time, the plurality of handwritten strokes in a handwriting input area of a handwriting input interface;

receiving one of a pinch gesture input and an expand gesture input over the plurality of handwritten strokes;

upon identifying two adjacent recognition units from the plurality of handwritten strokes and upon the one of a pinch gesture input and an expand gesture input being a pinch gesture input;

prior to receiving the pinch gesture input, displaying, in a candidate display area, a first initial recognition result comprising respective initial output characters recognized from the two adjacent recognition units;

subsequent to receiving the pinch gesture input, generating a first subsequent recognition result based on the plurality of handwritten strokes by treating the plurality of handwritten strokes as a single recognition unit;

replacing the first initial recognition result with the first subsequent recognition result in the candidate display area; and

in response to the pinch gesture input, re-rendering the plurality of handwritten strokes to reduce a distance between portions of the handwritten strokes corresponding to the two adjacent recognition units in the handwriting input area;

upon identifying a singular recognition unit from the plurality of handwritten strokes and upon the one of a pinch gesture input and an expand gesture input being an expand gesture input;

prior to receiving the expand gesture input, displaying, in the candidate display area, a second initial recognition result comprising an initial output character recognized from the singular recognition unit;

subsequent to receiving the expand gesture input, generating a second subsequent recognition result based on the plurality of handwritten strokes by treating the plurality of handwritten strokes as two separate recognition units pulled apart by the expand gesture input;

replacing the second initial recognition result with the second recognition result in the candidate display area, and

in response to the expand gesture input, re-rendering the plurality of handwritten strokes to increase a distance between portions of the handwritten strokes corresponding to the singular recognition unit in the handwriting input area.

2. The media of claim 1, wherein the pinch gesture input comprises two contacts on the touch-sensitive surface that converge toward each other in an area occupied by the plurality of handwritten strokes.

3. The media of claim 1, wherein the expand gesture input comprises two contacts on the touch-sensitive surface that diverge from each other in an area occupied by the plurality of handwritten strokes.

4. A method of providing handwriting recognition, comprising:

at a device having one or more processors and memory;

receiving a handwriting input from a user, the handwriting input comprising a plurality of handwritten strokes provided in a touch-sensitive surface coupled to the device;

rendering, in real-time, the plurality of handwritten strokes in a handwriting input area of a handwriting input interface;

receiving one of a pinch gesture input and an expand gesture input over the plurality of handwritten strokes;

upon identifying two adjacent recognition units from the plurality of handwritten strokes and upon the one of a pinch gesture input and an expand gesture input being a pinch gesture input;

prior to receiving the pinch gesture input, displaying, in a candidate display area, a first initial recognition result comprising respective initial output characters recognized from the two adjacent recognition units;

subsequent to receiving the pinch gesture input, generating a first subsequent recognition result based on the plurality of handwritten strokes by treating the plurality of handwritten strokes as a single recognition unit;

replacing the first initial recognition result with the first subsequent recognition result in the candidate display area; and

in response to the pinch gesture input, re-rendering the plurality of handwritten strokes to reduce a distance between portions of the handwritten strokes corresponding to the two adjacent recognition units in the handwriting input area;

upon identifying a singular recognition unit from the plurality of handwritten strokes and upon the one of a pinch gesture input and an expand gesture input being an expand gesture input;

prior to receiving the expand gesture input, displaying, in the candidate display area, a second initial recognition result comprising an initial output character recognized from the singular recognition unit;

subsequent to receiving the expand gesture input, generating a second subsequent recognition result based on the plurality of handwritten strokes by treating the plurality of handwritten strokes as two separate recognition units pulled apart by the expand gesture input;

replacing the second initial recognition result with the second recognition result in the candidate display area, and

in response to the expand gesture input, re-rendering the plurality of handwritten strokes to increase a distance between portions of the handwritten strokes corresponding to the singular recognition unit in the handwriting input area.

8. The method of claim 4, wherein the pinch gesture input comprises two contacts on the touch-sensitive surface that converge toward each other in an area occupied by the plurality of handwritten strokes.

9. The method of claim 4, wherein the expand gesture input comprises two contacts on the touch-sensitive surface that diverge from each other in an area occupied by the plurality of handwritten strokes.

5. A system, comprising:

one or more processors; and

memory having instructions stored thereon, the instructions, when executed by the one or more processors, cause the processors to perform operations comprising;

receiving a handwriting input from a user, the handwriting input comprising a plurality of handwritten strokes provided in a touch-sensitive surface coupled to the device;

rendering, in real-time, the plurality of handwritten strokes in a handwriting input area of a handwriting input interface;

receiving one of a pinch gesture input and an expand gesture input over the plurality of handwritten strokes;

upon identifying two adjacent recognition units from the plurality of handwritten strokes and upon the one of a pinch gesture input and an expand gesture input being a pinch gesture input;

prior to receiving the pinch gesture input, displaying, in a candidate display area, a first initial recognition result comprising respective initial output characters recognized from the two adjacent recognition units;

subsequent to receiving the pinch gesture input, generating a first subsequent recognition result based on the plurality of handwritten strokes by treating the plurality of handwritten strokes as a single recognition unit;

replacing the first initial recognition result with the first subsequent recognition result in the candidate display area; and

in response to the pinch gesture input, re-rendering the plurality of handwritten strokes to reduce a distance between portions of the handwritten strokes corresponding to the two adjacent recognition units in the handwriting input area;

upon identifying a singular recognition unit from the plurality of handwritten strokes and upon the one of a pinch gesture input and an expand gesture input being an expand gesture input;

prior to receiving the expand gesture input, displaying, in the candidate display area, a second initial recognition result comprising an initial output character recognized from the singular recognition unit;

subsequent to receiving the expand gesture input, generating a second subsequent recognition result based on the plurality of handwritten strokes by treating the plurality of handwritten strokes as two separate recognition units pulled apart by the expand gesture input;

replacing the second initial recognition result with the second recognition result in the candidate display area, and

in response to the expand gesture input, re-rendering the plurality of handwritten strokes to increase a distance between portions of the handwritten strokes corresponding to the singular recognition unit in the handwriting input area.

6. The system of claim 5, wherein the pinch gesture input comprises two contacts on the touch-sensitive surface that converge toward each other in an area occupied by the plurality of handwritten strokes.

7. The system of claim 5, wherein the expand gesture input comprises two contacts on the touch-sensitive surface that diverge from each other in an area occupied by the plurality of handwritten strokes.