INTEGRATING STROKE-DISTRIBUTION INFORMATION INTO SPATIAL FEATURE EXTRACTION FOR AUTOMATIC HANDWRITING RECOGNITION

• US 20140363082A1
• Filed: 05/30/2014
• Published: 12/11/2014
• Est. Priority Date: 06/09/2013
• Status: Abandoned Application

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: DIXON, Ryan S., DOLFING, Jannes G., MEIER, Ueli, BELLEGARDA, Jerome R.
• Application Number:
  US14/291,722
• Publication Number:
  US 20140363082A1
• Time in Patent Office: Days
• US Class Current: 382/187
• CPC Class Codes: G06V 30/32 Digital ink
• CPC Class Codes: G06V 30/333 Preprocessing; Feature extraction

Abstract

Methods, systems, and computer-readable media related to a technique for providing handwriting input functionality on a user device. A handwriting recognition module is trained to have a repertoire comprising multiple non-overlapping scripts and capable of recognizing tens of thousands of characters using a single handwriting recognition model. The handwriting input module provides real-time, stroke-order and stroke-direction independent handwriting recognition. In some embodiments, temporally-derived features are used to improve recognition accuracy without compromising the stroke-order and stroke-direction independence of the recognition system.

Claims

1. A non-transitory computer-readable media having instructions stored thereon, the instructions, when executed by one or more processors, cause the processors to perform operations comprising:

separately training a set of spatially-derived features and a set of temporally-derived features of a handwriting recognition model, wherein;

the set of spatially-derived features are trained on a corpus of training images each being an image of a handwriting sample for a respective character of an output character set, and

the set of temporally-derived features are trained on a corpus of stroke-distribution profiles, each stroke-distribution profile numerically characterizing a spatial distribution of a plurality of strokes in a handwriting sample for a respective character of the output character set;

combining the set of spatially-derived features and the set of temporally-derived features in the handwriting recognition model; and

providing real-time handwriting recognition for a user'"'"'s handwriting input using the handwriting recognition model.

2. The media of claim 1, wherein separately training the set of spatially-derived features further comprises:

training a convolutional neural network having an input layer, an output layer, and a plurality of convolutional layers including a first convolutional layer, a last convolutional layer, zero or more intermediate convolutional layers between the first convolutional layer and the last convolutional layer, and a hidden layer between the last convolutional layer and the output layer.

3. The media of claim 2, wherein separately training the set of temporally-derived features further comprises:

providing the plurality of stroke-distribution profiles to a statistical model to determine a plurality of temporally-derived parameters and respective weights for the plurality of temporally-derived parameters for classifying the respective characters of the output character set.

4. The media of claim 3, wherein combining the set of spatially-derived features and the set of temporally-derived features in the handwriting recognition model comprises:

injecting the plurality of spatially-derived parameters and the plurality of temporally-derived parameters into one of the convolutional layers or the hidden layer of the convolutional neural network.

5. The media of claim 4, wherein the plurality of temporally-derived parameters and respective weights for the plurality temporally-derived parameters are injected into the last convolutional layer of the convolutional neural network for handwriting recognition.

6. The media of claim 4, wherein the plurality of temporally-derived parameters and respective weights for the plurality temporally-derived parameters are injected into the hidden layer of the convolutional handwriting recognition.

7. The media of claim 1, comprising instructions, which when executed by the one or more processors, cause the processors to perform operations comprising:

generating the corpus of stroke-distribution profiles from a plurality writing samples,

wherein each of the plurality of handwriting samples corresponds to a character in the output character set and separately preserves respective spatial information for each constituent stroke of the handwriting sample as it was written, and

wherein generating the corpus of stroke-distribution profiles further comprises;

for each of the plurality of handwriting samples;

identifying constituent strokes in the handwriting sample;

for each of the identified strokes of the handwriting sample, calculate a respective occupancy ratio along each of a plurality of predetermined directions, occupancy ratio being a ratio between a projected span of said each stroke direction and a maximum projected span of said writing sample;

for each of the identified strokes of the handwriting sample, calculating a respective saturation ratio for said each stroke based on a ratio between a respective number of pixels within said each stroke and an overall number of pixels within said writing sample; and

generating a feature vector for the handwriting sample as the stroke-distribution profile of the writing sample, the feature vector including the respective occupancy ratios and the respective saturation ratio of at least N strokes in the handwriting sample, wherein N is a predetermined natural number.

8. The media of claim 7, wherein N is less than a maximum stroke count observed in any single writing sample within the plurality of writing samples.

9. The media of claim 8, comprising instructions, which when executed by the one or more processors, cause the processors to perform operations comprising:

for each of the plurality of handwriting samples;

sorting the respective occupancy ratios of the identified strokes in each of the predetermined directions in a descending order; and

including only N top-ranked occupancy ratios and saturation ratios of writing sample in the feature vector of the writing sample.

10. The media of claim 7, wherein the plurality of predetermined directions including a horizontal direction, a vertical direction, a positive 45 degree direction, and a negative 45 degree direction of the writing sample.

11. The media of claim 1, wherein providing real-time handwriting recognition for a user'"'"'s handwriting input using the handwriting recognition model further comprises:

receiving the user'"'"'s handwriting input;

in response to receiving the user'"'"'s handwriting input, providing a handwriting recognition output to the user substantially contemporaneously with the receipt of the handwriting input.

12. A method of providing hand-writing recognition, comprising:

at a device having one or more processors and memory;

separately training a set of spatially-derived features and a set of temporally-derived features of a handwriting recognition model, wherein;

the set of spatially-derived features are trained on a corpus of training images each being an image of a handwriting sample for a respective character of an output character set, and

the set of temporally-derived features are trained on a corpus of stroke-distribution profiles, each stroke-distribution profile numerically characterizing a spatial distribution of a plurality of strokes in a handwriting sample for a respective character of the output character set;

combining the set of spatially-derived features and the set of temporally-derived features in the handwriting recognition model; and

providing real-time handwriting recognition for a user'"'"'s handwriting input using the handwriting recognition model.

13. A system, comprising

one or more processors;

and

memory having instructions stored thereon, the instructions, when executed by the one or more processors, cause the processors to perform operations comprising;

separately training a set of spatially-derived features and a set of temporally-derived features of a handwriting recognition model, wherein;

the set of spatially-derived features are trained on a corpus of training images each being an image of a handwriting sample for a respective character of an output character set, and

the set of temporally-derived features are trained on a corpus of stroke-distribution profiles, each stroke-distribution profile numerically characterizing a spatial distribution of a plurality of strokes in a handwriting sample for a respective character of the output character set;

combining the set of spatially-derived features and the set of temporally-derived features in the handwriting recognition model; and

providing real-time handwriting recognition for a user'"'"'s handwriting input using the handwriting recognition model.