Learning-based system and process for synthesizing cursive handwriting

• US 20040148577A1
• Filed: 01/27/2003
• Published: 07/29/2004
• Est. Priority Date: 01/27/2003
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Wu, Chenyu, Shum, Heung-Yeung, Wang, Jue, Xu, Ying-Qing
• Granted Patent: US 7,227,993 B2
• Time in Patent Office: Days
• US Class Current: 715/530
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06T 11/203 Drawing of straight lines or curves
• CPC Class Codes: G06V 30/1423 the instrument generating sequences of position coordinates corresponding to handwriting preprocessing or recognising digital ink G06V30/32

Abstract

A process and system for modeling, learning and synthesizing cursive handwriting in a user'"'"'s personal handwriting style. The handwriting synthesis system and process described herein addresses the problem of learning the personal handwriting style of a user based on limited handwriting samples and producing novel scripts of the same style. The handwriting synthesis process includes segmenting handwriting samples into individual characters using a two-level writer-independent segmentation process, aligning samples of the same character into a common coordinate frame, and learning and modeling the individual character. Synthesis of handwriting is performed by generating individual letters from the models and concatenating the letters using a conditional sampling algorithm. The result is a smooth and fluid connection between letters that successfully mimics the personal handwriting style of a user.

Claims

1. A method for synthesizing handwriting, comprising:

segmenting a handwriting sample into individual characters;

generating learned models for each of the individual characters; and

synthesizing the individual characters from the learned models.

2. The method as set forth in claim 1, wherein segmenting further comprises calculating a script code from the handwriting sample.

3. The method as set forth in claim 2, further comprising clustering similar individual characters segmented from the handwriting sample based on the script code.

4. The method as set forth in claim 1, wherein generating learned models further comprises aligning point vectors into a common coordinate frame.

5. The method as set forth in claim 4, further comprising modeling a distribution of aligned point vectors using shape characteristics.

6. The method as set forth in claim 1, further comprising decomposing the segmented individual characters into point vectors.

7. The method as set forth in claim 6, further comprising extracting the point vectors from the decomposed segmented individual characters.

8. The method as set forth in claim 1, further comprising aligning the point vectors to generate a distribution of aligned point vectors having similar shape characteristics.

9. The method as set forth in claim 1, further comprising learning shape characteristics of the segmented individual characters.

10. The method as set forth in claim 9, further comprising learning statistical parameters for the shape characteristics of the segmented individual characters.

11. The method as set forth in claim 1, wherein synthesizing further comprises connecting adjacent synthesized individual characters as needed to form words.

12. The method as set forth in claim 11, further comprising minimizing a total energy of the learned models to connect the synthesized individual characters.

13. The method as set forth in claim 11, further comprising calculating the total energy including a concatenation energy and a sampling energy.

14. A computer-readable medium having computer-executable instructions for performing the method recited in claim 1.

15. A process for generating cursive handwriting, comprising:

segmenting each handwritten word in a handwriting specimen into separate characters;

learning shape characteristics of each of the characters;

building learned models of each of the characters using the learned shape characteristics; and

synthesizing the cursive handwriting from the learned models.

16. The process as set forth in claim 15, further comprising clustering characters having similar shape characteristics by calculating a script code for each of the characters.

17. The process as set forth in claim 15, further comprising decomposing the characters into point vectors and extracting the point vectors.

18. The process as set forth in claim 17, further comprising learning parameters of the shape characteristics.

19. The process as set forth in claim 15, further comprising calculating a sampling energy of the learned models.

20. The process as set forth in claim 19, further comprising calculating a concatenation energy of the learned models.

21. The process as set forth in claim 20, defining a total energy function that includes the concatenation energy and the sampling energy.

22. The process as set forth in claim 21, further comprising minimizing the total energy function to smoothly connect adjacent characters.

23. One or more computer-readable media having computer-readable instructions thereon which, when executed by one or more processors, cause the one or more processors to implement the method of claim 15.

24. A computer-readable medium having computer-executable instructions for synthesizing text in a handwriting style of a user, comprising:

automatically segmenting a handwriting sample written in the handwriting style of the user into individual characters using a two-level segmentation process;

building models of the individual characters in the handwriting style by analyzing the individual characters;

synthesizing the individual characters from the models; and

connecting the individual characters to form words in the user'"'"'s handwriting style.

25. The computer-readable medium of claim 24, further comprising designing the handwriting sample according to at least two criterion.

26. The computer-readable medium of claim 25, wherein one of the criterion includes having the handwriting sample comprise integrative sentences and paragraphs.

27. The computer-readable medium of claim 25, wherein one of the criterion includes ensuring that an appearance frequency of characters in the handwriting sample is consistent with an appearance frequency of characters in standard documents.

28. The computer-readable medium of claim 24, wherein the two-level segmentation process uses a level-building framework whereby both segmentation and recognition are merged.

29. The computer-readable medium of claim 24, wherein the two-level segmentation process is writer independent.

30. The computer-readable medium of claim 29, further comprising building and using a handwriting database to train the two-level segmentation process.

31. The computer-readable medium of claim 24, wherein the two-level segmentation process calculates a script code from the handwriting sample to use as a middle level feature.

32. The computer-readable medium of claim 24, wherein analyzing the individual characters further comprises matching the individual characters using a hierarchical process.

33. The computer-readable medium of claim 32, wherein the hierarchical process further comprises decomposing a trajectory of the individual characters into static strokes.

34. The computer-readable medium of claim 32, wherein the hierarchical process further comprises extracting a point vector from a trajectory of the individual characters.

35. The computer-readable medium of claim 24, wherein connecting the individual characters further comprises using a conditional sampling process to connect the characters smoothly and naturally.

36. The computer-readable medium of claim 35, wherein the conditional sampling process further comprises using a delta log-normal model.

37. The computer-readable medium of claim 24, wherein the individual characters are connected under the constraint of the models.

38. The computer-readable medium of claim 24, wherein the individual characters are connected under the constraint of the character'"'"'s neighbors.

39. A method for synthesizing a personal cursive handwriting style of a user, comprising:

collecting a cursive handwriting sample of the user;

segmenting the cursive handwriting sample into individual letters using a two-level segmentation technique;

automatically matching and aligning handwriting strokes for the individual letters to generate a training set;

learning a shape variation for the individual letters of the training set using a Principal Component Analysis (PCA) model;

selectively generating synthesized letters from the PCA model; and

smoothly connecting the synthesized letters within a word in order to produce a fluent cursive script that is an accurate reproduction of the user'"'"'s personal cursive handwriting style.

40. The method as set forth in claim 39, wherein the handwriting sample contains approximately two-hundred words.

41. The method as set forth in claim 39, wherein the two-level segmentation technique further comprises building a handwriting database.

42. The method as set forth in claim 39, wherein the two-level segmentation technique further comprises calculating a script code from the handwriting sample as the middle-level feature for sub-class clustering.

43. The method as set forth in claim 42, further comprising determining a distance between script codes of individual letters.

44. The method as set forth in claim 43, further comprising clustering similar segmented letters having similar structural topology into a cluster.

45. The method as set forth in claim 39, wherein the handwriting strokes are automatically matched and aligned by using an iterative alignment approach.

46. The method as set forth in claim 45, wherein the iterative alignment further comprises maintained an affine transform matrix for each of the segmented individual letters.

47. The method as set forth in claim 46, further comprising setting the affine transform matrix initially to identity.

48. The method as set forth in claim 46, further comprising computing a deformable energy-based criterion.

49. The method as set forth in claim 48, further comprising reducing the deformable energy-based criterion until convergence.

50. The method as set forth in claim 39, wherein smoothly connecting the synthesized letters further comprises calculating a concatenation energy of the word.

51. The method as set forth in claim 50, further comprising calculating a sampling energy.

52. The method as set forth in claim 51, further comprising calculating a total energy using the concatenation energy and the sampling energy.

53. The method as set forth in claim 52, further comprising minimizing the total energy to determine a point of connection between adjacent synthesized letters of the word.

54. The method as set forth in claim 39, wherein smoothly connecting the synthesized letters further comprises comparing a total energy of the word to an energy threshold value.

55. The method as set forth in claim 54, further comprising connecting adjacent synthesized letters when the total energy is less than the energy threshold value.

56. The method as set forth in claim 54, further comprising leaving adjacent synthesized letters unconnected when the total energy is greater or equal to the energy threshold value.

57. A process for reproducing a user'"'"'s personal handwriting style, comprising:

obtaining a handwriting sample of the user'"'"'s personal handwriting style;

segmenting the handwriting sample into individual characters using a two-level segmentation process having a level-building framework that uses recognition and segmentation to form training samples of the individual characters;

aligning the training samples to form a samples distribution;

modeling the samples distribution by applying Principal Component Analysis (PCA) to the training samples to build a handwriting model of letters; and

generating individual letters of a word from the handwriting model of letters under the constraints of neighboring letters in the word.

58. The process as set forth in claim 57, wherein each letter of an alphabet appears in the handwriting sample at least five times.

59. The process as set forth in claim 57, further comprising training the two-level segmentation process using a handwriting database.

60. The process as set forth in claim 59, wherein the handwriting database includes at least approximately 100,000 words written by at least approximately 100 persons.

61. The process as set forth in claim 57, further comprising calculating a script code from the handwriting sample, the script code having at least five kinds of key points.

62. The process as set forth in claim 61, wherein the at least five key points comprise at least one of:

(a) a point of maximum in a x coordinate direction;

(b) a point of minimum in the x coordinate direction;

(c) a point of maximum in a y coordinate direction;

(d) a point of maximum in the y coordinate direction;

(e) crossing points of the individual characters.

63. The process as set forth in claim 57, wherein generating individual letters of a word further comprises defining a concatenation energy.

64. The process as set forth in claim 63, further comprising defining at least three terms in the concatenation energy.

65. The process as set forth in claim 64, wherein the three terms include:

(a) a deformation constraint term;

(b) an end point alignment term;

(c) an interval diminishment term.

66. The process as set forth in claim 57, further comprising connecting adjacent individual letters of the word based on a total energy of the word.

67. The process as set forth in claim 66, further comprising minimizing the total energy using an iterative minimization process.

68. The process as set forth in claim 66, wherein the iterative minimization process further comprises:

deforming the strokes for each pair of adjacent letters to minimize concatenation energy of the pair;

projecting a deformed shape into a model coordinate frame;

updating model parameters; and

repeating the above process until convergence.

69. A handwriting synthesis system, comprising:

a segmentation module that segments a handwriting sample into training samples of individual characters;

a learning module that builds learned models of each handwritten character using the training samples; and

a synthesis module that synthesizes handwriting using the learned models.

70. The handwriting synthesis system as set forth in claim 69, wherein the segmentation module further comprises:

a script code distance module that divides samples of each of the individual characters in the handwriting sample into several clusters based on a distance between script codes; and

a clustering module that clusters the samples of individual characters having similar characteristics into a single cluster.

71. The handwriting synthesis system as set forth in claim 70, wherein the characteristics are shape characteristics of the individual characters.

72. The handwriting synthesis system as set forth in claim 70, wherein the single cluster represents a training sample one of the individual characters.

73. The handwriting synthesis system as set forth in claim 70, wherein each of the individual characters may be uniquely coded by the script codes.

74. The handwriting synthesis system as set forth in claim 69, wherein the learning module further comprises:

an alignment module that takes the training samples and normalizes and aligns point vectors for each training sample; and

a modeling module that takes the aligned point vectors and learns shape characteristics of each individual character.

75. The handwriting synthesis system as set forth in claim 74, wherein statistical analysis is used to learn the shape characteristics.

76. The handwriting synthesis system as set forth in claim 69, wherein the synthesis module further comprises a concatenation module that joins synthesized characters with neighboring characters as necessary to form text and words of synthesized handwriting.

77. The handwriting synthesis system as set forth in claim 76, further comprising:

a selection module that selects desired characters from the learned models of individual characters; and

a baseline alignment module that places the individual characters in proper sequence and aligns baselines of the characters.

78. The handwriting synthesis system as set forth in claim 76, wherein the concatenation module further comprises:

a concatenation energy module that calculates a concatenation energy of a synthesized word;

a sampling energy module that calculates a sampling energy to ensure that a deformed synthesized character is consistent with a model of that individual character; and

an energy minimization module that minimizes a total energy of the combined concatenation and sampling energies to determine a point of connection of a synthesized character to its neighbors.

79. A cursive handwriting synthesis system that synthesizes a cursive handwriting style of a user, comprising:

a segmentation module that extracts individual characters from a handwriting sample written in the cursive handwriting style of the user;

a learning module that analyzes the individual characters and builds models for each character;

a synthesis module that synthesizes single characters using the models for each character; and

a concatenation module within the synthesis module that connects the synthesized single characters to generate words in the cursive handwriting style of the user.

80. The cursive handwriting synthesis system as set forth in claim 79, wherein the learning module further comprises a learned model module that build the models for each character based on learned shape characteristics of each individual character.