Managing real-time handwriting recognition

• US 10,346,035 B2
• Filed: 09/23/2016
• Issued: 07/09/2019
• Est. Priority Date: 06/09/2013
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Xia, Mei-Qun, Dolfing, Jannes G., Dixon, Ryan S., Groethe, Karl M., Misra, Karan, Bellegarda, Jerome R., Meier, Ueli
• Primary Examiner(s): Dunphy, David F
• Application Number:
  US15/275,309
• Publication Number:
  US 20170010802A1
• Time in Patent Office: 1,019 Days
• CPC Class Codes: G06F 3/018 Input/output arrangements for oriental characters
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 40/166 Editing, e.g. inserting or deleting
• CPC Class Codes: G06F 40/274 Converting codes to words; Guess-ahead of partial word inputs
• CPC Class Codes: G06V 30/347 Sampling; Contour coding; Stroke extraction
• CPC Class Codes: G06V 30/36 Matching; Classification
• CPC Class Codes: G06V 30/387 using human interaction, e.g. selection of the best displayed recognition candidate

Abstract

Methods, systems, and computer-readable media related to a technique for providing handwriting input functionality on a user device. A handwriting recognition module is trained to have a repertoire comprising multiple non-overlapping scripts and capable of recognizing tens of thousands of characters using a single handwriting recognition model. The handwriting input module provides real-time, stroke-order and stroke-direction independent handwriting recognition for multi-character handwriting input. In particular, real-time, stroke-order and stroke-direction independent handwriting recognition is provided for multi-character, or sentence level Chinese handwriting recognition. User interfaces for providing the handwriting input functionality are also disclosed.

Claims

1. A method, comprising:

at an electronic device having a one or more processors, a touch sensitive surface, and a display;

displaying a user input interface on the display including a message area and a stroke input area;

receiving a first set of strokes on the touch sensitive surface in the stroke input area;

determining a first single character based on the first set of strokes;

displaying the first single character in the message area;

receiving a second set of strokes on the touch sensitive surface in the stroke input area after receiving the first set of strokes and displaying the first single character;

determining revised first single character based on the first set of strokes and the second set of strokes; and

replacing the display of the first single character with the revised first single character.

2. The method of claim 1, further comprising:

rendering the first set of strokes in the stroke input area; and

starting a fading process for the first set of strokes, wherein during the fading process, the rendering of the first set of strokes in the stroke input area becomes increasingly faded.

3. The method of claim 2, wherein the fading process for the first set of strokes is started when a predetermined time period has elapsed after the first set of strokes is completed by the user.

4. The method of claim 2, wherein the fading process for the first set of strokes is started when the user has started inputting the second set of strokes.

5. The method of claim 2, wherein an end state of the fading process for the first set of strokes is a state with a predetermined minimum visibility for the first set of strokes.

6. The method of claim 2, wherein an end state of the fading process for the first set of strokes is a state with zero visibility for the first set of strokes.

7. The method of claim 1, wherein displaying the first single character occurs before receiving the second set of strokes.

8. The method of claim 1, further comprising:

after receiving the second set of strokes and after displaying the revised first single character, receiving a third set of strokes on the touch sensitive surface in the stroke input area;

determining second single character based on the third set of strokes; and

displaying the second single character with the revised first single character.

9. The method of claim 1, wherein the first set of strokes is a single continuous stroke.

10. The method of claim 1, wherein the second set of strokes is a single continuous stroke.

11. The method of claim 1, wherein the first single character includes a character not included in the revised first single character.

12. A non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of an electronic device with a touch sensitive surface and a display, the one or more programs including instructions for:

displaying a user input interface on the display including a message area and a stroke input area;

receiving a first set of strokes on the touch sensitive surface in the stroke input area;

determining a first single character based on the first set of strokes;

displaying the first single character in the message area;

receiving a second set of strokes on the touch sensitive surface in the stroke input area after receiving the first set of strokes and displaying the first single character;

determining revised first single character based on the first set of strokes and the second set of strokes; and

replacing the display of the first single character with the revised first single character.

13. The non-transitory computer-readable storage medium of claim 12, further including instructions for:

rendering the first set of strokes in the stroke input area; and

starting a fading process for the first set of strokes, wherein during the fading process, the rendering of the first set of strokes in the stroke input area becomes increasingly faded.

14. The non-transitory computer-readable storage medium of claim 13, wherein the fading process for the first set of strokes is started when a predetermined time period has elapsed after the first set of strokes is completed by the user.

15. The non-transitory computer-readable storage medium of claim 13, wherein the fading process for the first set of strokes is started when the user has started inputting the second set of strokes.

16. The non-transitory computer-readable storage medium of claim 13, wherein an end state of the fading process for the first set of strokes is a state with a predetermined minimum visibility for the first set of strokes.

17. The non-transitory computer-readable storage medium of claim 13, wherein an end state of the fading process for the first set of strokes is a state with zero visibility for the first set of strokes.

18. The non-transitory computer-readable storage medium of claim 12, wherein displaying the first single character occurs before receiving the second set of strokes.

19. The non-transitory computer-readable storage medium of claim 12, further including instructions for:

after receiving the second set of strokes and after displaying the revised first single character, receiving a third set of strokes on the touch sensitive surface in the stroke input area;

determining second single character based on the third set of strokes; and

displaying the second single character with the revised first single character.

20. The non-transitory computer-readable storage medium of claim 12, wherein the first set of strokes is a single continuous stroke.

21. The non-transitory computer-readable storage medium of claim 12, wherein the second set of strokes is a single continuous stroke.

22. The non-transitory computer-readable storage medium of claim 12, wherein the first single character includes a character not included in the revised first single character.

23. An electronic device, comprising:

a touch sensitive surface;

a display;

one or more processors; and

a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for;

displaying a user input interface on the display including a message area and a stroke input area;

receiving a first set of strokes on the touch sensitive surface in the stroke input area;

determining a first single character based on the first set of strokes;

displaying the first single character in the message area;

receiving a second set of strokes on the touch sensitive surface in the stroke input area after receiving the first set of strokes and displaying the first single character;

determining revised first single character based on the first set of strokes and the second set of strokes; and

replacing the display of the first single character with the revised first single character.

24. The electronic device of claim 23, further including instructions for:

rendering the first set of strokes in the stroke input area; and

starting a fading process for the first set of strokes, wherein during the fading process, the rendering of the first set of strokes in the stroke input area becomes increasingly faded.

25. The electronic device of claim 24, wherein the fading process for the first set of strokes is started when a predetermined time period has elapsed after the first set of strokes is completed by the user.

26. The electronic device of claim 24, wherein the fading process for the first set of strokes is started when the user has started inputting the second set of strokes.

27. The electronic device of claim 24, wherein an end state of the fading process for the first set of strokes is a state with a predetermined minimum visibility for the first set of strokes.

28. The electronic device of claim 24, wherein an end state of the fading process for the first set of strokes is a state with zero visibility for the first set of strokes.

29. The electronic device of claim 23, wherein displaying the first single character occurs before receiving the second set of strokes.

30. The electronic device of claim 23, further including instructions for:

after receiving the second set of strokes and after displaying the revised first single character, receiving a third set of strokes on the touch sensitive surface in the stroke input area;

determining second single character based on the third set of strokes; and

displaying the second single character with the revised first single character.

31. The electronic device of claim 23, wherein the first set of strokes is a single continuous stroke.

32. The electronic device of claim 23, wherein the second set of strokes is a single continuous stroke.

33. The electronic device of claim 23, wherein the first single character includes a character not included in the revised first single character.

34. A method, comprising:

at an electronic device having one or more processors, a touch sensitive surface, and a display;

displaying a user input interface on the display including a message area and a stroke input area;

receiving a first set of strokes on the touch sensitive surface in the stroke input area;

determining first text based on the first set of strokes;

displaying the first text on the display in the message area;

determining one or more candidates based on the first set of strokes, wherein the one or more candidates include an emoji;

displaying the one or more candidates in a candidate display area;

while displaying the one or more candidates, receiving user input selecting a first candidate from the one or more candidates;

in response to the user input;

replacing display of the first text with display of the selected first candidate; and

clearing the one or more candidates from the candidate display area.

35. The method of claim 34, wherein the first text includes multiple characters.

36. The method of claim 34, wherein the first text is a single character.

37. The method of claim 34, further comprising:

receiving a second set of strokes on the touch sensitive surface in the stroke input area after receiving the first set of strokes and displaying the first text, wherein determining the one or more candidates is further based on the second set of strokes.

38. The method of claim 34, wherein the first set of strokes is a single continuous stroke.

39. The method of claim 37, wherein the second set of strokes is a single continuous stroke.

40. The method of claim 34, further comprising:

rendering the first set of strokes in the stroke input area; and

starting a fading process for the first set of strokes, wherein during the fading process, the rendering of the first set of strokes in the stroke input area becomes increasingly faded.

41. The method of claim 40, wherein the fading process for the first set of strokes is started when a predetermined time period has elapsed after the first set of strokes is completed by the user.

42. The method of claim 40, wherein an end state of the fading process for the first set of strokes is a state with a predetermined minimum visibility for the first set of strokes.

43. The method of claim 40, wherein an end state of the fading process for the first set of strokes is a state with zero visibility for the first set of strokes.

44. A non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of an electronic device with a touch sensitive surface and a display, the one or more programs including instructions for:

displaying a user input interface on the display including a message area and a stroke input area;

receiving a first set of strokes on the touch sensitive surface in the stroke input area;

determining first text based on the first set of strokes;

displaying the first text on the display in the message area;

determining one or more candidates based on the first set of strokes, wherein the one or more candidates include an emoji;

displaying the one or more candidates in a candidate display area;

while displaying the one or more candidates, receiving user input selecting a first candidate from the one or more candidates;

in response to the user input;

replacing display of the first text with display of the selected first candidate; and

clearing the one or more candidates from the candidate display area.

45. The non-transitory computer-readable storage medium of claim 44, wherein the first text includes multiple characters.

46. The non-transitory computer-readable storage medium of claim 44, wherein the first text is a single character.

47. The non-transitory computer-readable storage medium of claim 44, further including instructions for:

receiving a second set of strokes on the touch sensitive surface in the stroke input area after receiving the first set of strokes and displaying the first text, wherein determining the one or more candidates is further based on the second set of strokes.

48. The non-transitory computer-readable storage medium of claim 47, wherein the second set of strokes is a single continuous stroke.

49. The non-transitory computer-readable storage medium of claim 44, wherein the first set of strokes is a single continuous stroke.

50. The non-transitory computer-readable storage medium of claim 44, further including instructions for:

rendering the first set of strokes in the stroke input area; and

starting a fading process for the first set of strokes, wherein during the fading process, the rendering of the first set of strokes in the stroke input area becomes increasingly faded.

51. The non-transitory computer-readable storage medium of claim 50, wherein the fading process for the first set of strokes is started when a predetermined time period has elapsed after the first set of strokes is completed by the user.

52. The non-transitory computer-readable storage medium of claim 50, wherein an end state of the fading process for the first set of strokes is a state with a predetermined minimum visibility for the first set of strokes.

53. The non-transitory computer-readable storage medium of claim 50, wherein an end state of the fading process for the first set of strokes is a state with zero visibility for the first set of strokes.

54. An electronic device, comprising:

a touch sensitive surface;

a display;

one or more processors; and

a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for;

displaying a user input interface on the display including a message area and a stroke input area;

receiving a first set of strokes on the touch sensitive surface in the stroke input area;

determining first text based on the first set of strokes;

displaying the first text on the display in the message area;

determining one or more candidates based on the first set of strokes, wherein the one or more candidates include an emoji;

displaying the one or more candidates in a candidate display area;

while displaying the one or more candidates, receiving user input selecting a first candidate from the one or more candidates;

in response to the user input;

replacing display of the first text with display of the selected first candidate; and

clearing the one or more candidates from the candidate display area.

55. The electronic device of claim 54, wherein the first text includes multiple characters.

56. The electronic device of claim 54, wherein the first text is a single character.

57. The electronic device of claim 54, further including instructions for:

receiving a second set of strokes on the touch sensitive surface in the stroke input area after receiving the first set of strokes and displaying the first text, wherein determining the one or more candidates is further based on the second set of strokes.

58. The electronic device of claim 57, wherein the second set of strokes is a single continuous stroke.

59. The electronic device of claim 54, wherein the first set of strokes is a single continuous stroke.

60. The electronic device of claim 54, further including instructions for:

rendering the first set of strokes in the stroke input area; and

starting a fading process for the first set of strokes, wherein during the fading process, the rendering of the first set of strokes in the stroke input area becomes increasingly faded.

61. The electronic device of claim 60, wherein the fading process for the first set of strokes is started when a predetermined time period has elapsed after the first set of strokes is completed by the user.

62. The electronic device of claim 60, wherein an end state of the fading process for the first set of strokes is a state with a predetermined minimum visibility for the first set of strokes.

63. The electronic device of claim 60, wherein an end state of the fading process for the first set of strokes is a state with zero visibility for the first set of strokes.