System and method of handwriting recognition in diagrams

• US 10,643,067 B2
• Filed: 12/01/2015
• Issued: 05/05/2020
• Est. Priority Date: 10/19/2015
• Status: Active Grant

• Current Assignee: Myscript
• Original Assignee: Myscript
• Inventors: Bednarowicz, Romain, MeLinand, Robin, Sidoli, Claire, Ric, Fabien, Elagouni, Khaoula, Hebert, David, Valente, Fabio, Cousin, Gregory, Nagot, Mael, Cerovic, Cyril, Bonnaud, Anne
• Primary Examiner(s): Nakhjavan, Shervin K
• Application Number:
  US14/955,155
• Publication Number:
  US 20170109578A1
• Time in Patent Office: 1,617 Days
• Field of Search: None
• CPC Class Codes: G06F 16/353 into predefined classes
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 40/205 Parsing
• CPC Class Codes: G06T 11/206 Drawing of charts or graphs
• CPC Class Codes: G06V 30/1423 the instrument generating sequences of position coordinates corresponding to handwriting preprocessing or recognising digital ink G06V30/32
• CPC Class Codes: G06V 30/32 Digital ink
• CPC Class Codes: G06V 30/347 Sampling; Contour coding; Stroke extraction
• CPC Class Codes: G06V 30/387 using human interaction, e.g. selection of the best displayed recognition candidate
• CPC Class Codes: G06V 30/422 Technical drawings; Geographical maps

Abstract

A system, method and computer program product for hand-drawing diagrams including text and non-text elements on a computing device are provided. The computing device has a processor and a non-transitory computer readable medium for detecting and recognizing hand-drawing diagram element input under control of the processor. Display of input diagram elements in interactive digital ink is performed on a display device associated with the computing device. One or more of the diagram elements are associated with one or more other of the diagram elements in accordance with a class and type of each diagram element. The diagram elements are re-displayed based on one or more interactions with the digital ink received and in accordance with the one or more associations.

Claims

1. A system for hand-drawing diagrams including text and non-text elements on a computing device, the computing device comprising a processor and at least one non-transitory computer readable medium for detecting and recognizing hand-drawing diagram element input under control of the processor, the at least one non-transitory computer readable medium configured to:

cause display of a plurality of input diagram elements in interactive digital ink on a display device associated with the computing device, wherein the diagram elements comprise one or more of a geometric shape, a container, a connector, or text;

incrementally associate one or more of the diagram elements with one or more other of the diagram elements in accordance with a class and type of each diagram element;

identify the class of each diagram element by classifying one or more strokes of the hand-drawn input as selected from a group consisting of text, non-text, and junk classifications for each of the one or more strokes; and

incrementally cause re-display of the diagram elements, wherein a display characteristic of a diagram element is influenced based on one or more received interactions with the digital ink, the one or more incremental associations, and a position of the one or more interactions relative to the incrementally associated diagram elements.

2. A system as claimed in claim 1, wherein the strokes are grouped based on spatial and temporal information of the input.

3. A system as claimed in claim 1, wherein the class of each diagram element is identified by classifying the one or more strokes of the hand-drawn input as text and non-text.

4. A system as claimed in claim 1, the at least one non-transitory computer readable medium configured to:

parse the classified strokes to a handwriting recognition system for recognition of the diagram elements of the classified strokes.

5. A system as claimed in claim 4, the at least one non-transitory computer readable medium configured to:

identify the type of each diagram element based on the recognized diagram elements and positional relationships between the diagram elements.

6. A system as claimed in claim 1, wherein the display characteristic comprises at least one of resizing or repositioning one or more diagram elements.

7. A method for hand-drawing diagrams including text and non-text elements on a computing device, the computing device comprising a processor and at least one non-transitory computer readable medium for detecting and recognizing hand-drawing diagram element input under control of the processor, the method comprising:

displaying a plurality of input diagram elements in interactive digital ink on a display device associated with the computing device, wherein the diagram elements comprise one or more of a geometric shape, a container, a connector, or text;

incrementally associating one or more of the diagram elements with one or more other of the diagram elements in accordance with a class and type of each diagram element;

identifying the class of each diagram element by classifying one or more strokes of the hand-drawn input as selected from a group consisting of text, non-text, and junk classifications for each of the one or more strokes; and

incrementally re-displaying the diagram elements, wherein a display characteristic of a diagram element is influenced based one or more received interactions with the digital ink, the one or more incremental associations, and a position of the one or more interactions relative to the incrementally associated diagram elements.

8. A method as claimed in claim 7, wherein the strokes are grouped based on spatial and temporal information of the input.

9. A method as claimed in claim 7, wherein the class of each diagram element is identified by classifying the one or more strokes of the hand-drawn input as text and non-text.

10. A method as claimed in claim 7, comprising parsing the classified strokes to a handwriting recognition system for recognition of the diagram elements of the classified strokes.

11. A method as claimed in claim 10, comprising identifying the type of each diagram element based on the recognized diagram elements and positional relationships between the diagram elements.

12. A non-transitory computer readable medium having a computer readable program code embodied therein, said computer readable program code adapted to be executed to implement a method for hand-drawing diagrams including text and non-text elements on a computing device, the computing device comprising a processor and at least one non-transitory computer readable medium for detecting and recognizing hand-drawing diagram element input under control of the processor, the method comprising:

displaying a plurality of input diagram elements in interactive digital ink on a display device associated with the computing device, wherein the diagram elements comprise one or more of a geometric shape, a container, a connector, or text;

incrementally associating one or more of the diagram elements with one or more other of the diagram elements in accordance with a class and type of each diagram element;

identifying the class of each diagram element by classifying one or more strokes of the hand-drawn input as selected from a group consisting of text, non-text, and junk classifications for each of the one or more strokes; and

incrementally re-displaying the diagram elements, wherein a display characteristic of a diagram element is influenced based on one or more received interactions with the digital ink, the one or more incremental associations, and a position of the one or more interactions relative to the incrementally associated diagram elements.

13. A non-transitory computer readable medium as claimed in claim 12, wherein the strokes are grouped based on spatial and temporal information of the input.

14. A non-transitory computer readable medium as claimed in claim 12, wherein the class of each diagram element is identified by classifying the one or more strokes of the hand-drawn input as text and non-text.

15. A non-transitory computer readable medium as claimed in claim 12, the method comprising parsing the classified strokes to a handwriting recognition system for recognition of the diagram elements of the classified strokes.

16. A non-transitory computer readable medium as claimed in claim 15, the method comprising identifying the type of each diagram element based on the recognized diagram elements and positional relationships between the diagram elements.